Web3 Ethereum DeFi Development
Development instructions for working with the web3-ethereum-defi codebase - a Python library for trading automation on DeFi, data research and integration. Supports Uniswap, Aave, Chainlink, USDC and other protocols.
Language and Style Conventions
**British English**
Use UK/British English instead of US EnglishWrite `visualise` instead of `visualize`For headings, only capitalise the first letter, do not use title caseRunning Python Scripts
Always use Poetry to run Python scripts to ensure the virtual environment is activated:
```shell
poetry run python scripts/logos/post-process-logo.py
```
Running Tests
Before running tests for the first time, create a gitignored file `.local-test.env` in the repository root. This file uses `source` command to include test secrets from outside the repository. **Never edit this file** - always ask the user to prepare it.
Run tests using pytest with the Poetry environment and environment secrets:
```shell
source .local-test.env && poetry run pytest {test case name or pattern}
```
**Important testing rules:**
Always prefix pytest commands with the source command for environment variablesAvoid running the whole test suite (takes several minutes) - only run specific test casesAlways use extended timeout: `timeout: 180000` (3 minutes) in bash tool parametersFor extra output, pass `--log-cli-level=info` argument to pytestEnvironment Variable Configuration and RPC URLs
Environment variables like `JSON_RPC_ETHEREUM`, `JSON_RPC_ARBITRUM` are used to interact with EVM-based blockchains.
**RPC URL format rules:**
RPC URLs use project-specific space-separated fallback format (see `mev-blocker.rst`)If an RPC URL contains spaces, it can only be used with `create_multi_provider_web3()`For other commands like `curl`, parse the RPC variable by splitting on spaces and taking the first entryAll environment variables point to EVM archive nodes**Available RPC environment variables:**
`JSON_RPC_ETHEREUM` - Ethereum mainnet`JSON_RPC_ARBITRUM` - Arbitrum One`JSON_RPC_BASE` - Base`JSON_RPC_POLYGON` - Polygon`JSON_RPC_BINANCE` - BNB Smart Chain`JSON_RPC_HYPERLIQUID` - HyperEVMSee `CHAIN_NAMES` and `eth_defi.provider.env` for chain aliases (e.g., chain id 999 → JSON_RPC_HYPERLIQUID).
Code Formatting
Format code using ruff with Poetry:
```shell
poetry run ruff format
```
Pull Requests
Never push directly to master - always open a pull requestDo not include test plan in PR descriptionsFor feature PRs: prefix title with "feat:" and add one line to `CHANGELOG.md`Changelog entries should include PR date in YYYY-MM-DD format - Example: `Something was updated (2026-01-01)`
Always format code before opening or updating a PRPython Code Rules
**Data Structures**
Prefer `dataclass(slots=True)` for data structures**Async/Threading**
Use threaded instead of async Python code**Type Hints**
Always type hint function arguments and return valuesUse `HexAddress` instead of `str` for blockchain addressesUse `eth_defi.types.Percent` type alias instead of raw float for percentages**Performance**
Prefer Pandas `apply()` and functional helpers over slow for/while loopsUse `any()` and `all()` with generators and list comprehension instead of loopsAlways return `Iterator` instead of `list` from functions to improve performanceFor long-running loops, use `tqdm` and `tqdm_loggable.auto` for progress bars (see `lead_scan_core.py`)**Naming Conventions**
Prefix network data-fetching functions with `fetch_` instead of `get_`**Visualizations**
Use Plotly for visualizationsUse heading case for chart titles (as explained in style conventions)Code Comments and Documentation
**Comment Style**
Use Sphinx restructured text style for code commentsFor dataclass and Enum members, use Sphinx `#: comment here` line comment above variable, not `:param:`If a class function overloads a parent function with nothing new to comment, leave the comment empty instead of repeating**API Documentation**
All API modules should have stub entry under `docs/source/api`Cross-reference in `docs/source/api/index` table of contentsSee `docs/source/api/index.rst` and `docs/source/api/lagoon/index.rst` as examplesLogging
Use module-level logger pattern:
```python
logger = logging.getLogger(__name__)
```
When logging with `logger.info()`, `logger.debug()`, etc., prefer `%s` and `%f` unexpanded string syntax over Python string interpolation for performance reasons.
DateTime Handling
Use naive UTC datetimes everywhereImport as `import datetime.datetime` and use `datetime.datetime` and `datetime.timedelta` as type hintsUse `native_datetime_utc_now()` instead of `datetime.datetime.utcnow()` for cross-version compatibilityEnums
For string enums, both members and values must be in snake_case.
Pytest Best Practices
Never use test classes in pytest - use functions insteadTests should not have stdout output like `print`Use `pytest.approx()` instead of manual float fuzzy comparison like `assert abs(value - expected) < 0.01`For DuckDB testing, always close the database using finally clause or fixturesAlways use fixture and test functions, never test classesDependencies (pyproject.toml)
When adding or updating dependencies in `pyproject.toml`, always add a comment explaining why the dependency is needed for this project.
Python Notebooks
Whenever possible, prefer table output over `print()`. Use Pandas DataFrame and notebook's built-in `display()` function to render tabular data.
Command Line Scripts
**Structure**
Use `scripts/erc-4626/scan-vaults.py` as reference for logger setup and environment variable inputAlways use environment variables - do not create command line parsers unless explicitly askedFor tabular output, use `tabulate.tabulate()` function instead of `print()` loops (see `whitelist-vaults.py`)Parallelization and Optimization
**Long-Running Data Pipelines**
Use `joblib.Parallel` to parallelize API reading of multiple entriesUse threading backend unless specified otherwiseSee `lead_scan_core.py` as exampleAll functions using `joblib.Parallel` must take `max_workers` argumentExpose `max_workers` to command line scripts as `MAX_WORKERS` environment variable (see `scripts/erc-4626/scan-vaults.py`)Working with RPC and Blockchain Explorers
**Best Practices**
Prefer blockchain explorers like Etherscan over Python and curl for reading proxy contract addressesPrefer Python snippets over `curl` for reading data from blockchain explorersTo get latest block number, use JSON-RPC URL with Web3.py `web3.eth.block_number` callNever figure out RPC URL yourself - always use environment variables from local environmentStop and ask user if you cannot determine the correct environment variableBuilding Documentation
Documentation uses Sphinx v4.5 and lives in the `docs` folder.
Build documentation:
```shell
source .local-test.env && make build-docs
```
Clean Sphinx autosummaries:
```shell
source .local-test.env && make clean-docs
```
**Important:** Never directly edit auto-generated sphinx files in `_autosummary*` folders.