FinETL Development Assistant

Expert guidance for working with the FinETL Python library - a financial data ETL (Extract, Transform, Load) tool for financial market data.

Project Context

FinETL is a Python library that implements ETL patterns for financial data processing. It leverages yfinance for extracting financial market data and HuggingFace datasets for data handling and storage.

Development Workflow

When working on this project, follow these steps:

1. **Environment Setup**

- Ensure Python 3.12+ is installed

- Run `poetry install` to install all dependencies

- The project uses Poetry for dependency management

2. **Understanding the Architecture**

- The codebase follows a standard ETL pattern with four main modules in `src/finetl/`:

- `extraction/` - Handles data extraction from financial sources (primarily yfinance)

- `transformation/` - Performs data cleaning, validation, and processing

- `loading/` - Manages loading data to various destinations

- `pipeline.py` - Orchestrates complete ETL workflows by coordinating the above modules

3. **Making Code Changes**

- Always read existing code before proposing modifications

- Follow the established ETL pattern when adding new functionality

- Maintain separation of concerns between extraction, transformation, and loading

- Keep pipeline orchestration logic in `pipeline.py`

4. **Adding Dependencies**

- Use `poetry add <package>` to add new dependencies

- Document why the dependency is needed

- Ensure compatibility with Python 3.12+

5. **Testing**

- Run tests with `poetry run pytest`

- Write tests for new functionality

- Ensure tests pass before finalizing changes

6. **Key Libraries**

- **yfinance** - Primary source for financial market data extraction

- **datasets** (HuggingFace) - Data handling, storage, and versioning

- **Poetry** - Dependency and environment management

- **pytest** - Testing framework

Implementation Guidelines

Common Tasks

Important Notes