ML Pipeline Project

A complete machine learning pipeline with data generation, transformation, model training, and inference API.

Project Structure

.
├── config.yaml           # Pipeline configuration
├── requirements.txt      # Python dependencies
├── src/
│   ├── data_ingestion.py    # Data loading
│   ├── data_transformation.py # Feature engineering
│   ├── model_training.py     # Model training & tracking
│   ├── inference_api.py      # FastAPI endpoint
│   ├── pipeline.py           # Main orchestrator
│   └── generate_sample_data.py # Sample data generator
├── docs/
│   └── pipeline_development.md # Development journey and issues
└── tests/               # Test files

Setup Instructions

1. System Requirements

sudo apt update
sudo apt install python3-full python3-dev build-essential python3-venv

2. Virtual Environment Setup

# Create virtual environment
python3 -m venv venv --clear
source venv/bin/activate

# Install dependencies
pip install --break-system-packages --no-cache-dir -r requirements.txt

3. Running the Pipeline

1. Generate sample data:

python3 src/generate_sample_data.py

2. Run the pipeline:

python3 src/pipeline.py

3. Start the inference API:

uvicorn src.inference_api:app --host 0.0.0.0 --port 8000

4. Test the API:

python3 src/test_api.py

Development Workflow

1. Create a new branch for features/fixes
2. Make changes and test locally
3. Commit changes with descriptive messages
4. Create pull request for review
5. Merge after approval

Common Issues

See docs/pipeline_development.md for detailed documentation of issues encountered and their solutions.

Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.