CI/CD Pipelines

Active

GitHub Actions workflows with path filters for multi-project repo

Overview

Explain how you handle CI/CD across a monorepo with multiple data engineering projects.

Architecture

flowchart LR
    A[Git Push] --> B{Path Filter}
    B -->|TMDB-ELT/**| C[TMDB Tests<br/>& Deploy]
    B -->|clickstream/**| D[Clickstream<br/>Tests & Deploy]
    B -->|orchestration/**| E[DAG<br/>Validation]

    style A fill:#FAECE7,stroke:#D85A30,color:#712B13
    style B fill:#EEEDFE,stroke:#534AB7,color:#3C3489
    style C fill:#E6F1FB,stroke:#185FA5,color:#0C447C
    style D fill:#E6F1FB,stroke:#185FA5,color:#0C447C
    style E fill:#E1F5EE,stroke:#1D9E75,color:#085041

Workflow Examples

# Example: path-filtered workflow
name: TMDB Pipeline CI
on:
  push:
    paths:
      - 'TMDB-ELT/**'
  pull_request:
    paths:
      - 'TMDB-ELT/**'

Lessons Learned

---

View on GitHub