Modern data platform
Context
Data scattered across spreadsheets, production databases and manual exports. Every report is rebuilt from scratch and nobody can audit where a number came from.
01 · Sources
- Transactional database
- CRM and ERP
- Third-party APIs
- Product events
02 · Ingestion
- CDC for databases
- Scheduled batch
- Streaming for events
03 · Layers
- Raw: faithful copy of source
- Cleansed: typed and validated
- Serving: modeled by domain
04 · Transformation
- Versioned models
- Contract tests
- Traceable lineage
05 · Consumption
- BI and dashboards
- Reverse ETL to CRM
- Features for models
Design decisions
- The raw layer is never mutated, and that is what lets you reprocess when a rule changes.
- Every published model has a contract test: it breaks the pipeline before it breaks the dashboard.
- Nobody queries the raw layer directly. Needing to is a sign a model is missing.