Data Flows — OL Data Platform

Generated 2026-06-24 13:33 UTC · c4gen dev

Each scenario below replays one interaction as a C4 Dynamic diagram. Amber steps are asynchronous (queued / scheduled / event-driven).

App-database ingest into the raw lake (asynchronous)

The Dagster daemon triggers the per-connection sync_and_stage schedule in the lakehouse location, which materializes the Airbyte assets. Airbyte extracts a SOA app's Postgres/MySQL and lands raw Iceberg tables (carrying airbyte columns) in the lake on S3. Evidenced by OpenMetadata raw tables raw__mitlearn__app__postgres__ and raw__mitxonline__openedx__mysql__*.

dbt transform raw to dimensional marts (asynchronous)

After staging, the lakehouse location runs the dbt project; dbt executes its model DAG in Trino, reading raw and writing staging → intermediate → dimensional/marts in the Iceberg lake. OpenMetadata lineage confirms dim_course_run integrates raw__{micromasters,mitxonline,xpro}__app__postgres, raw__mitx__openedx__mysql, raw__edxorg__s3 and raw__irx__edxorg__bigquery.

Content webhook back into MIT Learn (asynchronous)

Today the platform's reverse path is HMAC-signed content webhooks (not Hightouch — Hightouch lives on the mit-learn side). After an Open edX course export, the openedx location POSTs to MIT Learn's /api/v1/webhooks/content_files/ to trigger ingest of the new content files.

OpenMetadata catalog & lineage harvest (asynchronous)

The data_platform location runs the OpenMetadata metadata workflow against Trino, while the lakehouse location uploads dbt docs artifacts to S3 that OpenMetadata ingests — together producing the cross-system table/column lineage and PII classification seen in the catalog.

TARGET: consolidate app ETL onto the platform

Goal state: heavy ETL currently embedded in each SOA app's Celery workers (MITx Online edX-sync & certificate generation; MIT Learn edx_content catalog/resource ingest) is relocated onto Dagster on this platform, leaving the apps to serve requests. These edges are tagged target and are not yet implemented.

Ingestion sources (ETL)

Every external source the edx_content / default Celery workers pull from, with transport and cadence. ⚠️ marks brittle linkages (HTML/token scrapes, hardcoded URLs).