Data Pipelines: Batch, Streaming & CDC
ETL vs ELT, batch vs stream, lambda vs kappa, windows, event time and watermarks, exactly-once checkpoints, CDC with Debezium, lakes vs warehouses, real-time dashboards and reconciliation.
About 115 minutes · 17 exam questions · 9 interview questions · 2 code labs
What you will learn
- The pipeline landscape: sources, sinks, ETL vs ELT
- Batch processing: MapReduce, Spark and idempotent reruns
- Stream processing: Flink, Kafka Streams and stateful operators
- Lambda vs Kappa architecture
- Windowing: tumbling, hopping, sliding and session windows
- Event time vs processing time, watermarks and late data
- Exactly-once in streaming: checkpoints, 2PC sinks and idempotent writes
- Change Data Capture with Debezium (and the outbox pattern)
- Storage: data lake vs warehouse vs lakehouse
- Real-time dashboards: pre-aggregation and OLAP serving
- Reconciliation, data quality and the pipeline scenario playbook
- Column-oriented storage, materialized views and data cubes