waitingforcode

Apache Spark shuffle writers: SortShuffleWriter

Apache Airflow behavior for TriggerRule.ALL_DONE task

Apache Spark SQL and corrupted files management

Wildcard path and partition values in Apache Spark SQL

Apache Spark Structured Streaming - mapGroupsWithState with and without state removal

Apache Spark Structured Streaming, watermark and window processing

JIT compilation and Apache Spark SQL

Apache Spark 3 and Dynamic Partition Pruning

PySpark and serializers

Apache Flink and temporal table joinb

Delta Lake, commits and conflicts management

Lesson 03: Who is a data engineer?

Apache Airflow and DAG triggered with datetime.now()

Lesson 01: Plan

Lesson 08: Data organization

Lesson 02: Data engineering and archeology

Apache Spark 3.0 and reuse subquery optimization in the Adaptive Query Execution

Apache Spark shuffle writers: SortShuffleWriter

Apache Spark Structured Streaming 3.2.0 features - Apache Kafka

Apache Spark 3.1.1 - table API in Structured Streaming

Crypto-shredding - encryption and decryption for Java, Python and SQL

Apache Airflow data sensor example

Apache Spark on Kubernetes demo

Azure Data Factory controls flows mapped to Apache Airflow DAGs

Apache Spark Structured Streaming and broadcast join internals

Structured Streaming and Apache Kafka source - maxOffsetsPerTrigger impact on reprocessing, part 1

PySpark one-column DataFrame schema inference

Module 2: Data ingestion offline - plan

Scala - ExecutionContext impact on the program execution

Apache Spark Structured Streaming and broadcast variable

Writing data to multiple Apache Kafka topics from Apache Spark

Structured Streaming and Apache Kafka source - maxOffsetsPerTrigger impact on reprocessing, part 2

Apache Spark Structured Streaming and left outer join bug fix

Apache Spark 3.1 - left semi join support in Structured Streaming

Apache Spark SQL and the difference between .select() and .drop()

Master stream processing - Stateful stream processing module

Dataset observation in Apache Spark SQL

Delta Lake and the VACUUM command

Apache Kafka headers in Apache Spark Structured Streaming source

Delta Lake and the orphan files

Row-level runtime filters in Apache Spark 3.3.0

Coalesce and data locality

Internal and external tables in Apache Spark

Lesson 06: Stream processing

Azure Durable Functions

Lesson 11: Data processing - distributed

Apache Spark SQL and broadcast join internals

Apache Spark 3.1.1 - predicate pushdown in JSON data source

Apache Spark shuffle writers: BypassMergesortShuffleWriter

Apache Kafka transactional producer with Apache Spark Structured Streaming

Lesson 04: Data pipelines