Netflix's data platform, end to end

Same companyHow it works (system design) →
0/7
☰
Netflix · data engineering

Every play, pause and scroll on Netflix becomes an event; a few minutes later it's in a table that decides experiments, recommendations and the finance report.

+ The whole system in one paragraph

Netflix's data starts as events from apps and microservices and as changes in its databases. Kafka carries it (the Keystone pipeline, now Netflix's platform called Data Mesh — not the data-mesh org concept); Flink routes it in real time to search and to the warehouse; the warehouse is Apache Iceberg tables on S3 — a format Netflix created. Spark jobs, scheduled by Netflix's orchestrator Maestro, turn raw events into the tables that A/B tests, recommendations and finance read. Before a table is published, it's audited.

~500B
events a day into the pipeline (2016)[2]
700B+
Kafka messages on an average day, 36 clusters (2016)[3]
100s of PB
data warehouse in S3 (2020)[16]

In this lesson

7 chapters · ~16 min · a quick check in each

Everything is on this page — just scroll down ↓

Chapter 1 of 7

The whole system

1
Requirements

Who needs the data — and how fresh

Data engineering starts with the readers, not the pipes. At Netflix four kinds set the rules — experiments, personalization, content analysts, finance — each needing a different freshness.

ⓘ Details & sources

Yellow = who reads it. Blue = how fresh or how correct it must be.[10][12][13][21]

A/B tests: everyproduct change istested; each membersits in one group (a“cell”) per test, inmany tests at once(2016).Personalization:viewing data drivesbookmarks andrecommendations(2018).Creative insights(Muse): analyticsover trillion-rowdata (2025).Finance: membershiptables feedNetflix's financialreports (2023).Real-time paths:under a minute(2016).Batch pipelines:hourly, intraday ordaily.Finance waits a fewhours on purpose:room forlate-arriving data.Analysts query anytable with Spark orPresto (2019).
2
Architecture

Netflix's data platform, on one page

Every piece of the data platform and how data moves between them. Streaming runs along the top in seconds; batch runs underneath on a schedule.

ⓘ Details & sources

Play a flow: a play event, a database change, the hourly table, an A/B test.

Watch it happen:
Streaming — secondsBatch — hourly / daily12334455Apps + services: Every app and microservice reports what happened as an event.Apps + serviceseventsService DBs: The databases behind each service; their changes are data too.Service DBsMySQL, Postgres…CDC (DBLog): Copies every row change out of a database without slowing it down.CDC (DBLog)reads the logKafka: The front door: a log every event is written to first.KafkaKeystone / Data MeshFlink jobs: Small streaming jobs that filter events and send them where they belong.Flink jobsroute in real timeMaestro: Netflix's orchestrator: runs each batch job when its inputs are ready.Maestrothe schedulerSpark jobs: The big batch jobs that turn raw events into clean tables.Spark jobsbatch ETLElasticsearch: An index for searching events seconds after they happen.Elasticsearchsearch, logsIceberg on S3: The data warehouse: tables of files in S3, in a format Netflix created.Iceberg on S3the warehouseMetacat: Knows every table: where it lives, its columns, its owner and size.Metacatthe table catalogDruid: A store built for fast dashboard queries over huge tables.Druidfast analyticsA/B tests: Every product change is tested; the results come from these tables.A/B testsexperimentsRecs: Viewing data drives your bookmarks and recommendations.RecspersonalizationFinance: Membership tables feed Netflix's financial reports.FinancereportsMuse: Dashboards that show how artwork and trailers perform.Musecreative insights
clientserviceworkerdatabasequeuestorageexternal dashed border = simplified model
12345
Step 1 of 5

You press play. The app sends a play event, and it lands in Kafka first.

3
Freshness

From “you pressed play” to “it's in a table”

The same event reaches readers at very different times: real-time ones in under a minute; in 2016 the batch path landed it within 10 minutes; tables are rebuilt hourly or daily.

ⓘ Details & sources

Times are Netflix's own targets from 2016 (pipeline) and 2023 (membership batch).[2][13]

0 sYou press playthe app sends an eventsecondsIn Kafkathe front door of the pipeline< 1 minReal-time readersabout 30% of events branched there (2016)real-time = sub-minute≤ 10 minLanded in S3the 2016 batch pathhourly / dailyWarehouse tablesrebuilt by scheduled jobsa few hours lateFinance tableswaits for late data
Quick check

Why do Netflix's finance tables arrive a few hours late — on purpose?

Pick one — or just keep scrolling, it's not graded.

Keep going — it's free

Create your free account to unlock the rest of Netflix's system design, keep your progress and pick up right where you left off.

  1. 2Sources and volume
  2. 3The pipeline, end to end
  3. 4The data model and storage
  4. 5Orchestration
  5. 6What the data feeds
  6. 7Quality, monitoring and alerts
Create my free account →I already have one — log in
Questions & comments

Ask anything about this system, or share how you'd design it.

Log in to join the conversation