Efe Kaya

I’m a software engineer working on large-scale compute infrastructure and I work on Kubernetes fleets, cluster management, and custom controllers. I started with a fascination for quantum computing but it over time evolved into a deep interest in distributed systems and how we run them reliably at scale.

Thoughts

How Big Companies Actually Use Kafka

A short breakdown of how The New York Times, Pinterest, and a large enterprise customer run Kafka in production, and what their architectures teach us about topics, clusters, storage, and cost.

Back

How Big Companies Actually Use Kafka

The New York Times: One Topic for All Content Since 1850

The New York Times uses Kafka end-to-end to publish its content. Without any extra database in the system, all articles going back to the 1850s are stored chronologically in a single topic.

Materialized views and KTables are exposed using Kafka Streams. Because there are different schemas, different teams, and different data formats, they didn't build any separate API or microservice layer just for publishing. All services consume their data and assets directly from Kafka; what they do with that data afterwards and where they store it is up to them.

When a new article or piece of content is published, it's pushed to that same topic in Kafka and becomes accessible to every service at the same time.

NYT Kafka Architecture

Figure 1: The new New York Times log/Kafka-based publishing architecture.