profile view → kafka → feed, ads, search, analytics
LinkedIn built Kafka around 2010 because every system needed the same activity data, and wiring them together one by one had become a mess. It became open source in 2011.
District 2 · Module 2.3 · about 8 minutes
Why Kafka?
You will feel the problem first, try the obvious fix, watch it fail, and only then meet Kafka. Tap any for a plain-language card.
The pain
Your checkout waits for everyone.
A customer taps Place order. Your checkout service then calls five other services, one after another, and waits for each reply before telling the customer “done”.
That works on a good day. Now make the email service slow, or take it down, and press the button again.
Notice: the customer does not care about the email. Why are they waiting for it?
Customer waited0 ms
Order result–
Press Place order to send one order through the system.
The obvious fix
“Just put a queue in the middle.”
Good instinct. Checkout drops an “order placed” note in a queue and replies to the customer instantly. Other services pick notes up later.
But a classic queue hands each note to one reader, then deletes it. Analytics and Fraud both need every order. Send some orders and count.
Then switch to Kafka log mode and send again.
Analytics saw0
Fraud check saw0
Simple queue: each order goes to one reader, then it is deleted.
Meet Kafka
A notebook that never forgets.
Kafka is a giant append-only notebook. A writes each event at the end of a . The topic is split into so many machines can share the work. Each event gets a number called an .
Readers live in . Every group gets every event and keeps its own bookmark, so a slow group falls behind (that gap is called ) without slowing anyone else. Partitions live on servers called .
Events written0
Email lag0
Analytics lag0
Partitions3
Group: email-service
- reader 1 → P0, P2
- reader 2 → P1
Group: analytics
- reader 1 → P0, P1, P2
Try this: push orders to 30 per second and watch email lag grow. Then add email readers. What happens past 3 readers?
Who uses it
Companies you use every day.
Kafka was built for one company's pain and then given away. Today it moves events inside most large tech companies. See the list on Apache Kafka's Powered By page.
driver location → kafka → matching, pricing, ETAs
Trips and driver locations are streams of events. Many teams read the same stream for matching, surge pricing and analytics, each at their own speed.
play, pause, error → kafka → recommendations, monitoring
Netflix's Keystone data pipeline uses Kafka to carry viewing and operational events from services to the systems that process them.
Interview angle
What interviewers actually ask.
Interviewers rarely ask “what is Kafka”. They ask why you chose it, and what breaks. Three questions, each worth 20 XP.
Say the trade-off out loud. “Kafka, because three teams need the same events and we may replay them” beats “Kafka is fast”.
Question 1 / 3
Email and analytics both need every order event. Analytics may be offline for an hour and must catch up. Which fits best?
Tip: press 1 to 3 to answer, Enter for the next one.
Break It
Pick a bad key. Watch one partition melt.
Kafka decides which partition gets an event by its . Same key, same partition, always in order.
It is the Big Billion Days sale and one seller has a flash deal. 32 orders per second arrive. Each partition's reader handles 10 per second. Choose a key and run the sale.
Worst lag0
Hottest partitionnone
Bars show waiting events per partition. Pick a key, then run the sale.