Skip to main content
Backend, Data / ML, Engineering

Enabling Seamless Kafka Async Queuing with Consumer Proxy

August 31, 2021 / Global
Featured image for Enabling Seamless Kafka Async Queuing with Consumer Proxy
Figure 1: Kafka Ecosystem at Uber
Figure 2: Ordered Streaming System
Figure 3: Unordered Queueing System
Figure 4: Non-uniform Processing Latency
Figure 5: Poison Pill Messages
Figure 6: High Level Consumer Proxy Architecture
Figure 7: Parallel Processing within Partitions
Figure 8: Out-of-Order Commit
Figure 9: Dead Letter Queue
Yang Yang

Yang Yang

Yang Yang is a Staff Software Engineer on Uber’s Streaming Data Team. She works on building a highly scalable, reliable Kafka ecosystem at Uber, including uReplicator, Kafka Consumer Proxy, and other internal tooling.

Zhifeng Chen

Zhifeng Chen

Zhifeng Chen is a Senior Software Engineer II on Uber’s Streaming Data Team. He has been working on highly available, fault resilient streaming system, including resilient Kafka Rest Proxy and Kafka Consumer Proxy.

Qichao Chu

Qichao Chu

Qichao Chu is a Senior Software Engineer on Uber’s Streaming Data Team. He has been working on highly available, fault resilient streaming systems, including Kafka Consumer Proxy and Uber internal Kafka client libraries.

Haitao Zhang

Haitao Zhang

Haitao Zhang is a former Software Engineer on Uber’s Streaming Data Team. He has worked on projects and tools to improve the user experience of Kafka as a reliable and scalable async message queue, as well as improve monitoring and alerting of Kafka ecosystem.

George Teo

George Teo

George Teo is a former Senior Software Engineer on Uber’s Streaming Data Team. He's worked on improving the user experience of Kafka as a reliable and scalable async message queue with projects such as Consumer Proxy, Kafka Dead Letter Queue, and Kafka Rest Proxy.

Posted by Yang Yang, Zhifeng Chen, Qichao Chu, Haitao Zhang, George Teo