Skip to footer

kafka - search results

Marmaray: An Open Source Generic Data Ingestion and Dispersal Framework and Library for Apache Hadoop


Connecting users worldwide on our platform all day, every day requires an enormous amount of data management. When you consider the hundreds of operations and data science teams analyzing large sets of anonymous, aggregated data, using a variety of different

Out of the Classroom: A Snapshot of Uber’s Summer 2018 Interns

Uber’s engineering intern program gives college students a chance to find out how their studies can be put to practical use in an industry setting. Working under the guidance of a mentor, interns at Uber embed within a team and

JVM Profiler: An Open Source Tool for Tracing Distributed JVM Applications at Scale

Computing frameworks like Apache Spark have been widely adopted to build large-scale data applications. For Uber, data is at the heart of strategic decision-making and product development. To help us better leverage this data, we manage massive deployments of Spark

Scaling Uber’s Apache Hadoop Distributed File System for Growth


Three years ago, Uber Engineering adopted Hadoop as the storage (HDFS) and compute (YARN) infrastructure for our organization’s big data analysis. This analysis powers our services and enables the delivery of more seamless and reliable user

Queryparser, an Open Source Tool for Parsing and Analyzing SQL

In early 2015, Uber Engineering migrated its business entities from integer identifiers to UUID identifiers as part of an initiative towards using multiple active data centers. To achieve this, our Data Warehouse team was tasked with identifying every foreign-key relationship

Introducing AthenaX, Uber Engineering’s Open Source Streaming Analytics Platform

Uber facilitates seamless and more enjoyable user experiences by channeling data from a variety of real-time sources. These insights range from in-the-moment traffic conditions that provide guidance on trip routes to the Estimated Time of Delivery (ETD) of an UberEATS

Engineering Restaurant Manager, our UberEATS Analytics Dashboard


At Uber, we use data analytics to architect more magical user experiences across our products. Whenever possible, we harness these data engineering capabilities to empower our partners to better serve their customers. For instance, in late 2016, the UberEATS engineering

Meet Michelangelo: Uber’s Machine Learning Platform


Uber Engineering is committed to developing technologies that create seamless, impactful experiences for our customers. We are increasingly investing in artificial intelligence (AI) and machine learning (ML) to fulfill this vision. At Uber, our contribution to this space is Michelangelo,

Deploying More Reliable Apps with Uber Engineering’s XP Background Push


Unlike server-side programming, mobile code cannot be retracted once shipped and can only be updated once a user opts in to an app upgrade. For Uber Engineering, this presents unique challenges when it comes to releasing new features incrementally, fixing

Engineering Uber Predictions in Real Time with ELK


Uber’s services rely on the accuracy of our event prediction and forecasting tools. From estimating rider demand on a given date to predicting

Engineering Data Analytics with Presto and Apache Parquet at Uber


From determining the most convenient rider pickup points to predicting the fastest routes, Uber uses data-driven analytics to create seamless trip experiences. Within engineering, analytics inform decision-making processes across the board. As we expand to new markets, the ability to

Redesigning Uber Engineering’s Mobile Content Delivery Ecosystem


Supporting a quick and efficient in-app communication channel for people who drive with Uber is critical to our business. If we are unable to effectively communicate messages on the app, it can prevent drivers from receiving important information. In 2015,

Powering UberEATS with React Native and Uber Engineering


With UberEATS, our aim is to make ordering food from your favorite restaurants as seamless as requesting a ride with uberX or uberPOOL. Like launching any new product, building out a food delivery network came with its fair share

Evolving Distributed Tracing at Uber Engineering

Distributed tracing is quickly becoming a must-have component in the tools that organizations use to monitor their complex, microservice-based architectures. At Uber Engineering, our open source distributed tracing system Jaeger saw large-scale internal adoption throughout 2016, integrated into hundreds

Cherami: Uber Engineering’s Durable and Scalable Task Queue in Go


Cherami is a distributed, scalable, durable, and highly available message queue system we developed at Uber Engineering to transport asynchronous tasks. We named our task queue after a heroic carrier pigeon with the hope that this system would be just

Designing Euclid to Make Uber Engineering Marketing Savvy

Fast, granular, reliable ROI on ad performance was our bugle call to build Euclid, Uber’s in-house marketing platform. Early this year, Euclid replaced a legacy system, which processed ROI data somewhat manually as it struggled to keep up with Uber’s

The Uber Engineering Tech Stack, Part II: The Edge and Beyond

Uber Engineering

Uber’s mission is transportation as reliable as running water, everywhere, for everyone. Last time, we talked about the foundation that powers Uber Engineering. Now, we’ll explore the parts of the stack that face riders and drivers, starting

The Uber Engineering Tech Stack, Part I: The Foundation

Update: This article discusses the lower half of the stack. For the rest, see Part II: The Edge and Beyond.

Uber Engineering

Uber’s mission is transportation as reliable as running water, everywhere, for everyone. To make that possible, we

Streamific, the Ingestion Service for Hadoop Big Data at Uber Engineering


While Uber moves people and packages around the world, data moves Uber. Systems like Hadoop and Spark power data decisions both large and small in the company. The Uber data engineering team builds big data solutions on top of these