Skip to footer

Open Source

Improving HDFS I/O Utilization for Efficiency

Scaling our data infrastructure with lower hardware costs while maintaining high performance and service reliability has been no easy feat. To accommodate the exponential...

Building Uber’s Fulfillment Platform for Planet-Scale using Google Cloud Spanner

  Introduction The Fulfillment Platform is a foundational Uber domain that enables the rapid scaling of new verticals. The platform handles billions of database transactions each...

Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot

Uber recently launched a new capability: Ads on UberEats. With this new ability came new challenges that needed to be solved at Uber, such...

Streaming Real-Time Analytics with Redis, AWS Fargate, and Dash Framework

Introduction Uber’s GSS (Global Scaled Solutions) team runs scaled programs for diverse products and businesses, including but not limited to Eats, Rides, and Freight. The...

Enabling Seamless Kafka Async Queuing with Consumer Proxy

Uber has one of the largest deployments of Apache Kafka in the world, processing trillions of messages and multiple petabytes of data per day....

Building Scalable Streaming Pipelines for Near Real-Time Features

Background Uber is committed to providing reliable services to customers across our global markets. To achieve this, we heavily rely on machine learning (ML) to...

Pinot Real-Time Ingestion with Cloud Segment Storage

Introduction Apache Pinot is an open source data analytics engine (OLAP), which allows users to query data ingested from as recently as a few seconds...

Containerizing Apache Hadoop Infrastructure at Uber

Introduction As Uber’s business grew, we scaled our Apache Hadoop (referred to as ‘Hadoop’ in this article) deployment to 21000+ hosts in 5 years, to...

‘Orders Near You’ and User-Facing Analytics on Real-Time Geospatial Data

Introduction By its nature, Uber’s business is highly real-time and contingent upon geospatial data. PBs of data are continuously being collected from our drivers, riders,...

Analyzing Customer Issues to Improve User Experience

Introduction The primary goal for customer support is to ensure users’ issues are addressed and resolved in a timely and effective manner. The kind of...

Customer Support Automation Platform at Uber

High Level Overview of the Problem Introduction If you’ve used any online/digital service, chances are that you are familiar with what a typical customer service experience...

Tuning Model Performance

Introduction Uber uses machine learning (ML) models to power critical business decisions. An ML model goes through many experiment iterations before making it to production....

Elastic Distributed Training with XGBoost on Ray

Introduction Since we productionized distributed XGBoost on Apache Spark™ at Uber in 2017, XGBoost has powered a wide spectrum of machine learning (ML) use cases...

Continuous Integration and Deployment for Machine Learning Online Serving and Models

Introduction At Uber, we have witnessed a significant increase in machine learning adoption across various organizations and use-cases over the last few years. Our machine...

Introducing Orbit, An Open Source Package for Time Series Inference and Forecasting

Orbit is a general interface for Bayesian time series modeling. The goal of Orbit development team is to create a tool that is easy...

Optimal Feature Discovery: Better, Leaner Machine Learning Models Through Information Theory

Introduction  Suppose you own a production ML model that already works reasonably well. You know that adding relevant and diverse sources of signal to your...

Freight Pricing with a Controlled Markov Decision Process

Intro Uber Freight was launched in 2017 to revolutionize the business of matching shippers and carriers in the huge and inefficient freight trucking industry (around...

Elastic Deep Learning with Horovod on Ray

Introduction In 2017, we introduced Horovod, an open source framework for scaling deep learning training across hundreds of GPUs in parallel.  At the time, most...

Horovod v0.21: Optimizing Network Utilization with Local Gradient Aggregation and Grouped Allreduce

We originally open-sourced Horovod in 2017, and since then it has grown to become the standard solution in industry for scaling deep learning training...

Revolutionizing Money Movements at Scale with Strong Data Consistency

Uber as a platform invites its users to leverage it, earn from it, and be delighted by it. Serving more than 18 million requests...

Popular Articles