Skip to footer

Uber Case Study: Choosing the Right HDFS File Format for Your Apache Spark Jobs

Uber's Maps Collection and Reporting (MapCARs) team shares best practices when choosing which HDFS file formats are optimal for use with Apache Spark.

Solving Big Data Challenges with Data Science at Uber

server racks
How engineers and data scientists at Uber came together to come up with a means of partially replicating Vertica clusters to better scale our data volume.

Accessible Machine Learning through Data Workflow Management

Complex freeway interchange
Uber engineers offer two common use cases showing how we orchestrate machine learning model training in our data workflow engine.

Announcing the Uber Open Summit Sofia 2019

Sunset over Temple St. Cyril and Methodius in Sofia. Bulgaria
The next Uber Open Summit, where we present our latest and most impactful open source projects, will be in Sofia, Bulgaria on April 20, 2019.

DBEvents: A Standardized Framework for Efficiently Ingesting Data into Uber’s Apache Hadoop Data Lake

Elephant silhouette
Uber engineers discuss the development of DBEvents, a change data capture system designed for high data quality and freshness that is capable of operating on a global scale.

Data Science at Scale: A Conversation with Uber’s Fran Bell

We spoke to Data Science Director Fran Bell about machine learning at Uber and what she finds most challenging—and rewarding—about her work.

Mitigating Risk in a Three-Sided Marketplace: A Conversation with Trupti Natu and Neel Mouleeswaran on the Uber Eats Risk Team

We sat down with a risk strategy manager and a risk engineer to discuss how they build solutions to minimize risk in the Uber Eats three-sided marketplace.

Open Sourcing Peloton, Uber’s Unified Resource Scheduler

First introduced by Uber in November 2018, Peloton manages resources across large-scale, distinct workloads, combining separate compute clusters.

Using Machine Learning to Ensure the Capacity Safety of Individual Microservices

Uber leveraged machine learning to design our capacity safety forecasting tooling with a special emphasis on calculating a quality of reliability score.

Introducing Kraken, an Open Source Peer-to-Peer Docker Registry

Developed by Uber, Kraken is an open source peer-to-peer Docker registry capable of distributing terabytes of data in seconds.

The Uber Engineering Internship Experience: European Edition

Uber's European interns
Engineering interns from Uber's European offices talk about their experiences, including the projects they worked on, the people they worked with, and the social activities they engaged in.

Architecting a Safe, Scalable, and Server-Driven Platform for Driver Preferences with RIBs

Subway platform
Our driver app's new server-driven preferences section enables driver-partners to customize their experiences to make the app better fit into their lives.

Managing Uber’s Data Workflows at Scale

In this article, we discuss Uber's journey toward a unified, multi-tenant, and scalable data workflow management system.

Profiles in Coding: Tatiana Romanova, Uber Payments Platform, Amsterdam

Amsterdam
Uber site reliability engineer Tatiana Romanova, based in our Amsterdam engineering office, discusses her computer science background, her journey to Uber, and her work maintaining our Payments Platform.

Creating Custom Plugins with Fusion.js, Uber’s Open Source Web Framework

In addition to providing official plugins, Fusion.js enables developers to build and integrate their own plugins by leveraging dependency injection.

Horovod Adds Support for PySpark and Apache MXNet and Additional Features for Faster Training

Horovod adds support for more frameworks in the latest release and introduces new features to improve versatility and productivity.

Pyro Accepted by the LF Deep Learning Foundation as a Hosted Project

Created by Uber in 2017, Pyro was voted in by the Linux Foundation Deep Learning Technical Board as the latest incubation project to join its foundation.

Uber Open Source: Catching Up with Fritz Obermeyer and Noah Goodman from the Pyro Team

We spoke with Fritz Obermeyer and Noah Goodman, Pyro project co-leads, about the potential of open source AI software at Uber and beyond.

Meet Uber Sofia’s Women in Tech

Uber Sofia is also home to a vibrant community of women technologists, from software engineers and data analysts to designers and program managers.

Introducing AVS, an Open Standard for Autonomous Vehicle Visualization from Uber

Rendered image of street and vehicles
Uber announces the release of the Autonomous Visualization System (AVS) as an open source project. AVS is a standard for creating a visual environment based on sensor data from autonomous vehicles, with playback available in multiple formats, including the web and video.

Modeling Censored Time-to-Event Data Using Pyro, an Open Source Probabilistic Programming Language

Censored time-to-event data is critical to the proper modeling and understanding of customer engagement on the Uber platform. In this article, we demonstrate an easier way to model this data using Pyro.

Engineering Uber Beacon: Matching Riders and Drivers in 24-bit RGB Colors

Uber Beacon app screen
In our ongoing series about rewriting the Uber driver app, engineer Kevin Babcock explains how we built the connection between the app and the Uber Beacon device, which displays a color remotely selected through a rider's app.

First Uber Science Symposium: Discussing the Next Generation of RL, NLP, ConvAI, and DL

The Uber Science Symposium featured talks from members of the broader scientific community about the the latest innovations in RL, NLP, and other fields.

Introducing Ludwig, a Code-Free Deep Learning Toolbox

Uber AI developed Ludwig, a code-free deep learning toolbox, to make deep learning more accessible to non-experts and enable faster model iteration cycles.

Why Financial Planning is Exciting… At Least for a Data Scientist

Model flow showing rider and driver sign-ups
In this article, Uber’s Marianne Borzic Ducournau discusses why financial planning at Uber presents unique and challenging opportunities for data scientists.

Building Locally, Scaling Globally: Meet the Tech Team at Uber New York City

Ever wondered what it’s like to work in tech at Uber New York City? Just blocks from Times Square and Bryant Park, Uber’s new office in midtown Manhattan is home to more than a dozen teams, hundreds of employees (and growing), and a wide variety of engineering roles.

Increasing Representation at Uber through The Hidden Genius Project

Collage showing members and mentors for the Career Prep Program
In its first year, the Career Prep Program, a collaboration between Uber and The Hidden Genius Project, demonstrated how technology-focused companies can embrace and reinforce values of diversity and inclusion among engineers, while having a positive impact in the community.

Introducing AresDB: Uber’s GPU-Powered Open Source, Real-time Analytics Engine

AresDB, Uber's open source real-time analytics engine, leverages GPUs to enable real-time computation and data processing in parallel.

How Uber Leverages Applied Behavioral Science at Scale

Uber Labs utilizes insights and methodologies from behavioral science to build programs and products that are intuitive and enjoyable for users on our platform.

Expanding Access: Engineering Uber Lite

Uber Lite pickup screens
Many people around the world use Android phones based on hardware developed in 2015 and earlier. Uber engineers explain how they developed a lightweight rider app to serve this global audience.

Aarhus Engineering Internship: Building Aggregation Support for YQL, Uber’s Graph Query Language for Grail

Uber intern Lau Skorstengaard shares his experience working on YQL, the graph query language for our in-house infrastructure state aggregation platform.

Manifold: A Model-Agnostic Visual Debugging Tool for Machine Learning at Uber

Uber built Manifold, a model-agnostic visualization tool for ML performance diagnosis and model debugging, to facilitate a more informed and actionable model iteration process.

Building a Scalable and Reliable Map Interface for Drivers

Uber driver app screen
In our ongoing series about rewriting the Uber driver app, engineer Chris Haugli explains how we designed the map display to be resilient, and always show the most useful information.

Creating a Zoo of Atari-Playing Agents to Catalyze the Understanding of Deep Reinforcement Learning

Uber AI Labs releases Atari Model Zoo, an open source repository of both trained Atari Learning Environment agents and tools to better understand them.

POET: Endlessly Generating Increasingly Complex and Diverse Learning Environments and their Solutions through the Paired Open-Ended Trailblazer

Uber AI Labs introduces the Paired Open-Ended Trailblazer (POET), an algorithm that leverages open-endedness to push the bounds of machine learning.

From Self-Driving Cars to Optimizing Claims Efficiency: My Unconventional Journey to Insurance Engineering

In this article, engineering manager Lili Kan reflects on her decision to lead Uber's Insurance Engineering team and discusses the challenges—and opportunities—of building insurance products for our platform.

How to Ship an App Rewrite Without Risking Your Entire Business

How to Ship an App Rewrite Without Risking Your Entire Business
Rather than shipping out our new driver app as a simple update to Android phones, Uber engineers delivered a dual binary package, enabling a safe and structured rollout of the new app while maintaining support for the previous version.

Women in Data Science at Uber: Moving the World With Data

During an October 2018 meetup, members of our Women in Statistics, Data, Optimization, and Machine Learning (WiSDOM) group presented on their technical work at Uber.

Profiles in Coding: Sylvain Francois, Uber Freight

Uber Freight truck driving down freeway
For Uber's Profiles in Coding series, we interview Uber Freight engineer Sylvain Francois to find out the nature of his daily work and his best tips for coders.

How Uber Beacon Helps Improve Safety for Riders and Drivers

The Uber Beacon leverages visual signaling, an accelerometer, and a gyroscope to improve the accuracy of in-app safety products like our automatic crash detection feature.

Year in Review: 2018 Highlights from the Uber Engineering Blog

Our editors spotlight some of the year's most popular articles, from an overview of our Big Data platform to a first-person account of an engineer's immigrant journey.

Year in Review: 2018 Highlights from Uber Open Source

Brian Hsieh, Uber's Open Source program lead, reflects on open source accomplishments, project launches, and collaborations in 2018.

Four Ways Uber Visualization Made an Impact in 2018

Uber's Head of Urban Computing & Visualization reflects on his team's work visualizing data to better understand urban mobility in 2018—and beyond.

Interning at Uber: Building the Uber Eats Menu Scheduler

Jonathan Levi recounts his experience as an intern at Uber during Summer 2018, including building a useful project for the Uber Eats team.

Horovod Joins the LF Deep Learning Foundation as its Newest Project

Horovod, Uber's distributed training framework, joins the LF Deep Learning Foundation to help advance open source innovation in AI, ML, and deep learning.

Open Source at Uber: Meet Alex Sergeev, Horovod Project Lead

We sat down with Horovod project lead, Alex Sergeev, to discuss his path to open source and what most excites him about the future of Uber's distributed deep learning framework.

Scaling Cash Payments in Uber Eats

Scaling Cash Payments in Uber Eats - feature_image
Uber's new driver app leverages its offline mode along with a cash-drop system organized around restaurants so that Uber Eats customers can pay for deliveries with cash.

Faster Neural Networks Straight from JPEG

Uber AI Labs introduces a method for making neural networks that process images faster and more accurately by leveraging JPEG representations.

The Billion Data Point Challenge: Building a Query Engine for High Cardinality Time Series Data

Part of Uber's open source M3 metrics system, our query engine can support real-time, large-scale computation and multiple query languages.

Introducing Makisu: Uber’s Fast, Reliable Docker Image Builder for Apache Mesos and Kubernetes

Uber built Makisu, our open source Docker image builder, to enable the quick, reliable generation of Dockerfiles in Mesos and Kubernetes ecosystems.

Popular Articles