Skip to footer

Making Apache Spark Effortless for All of Uber

Chinese Water Dragon photo by InspiredImages/Pixabay
Uber engineers created uSCS, a Spark-as-a-Service solution that helps manage Apache Spark jobs throughout large organizations.

Introducing the Plato Research Dialogue System: A Flexible Conversational AI Platform

The Plato Research Dialogue System enables experts and non-experts alike to quickly build, train, and deploy conversational AI agents.

MoneyCon ’19: Uber Hosts its First FinTech Engineering Conference

Uber's MoneyCon brought together industry leaders to discuss the latest technologies and key learnings in the payments and finance engineering space.

Visualizing City Cores with H3, Uber’s Open Source Geospatial Indexing System

In a selection of presentations delivered at a June 2019 Uber meetup, we discuss how to use H3, our open source hexagonal indexing system, to facilitate the granular mining of large geospatial data sets.

Solving for Urban Air Travel: A Q&A with François Sillion, Director of Uber ATCP

As head of Uber's Advanced Technologies Center in Paris, Francois Sillion and his team are responsible for supporting the R&D behind Uber Air, our effort to add a third dimension to our platform using flying vehicles.

Gaining Insights in a Simulated Marketplace with Machine Learning at Uber

Uber's Marketplace simulation platform leverages ML to rapidly prototype and test new product features and hypotheses in a risk-free environment.

Using Causal Inference to Improve the Uber User Experience

Uber Labs leverages causal inference, a statistical method for better understanding the cause of experiment results, to improve our products and operations analysis.

No Coding Required: Training Models with Ludwig, Uber’s Open Source Deep Learning Toolbox

Uber AI's Piero Molino discusses Ludwig's origin story, common use cases, and how others can get started with this powerful deep learning framework built on top of TensorFlow.

Setting the Pace: A Q&A with Jennifer Anderson, Senior Director of Engineering at Uber

Bridge at night with cars
Jennifer Anderson, a veteran of Silicon Valley technology companies, leads Uber's Product Platform organization, which hosts our core services. In this interview, she describes her organization and the lessons she has learned.

Learnings in Web Development: Design Patterns, Elm, and Progressive Enhancement

Uber Destination:Web logo
Uber's Destination:Web meetup series gives great insight about the most current web building tools and techniques. These three videos from Uber presenters offer tips on a mysterious design pattern, the Elm language, and Progressive Enhancement.

Power On: Accelerating Uber’s Self-Driving Vehicle Development with Data

A key challenge faced by self-driving vehicles comes during interactions with pedestrians. In our development of self-driving vehicles, the Data Engineering and Data Science teams at Uber ATG (Advanced Technologies Group) contribute to the data processing and analysis that help make these interactions safe.

Second Uber Science Symposium: Showcasing Developments in Programming Systems and Tools

On May 3, 2019, Uber’s Programming Systems Team hosted the Programming Systems and Tools Track of the company’s Second Uber Science Symposium, featuring a full day of talks by leading researchers and practitioners in the the field.

Activity/Service as a Dependency: Rethinking Android Architecture for the Uber Driver App

Uber driver app and RIB tree
Architecting the Uber driver app for Android, which needs to run for extended periods of time in the background, involved a unique idea where Activities and Services were not included in the structural foundations.

Uber Open Source: Catching Up with Felix Cheung, Data Platform Engineering Manager

Seattle skyline
Uber Engineering Manager and open source software community member Felix Cheung talks about his work with the Apache Software Foundation, open source at Uber, and XGBoost, a machine learning library for optimized distributed gradient boosting.

Second Uber Science Symposium: Exploring Advances in Behavioral Science

On May 3, 2019, Uber’s Applied Behavioral Science team hosted the Behavioral Science Track of the Second Uber Science Symposium, featuring a full day of presentations delivered by leading researchers in the field.

Visualizing Traffic Safety with Uber Movement Data and Kepler.gl

Map of dangerous traffic in NYC
Learn how to use Kepler.gl for data visualization through our tutorial, where we show how easy it is to load multiple datasets into Kepler.gl to visualize traffic safety in Manhattan.

Employing QUIC Protocol to Optimize Uber’s App Performance

Implementing QUIC protocol against TCP over cellular networks on our apps led to a reduction of 10-30 percent in tail-end latencies for HTTP traffic.

Uber Engineering Celebrates Take Your Kids to Work Day

In this article, we share highlights from Uber’s annual Take Your Kids to Work Day celebration, an opportunity for the children of Uber parents to get a taste of what it’s like to work at a technology company.

Uber Joins Urban Computing Foundation, Contributes Kepler.gl as Organization’s First Hosted Project

In addition to joining the Urban Computing Foundation, Uber is contributing Kepler.gl, an open source geospatial analysis tool, as the organization's first hosted project.

Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

Uber builds upon the Lottery Ticket Hypothesis by proposing explanations behind these mechanisms and deriving a surprising by-product: the Supermask.

Introducing the Uber Research Publications Site

Uber's Chief Scientist announces the launch of the Uber Research Publications Site, a portal for showcasing our contributions to the research community.

Measuring Kotlin Build Performance at Uber

To determine if it made sense to adopt Kotlin for our Android monorepo, Uber's Mobile Engineering team measured Kotlin build performance at scale across a variety of project structures.

Improving Uber’s Mapping Accuracy with CatchME

CatchMapError (CatchMe) is a system that automatically catches errors in Uber's map data with anonymized GPS traces from the driver app.

Consistent Data Partitioning through Global Indexing for Large Apache Hadoop Tables at Uber

elephant
Performing updates of individual records in Uber's over 100 petabyte Apache Hadoop data lake required building Global Index, a component that manages data bookkeeping and lookups at scale.

Uber Submits Hudi, an Open Source Big Data Library, to The Apache Software Foundation

We submitted Hudi to the Apache Incubator to ensure the long-term growth and sustainability of the project under The Apache Software Foundation.

Optimizing M3: How Uber Halved Our Metrics Ingestion Latency by (Briefly) Forking the Go Compiler

Latency graph
Noticing increased latency in our metrics platform, Uber engineers track down a bug related to stack growth in a goroutine, resulting in a fix elevated to the Go open source GitHub repository.

Introducing Base Web, Uber’s New Design System for Building Websites in React

Base Web is a React component library which implements the Base design language to act as a device-agnostic foundation for easily creating web applications.

Learning on the Go: Engineering Efficiency with Concise Documentation

People crossing a street
Uber Technical Writer Shannon Brown offers three tips for creating concise, meaningful documentation.

Uber Open Source: Catching Up with Celina Ward, M3 Observability Engineer

Celina Ward discusses her journey to engineering, what it was like to present at Kubecon 2018, and what’s next for M3, Uber's open source metrics platform.

Bridging the Gap from Sofia to San Francisco: A Conversation with Engineering Manager Marin Dimitrov

Sofia, Bulgaria
Uber Engineering Manager Marin Dimitrov shares his best practices for motivating an engineering team, keeping connected when your headquarters office is over 6,000 miles and 10 hours away, and leveraging open source software.

Scaling Mobile Device Management for macOS with Chef at Uber

Uber's IT Engineering team scaled mobile device management on macOS by leveraging open source tools and custom API-driven Chef cookbooks.

Build a ‘Hello World!’ Application in 5 Minutes with Fusion.js

Fusion.js start page
In this short tutorial, we show you how to build a Hello World! application using Fusion.js, an open source universal web framework suitable for building interactive web applications.

Improving the User Experience with Uber’s Customer Obsession Ticket Routing Workflow and Orchestration Engine

Uber adopted workflow orchestration and Cadence, our open source orchestration engine, to better handle customer support ticket routing at scale.

Building a Real-time Earnings Tracker into Uber’s New Driver App

Uber driver
What began as a means of showing Uber's driver-partners their real-time earnings quickly became an extensible means of communicating not just earnings, but also incentives and other useful information within our new driver app.

Uber Case Study: Choosing the Right HDFS File Format for Your Apache Spark Jobs

Uber's Maps Collection and Reporting (MapCARs) team shares best practices when choosing which HDFS file formats are optimal for use with Apache Spark.

Solving Big Data Challenges with Data Science at Uber

server racks
How engineers and data scientists at Uber came together to come up with a means of partially replicating Vertica clusters to better scale our data volume.

Accessible Machine Learning through Data Workflow Management

Complex freeway interchange
Uber engineers offer two common use cases showing how we orchestrate machine learning model training in our data workflow engine.

Announcing the Uber Open Summit Sofia 2019

Sunset over Temple St. Cyril and Methodius in Sofia. Bulgaria
The next Uber Open Summit, where we present our latest and most impactful open source projects, will be in Sofia, Bulgaria on April 20, 2019.

DBEvents: A Standardized Framework for Efficiently Ingesting Data into Uber’s Apache Hadoop Data Lake

Elephant silhouette
Uber engineers discuss the development of DBEvents, a change data capture system designed for high data quality and freshness that is capable of operating on a global scale.

Data Science at Scale: A Conversation with Uber’s Fran Bell

We spoke to Data Science Director Fran Bell about machine learning at Uber and what she finds most challenging—and rewarding—about her work.

Mitigating Risk in a Three-Sided Marketplace: A Conversation with Trupti Natu and Neel Mouleeswaran on the Uber Eats Risk Team

We sat down with a risk strategy manager and a risk engineer to discuss how they build solutions to minimize risk in the Uber Eats three-sided marketplace.

Open Sourcing Peloton, Uber’s Unified Resource Scheduler

First introduced by Uber in November 2018, Peloton manages resources across large-scale, distinct workloads, combining separate compute clusters.

Using Machine Learning to Ensure the Capacity Safety of Individual Microservices

Uber leveraged machine learning to design our capacity safety forecasting tooling with a special emphasis on calculating a quality of reliability score.

Introducing Kraken, an Open Source Peer-to-Peer Docker Registry

Developed by Uber, Kraken is an open source peer-to-peer Docker registry capable of distributing terabytes of data in seconds.

The Uber Engineering Internship Experience: European Edition

Uber's European interns
Engineering interns from Uber's European offices talk about their experiences, including the projects they worked on, the people they worked with, and the social activities they engaged in.

Architecting a Safe, Scalable, and Server-Driven Platform for Driver Preferences with RIBs

Subway platform
Our driver app's new server-driven preferences section enables driver-partners to customize their experiences to make the app better fit into their lives.

Managing Uber’s Data Workflows at Scale

In this article, we discuss Uber's journey toward a unified, multi-tenant, and scalable data workflow management system.

Profiles in Coding: Tatiana Romanova, Uber Payments Platform, Amsterdam

Amsterdam
Uber site reliability engineer Tatiana Romanova, based in our Amsterdam engineering office, discusses her computer science background, her journey to Uber, and her work maintaining our Payments Platform.

Creating Custom Plugins with Fusion.js, Uber’s Open Source Web Framework

In addition to providing official plugins, Fusion.js enables developers to build and integrate their own plugins by leveraging dependency injection.

Horovod Adds Support for PySpark and Apache MXNet and Additional Features for Faster Training

Horovod adds support for more frameworks in the latest release and introduces new features to improve versatility and productivity.

Popular Articles