Skip to main content
Uber logo

Schedule rides in advance

Reserve a rideReserve a ride

Schedule rides in advance

Reserve a rideReserve a ride
Engineering

Introducing Neuropod, Uber ATG’s Open Source Deep Learning Inference Engine

June 8, 2020 / Global
Featured image for Introducing Neuropod, Uber ATG’s Open Source Deep Learning Inference Engine
Figure 1. Over the years, Uber ATG has evolved its approach to machine learning, leveraging different popular deep learning frameworks.
Figure 2. It’s difficult to add support for a new framework because every piece of infrastructure that runs a model needs to support all frameworks. These infrastructure components could be metrics pipelines, model serving, or other production and testing environments.
Figure 3. Neuropod is an abstraction layer that provides a uniform interface to run deep learning models from multiple frameworks.
Figure 4. Neuropod features a framework-specific packaging API and a framework agnostic inference API.
Figure 5. Normally, applications directly interact with a deep learning framework throughout the inference process.
Figure 6. With Neuropod, applications interact with a framework-agnostic API and Neuropod interacts with the underlying framework.
Figure 7. Neuropod lets us build a single, optimized input-building pipeline that works with many models instead of building a separate one for every model or every framework.
Figure 8. Adding model-serving as a Neuropod backend allows any application that uses Neuropod to run models remotely without significant changes.
Figure 9. Applications can use Neuropod to proxy model execution to a remote machine.
Figure 10. Neuropod includes support for running models in isolated worker processes using a low-latency shared-memory bridge.
Vivek Panyam

Vivek Panyam

Vivek Panyam is a Senior Autonomy Engineer at Uber ATG. He works on the perception team where he leads the development of Neuropod. Previously, he led teams integrating deep learning and machine learning models into the self-driving fleet.

Posted by Vivek Panyam

Category: