Andrej Karpathy Details Autopilot in 10 Minutes

Andrej Karpathy Details Autopilot's Inner Workings at PyTorch Conference
Andrej Karpathy Details Autopilot's Inner Workings at PyTorch Conference

Tesla’s Director of Artificial Intelligence, Andrej Karpathy, spoke at the 2019 PyTorch Developer Conference and shared some of the details around Tesla’s Autopilot Neural Network.

Check out the short 10 minute video here:

Famously, Tesla relies primarily on cameras to perceive its environment (plus a front facing radar and ultrasonic sensors). Unlike other self-driving automotive players, it does not use LiDAR (see LiDAR vs Cameras for Self-Driving for more). All new Tesla vehicles with AP2 and above (see AP1 vs AP2) have eight cameras surrounding the vehicle.

Tesla’s Eight Cameras for Autopilot Perception

They process these images with Convolutional neural networks. Since Tesla is a vertically integrated company, they own the entire stack.

Tesla Full Stack AI

One of the biggest component centers around analyzing images in the PyTorch distributed training portion of the stack. In just a single image frame there are numerous elements the network needs to identify and tag.

Example elements to identify in a driving scene.

The “Hydranet” ties together all the cameras to help identify key road features:

Tesla HydraNet
Tesla Camera Hydranet identifying lane lanes

The Tesla Hydranet is also used, of course, in the case of Smart Summon, as shown here:

Tesla HydraNet used in Smart Summon
Tesla HydraNet used in Smart Summon to Show Driving Space

The Tesla Hydranet is also used for road layout prediction, stitching together a top-down view of the road as shown here:

Tesla Autopilot Road Layout Prediction
Tesla Autopilot Road Layout Prediction

Quite a bit of the newer featured being rolled out via these neural networks target the new Full Self Driving Computer (FSD Computer, aka Hardware 3) which is an order of magnitude faster then the original NVIDIA that was used in Hardware 2 (AP2) Tesla vehicles (see AP1 vs AP2 vs AP3).

Tesla Full Self Driving Computer (aka Hardware 3)
Tesla Full Self Driving Computer (aka Hardware 3)

Elon Musk has said that he expects Full Self Driving to become available in 2020 based on the rapid progress the machine learning and training systems are making from the almost quarter million vehicles on the road helping to train the network.

Learn More About Tesla Autopilot