T#5: End2End Driving
Last updated
Was this helpful?
Last updated
Was this helpful?
, Software Engineer II, Microsoft
, Program Manager, Microsoft
Y.-K Kim, 2020-07-07
In this tutorial, you will learn how to train and test an end-to-end deep learning model for autonomous driving using data collected from the . You will train a model to learn how to steer a car through a portion of the Mountain/Landscape map in AirSim using a single front facing webcam for visual input. Such a task is usually considered the "hello world" of autonomous driving, but after finishing this tutorial you will have enough background to start exploring new ideas on your own. Through the length of this tutorial, you will also learn some practical aspects and nuances of working with end-to-end deep learning methods.
Here's a short sample of the model in action:
This tutorial is presented to you in the form of Python notebooks. Python notebooks make it easy for you to read instructions and explanations, and write and run code in the same file, all with the comfort of working in your browser window. You will go through the following notebooks in order:
You should also be comfortable with Python. At the very least, you should be able to read and understand code written in Python.
Download the repository as zip file or
From a terminal window, change to the local directory where you want to clone and run
example: if the virtual environment name is 'tf22'
conda install -c anaconda keras
jupyter
matplotlib v. 2.1.2
image
keras_tqdm
opencv
msgpack-rpc-python
pandas
numpy
scipy
.\AD_Cookbook_Start_AirSim.ps1 landscape
to start the simulator in the landscape environment.
It is highly recommended that a GPU is available for running the code in this tutorial. While it is possible to train the model using just a CPU, it will take a very long time to complete training. This tutorial was developed with an Nvidia GTX970 GPU, which resulted in a training time of ~45 minutes.
Stops with error in generating test/validation dataset of .h5 files in DataExplorationAndPreparation It seems it creates train.h5 but does not create test.h5 and eval.h5
When running this code in DataExplorationAndPreparation
Iteration stops with this output message. It seems it creates train.h5 but does not create test.h5 and eval.h5 Codes before this line worked fine.
Tutorial used: AirSimE2EDeepLearning-> DataExplorationAndPreparation
Environment used: landscape
Versions of artifacts used (if applicable): Python 3.7.7 Keras 2.3.1 Using Tensorflow backened, h5py
{% youtube src="
The code presented in this tutorial is written in , a high-level deep learning Python API capable of running on top of , or . The fact that Keras lets you work with the deep learning framework of your choice, along with its simplicity of use, makes it an ideal choice for beginners, eliminating the learning curve that comes with most popular frameworks.
1.
2.
3.
If you have never worked with Python notebooks before, we highly recommend .
You should be familiar with the basics of neural networks and deep learning. You are not required to know advanced concepts like LSTMs or Reinforcement Learning but you should know how Convolutional Neural Networks work. A really good starting point to get a strong background in a short amount of time is written by Michael Nielsen. It is free, very short and available online. It can provide you a solid foundation in less than a week's time.
Clone the tutorial git repository ''
Note for more help
. Be sure to add the location for the AzCopy executable to your system path.
with Python 3.5 or higher.
Note: Use virtual environment to install Tensorflow, Keras and other dependencies Note
Note
and to work with TensorFlow (default).
Install the other dependencies. From your anaconda virtual environment, run as root or administrator. This installs the following packages into your environment:
We have created a standalone build of the AirSim simulation environment for the tutorials in this cookbook. . Consider using , as the file size is large. After downloading the package, unzip it and run the PowerShell command
If you do not have a GPU available, you can spin up a , which comes with all the dependencies and libraries installed (use the provided py35 environment if you are using this VM).
The dataset for the model is quite large. . The first notebook will provide guidance on how to access the data once you have downloaded it. The final uncompressed data set size is approximately 3.25GB (which although is nothing compared to the petabytes of data needed to train an actual self-driving car, should be enough for the purpose of this tutorial).
We have made our best effort to ensure this tutorial can help you get started with the basics of autonomous driving and get you to the point where you can start exploring new ideas independently. We would love to hear your feedback on how we can improve and evolve this tutorial. We would also love to know what other tutorials we can provide you that will help you advance your career goals. Please feel free to use the GitHub issues section for all feedback. All feedback will be monitored closely. If you have ideas you would like to on, please feel free to reach out to us and we will be happy to work with you.
