T#5: End2End Driving

Authors:

Mitchell Spryn, Software Engineer II, Microsoft

Aditya Sharma, Program Manager, Microsoft

Modified:

Y.-K Kim, 2020-07-07

Overview

In this tutorial, you will learn how to train and test an end-to-end deep learning model for autonomous driving using data collected from the AirSim simulation environment. You will train a model to learn how to steer a car through a portion of the Mountain/Landscape map in AirSim using a single front facing webcam for visual input. Such a task is usually considered the "hello world" of autonomous driving, but after finishing this tutorial you will have enough background to start exploring new ideas on your own. Through the length of this tutorial, you will also learn some practical aspects and nuances of working with end-to-end deep learning methods.

Here's a short sample of the model in action:

Demo

https://youtu.be/CauKo089zm0

Demonstration

{% youtube src="https://www.youtube.com/watch?v=cFtnflNe5fM"}

Structure of this tutorial

The code presented in this tutorial is written in Keras, a high-level deep learning Python API capable of running on top of CNTK, TensorFlow or Theano. The fact that Keras lets you work with the deep learning framework of your choice, along with its simplicity of use, makes it an ideal choice for beginners, eliminating the learning curve that comes with most popular frameworks.

This tutorial is presented to you in the form of Python notebooks. Python notebooks make it easy for you to read instructions and explanations, and write and run code in the same file, all with the comfort of working in your browser window. You will go through the following notebooks in order:

1. DataExplorationAndPreparation

Modified notebook

2. TrainModel

3. TestModel

If you have never worked with Python notebooks before, we highly recommend checking out the documentation.

Prerequisites and setup

Background needed

You should be familiar with the basics of neural networks and deep learning. You are not required to know advanced concepts like LSTMs or Reinforcement Learning but you should know how Convolutional Neural Networks work. A really good starting point to get a strong background in a short amount of time is this highly recommended book on the topic written by Michael Nielsen. It is free, very short and available online. It can provide you a solid foundation in less than a week's time.

You should also be comfortable with Python. At the very least, you should be able to read and understand code written in Python.

Environment Setup

Clone the tutorial git repository 'https://github.com/microsoft/AutonomousDrivingCookbook'
- Download the repository as zip file or
- From a terminal window, change to the local directory where you want to clone and run
  git clone https://github.com/microsoft/AutonomousDrivingCookbook.git
  Note Read how to clone a repository for more help
Install AzCopy. Be sure to add the location for the AzCopy executable to your system path.
Install Anaconda with Python 3.5 or higher.

Note: Use virtual environment to install Tensorflow, Keras and other dependencies Note Read how to use virtual environment in Anaconda

Install Tensorflow

example: if the virtual environment name is 'tf22'

     activate tf22
     conda install -c anaconda tensorflow-gpu

Note Read how to install Tensorflow with Anaconda

Install h5py
```
 `conda install h5py`
```
Install Keras and configure the Keras backend to work with TensorFlow (default).
conda install -c anaconda keras
Install the other dependencies. From your anaconda virtual environment, run InstallPackages.py as root or administrator. This installs the following packages into your environment:
- jupyter
- matplotlib v. 2.1.2
- image
- keras_tqdm
- opencv
- msgpack-rpc-python
- pandas
- numpy
- scipy

Simulator Package

We have created a standalone build of the AirSim simulation environment for the tutorials in this cookbook. You can download the build package from here. Consider using AzCopy, as the file size is large. After downloading the package, unzip it and run the PowerShell command

.\AD_Cookbook_Start_AirSim.ps1 landscape

to start the simulator in the landscape environment.

Hardware

It is highly recommended that a GPU is available for running the code in this tutorial. While it is possible to train the model using just a CPU, it will take a very long time to complete training. This tutorial was developed with an Nvidia GTX970 GPU, which resulted in a training time of ~45 minutes.

If you do not have a GPU available, you can spin up a Deep Learning VM on Azure, which comes with all the dependencies and libraries installed (use the provided py35 environment if you are using this VM).

Dataset

The dataset for the model is quite large. You can download it from here. The first notebook will provide guidance on how to access the data once you have downloaded it. The final uncompressed data set size is approximately 3.25GB (which although is nothing compared to the petabytes of data needed to train an actual self-driving car, should be enough for the purpose of this tutorial).

A note from the curators

We have made our best effort to ensure this tutorial can help you get started with the basics of autonomous driving and get you to the point where you can start exploring new ideas independently. We would love to hear your feedback on how we can improve and evolve this tutorial. We would also love to know what other tutorials we can provide you that will help you advance your career goals. Please feel free to use the GitHub issues section for all feedback. All feedback will be monitored closely. If you have ideas you would like to collaborate on, please feel free to reach out to us and we will be happy to work with you.

ISSUES

1. Fails to create test.h5 and eval.h5 in DataExplorationAndPreparation

Problem description

Stops with error in generating test/validation dataset of .h5 files in DataExplorationAndPreparation It seems it creates train.h5 but does not create test.h5 and eval.h5

Problem details

When running this code in DataExplorationAndPreparation

    train_eval_test_split = [0.7, 0.2, 0.1]
    full_path_raw_folders = [os.path.join(RAW_DATA_DIR, f) for f in DATA_FOLDERS]
    Cooking.cook(full_path_raw_folders, COOKED_DATA_DIR, train_eval_test_split)

Iteration stops with this output message. It seems it creates train.h5 but does not create test.h5 and eval.h5 Codes before this line worked fine.

    Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_1...
    Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_2...
    Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_3...
    Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_4...
    Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_5...
    Reading data from ../dataset/EndToEndLearningRawData/data_raw/normal_6...
    Reading data from ../dataset/EndToEndLearningRawData/data_raw/swerve_1...
    Reading data from ../dataset/EndToEndLearningRawData/data_raw/swerve_2...
    Reading data from ../dataset/EndToEndLearningRawData/data_raw/swerve_3...
    Processing ../dataset/data_cooked/train.h5...
    ---------------------------------------------------------------------------
    StopIteration                             Traceback (most recent call last)
    ~\Git_Program\AirSim\ADCookBook\Tutorial_E2EDeepLearning\Cooking.py in generatorForH5py(data_mappings, chunk_size)
        129                 raise StopIteration
    --> 130     raise StopIteration
        131 

    StopIteration: 

    The above exception was the direct cause of the following exception:

    RuntimeError                              Traceback (most recent call last)
    <ipython-input-13-bce568193587> in <module>
        1 train_eval_test_split = [0.7, 0.2, 0.1]
        2 full_path_raw_folders = [os.path.join(RAW_DATA_DIR, f) for f in DATA_FOLDERS]
    ----> 3 Cooking.cook(full_path_raw_folders, COOKED_DATA_DIR, train_eval_test_split)

    ~\Git_Program\AirSim\ADCookBook\Tutorial_E2EDeepLearning\Cooking.py in cook(folders, output_directory, train_eval_test_split)
        192     for i in range(0, len(split_mappings), 1):
        193         print('Processing {0}...'.format(output_files[i]))
    --> 194         saveH5pyData(split_mappings[i], output_files[i])
        195         print('Finished saving {0}.'.format(output_files[i]))
        196 

    ~\Git_Program\AirSim\ADCookBook\Tutorial_E2EDeepLearning\Cooking.py in saveH5pyData(data_mappings, target_file_path)
        162         dset_previous_state[:] = previous_state_chunk
        163 
    --> 164         for image_names_chunk, label_chunk, previous_state_chunk in gen:
        165             image_chunk = np.asarray(readImagesFromPath(image_names_chunk))
        166 
    RuntimeError: generator raised StopIteration

Experiment/Environment details

Tutorial used: AirSimE2EDeepLearning-> DataExplorationAndPreparation
Environment used: landscape
Versions of artifacts used (if applicable): Python 3.7.7 Keras 2.3.1 Using Tensorflow backened, h5py

PreviousUntitled NextUseful Resources

Last updated 3 years ago

Was this helpful?