Implement autonomous mode using Tensorflow.js

[Ezward]

See this repo for an example using the Udacity Self Driving Simulator; https://github.com/bakoushin/self-driving-car-javascript

Here is a medium article about this; https://levelup.gitconnected.com/run-a-self-driving-car-using-javascript-and-tensorflow-js-8b9b3f7af23d

Strategy:

• This is mediated by the webUI. The user interacts with the web ui in 3 modes; collection mode, training mode and autopilot mode.
• There is a server involved the get's the collected data, does the training and returns the resulting model to the client.
• The client initiates a websocket connection with the server. Once connected to the server, the collection and training modes are enabled. Autopilot mode is available without a server if there are models stored in local indexedDB; if not then autopilot mode becomes available when the server is connected. Each mode is detailed in its' own section below.
• Normal telemetry is displayed in collection mode and autopilot mode.
• Training happens in the server (probably nodejs) application. During training the rover can continue to be operated, including running collection or autopilot mode.

Collection Mode

• The user chooses collection mode and provides the address of the server and chooses to start a new data set or append to the current data set.
• The webui closes it's camera socket
• The webui opens a socket to the server
• The webui sends the address of the rover to the server
• The rover opens a camera socket to the rover
• When the user selects to start capture or capture is triggered automatically by a non-zero throttle, the webui sends a message to the server to start recording.
• The server chooses the data set or creates a new one depending on what the webui indicated.
• We will modify the camera socket so that the rover always sends metadata with the image; steering angle, throttle, angular velocity, linear velocity, pose.x, pose.y, pose.angle, timestamp
• The server will save the data as it arrives (naming each image uniquely and in a way that is increasing monotonically, probably saving the metadata in a csv file and including relative path to the image)
• The server will forward the data to the webui through it's websocket so it can be displayed.
• When the web ui indicates recording should stop, the server stops saving data to the dataset and closes it's websocket connection to the rover. The webui reopens it's camera socket connection to the rover.

Training Mode

• The user chooses training mode and provides the address of the server (we will remember this in localstorage so we can provide a good default).
• The webui connects to the server (if it is not already) and asks for the list of datasets.
• The webui shows the list of datasets.
• The user checks the datasets they wish to train on.
• The webui sends a message to the server with the list of datasets to train on.
• The server starts training and sends status updates to the webui.
• The webui shows the status of the training.
• When completed, the server saves the model to disk and messages the webui, the webui saves the model in indexedDb

Autopilot Mode

• The user chooses autopilot mode and optionally prov
• The webui shows the models that are saved in indexedDb
• If connected to a server, the webui asks for the list of models persisted to disk and merges this with those saved locally in indexedDB
• The user chooses a model.
• The pose is reset to (0, 0, 0)
• The webui enters the autopilot loop;
  • an image and metadata are received from the rover.
  • using the image, a steering angle is inferred using the model.
  • the steering angle and throttle are sent to the rover.
  • the rover uses an inverse kinematic model to calculate wheel velocities to match the desires velocity and steering angle and updates the wheel velocities. All normal telemetry is sent to the client and the client displays the telemetry.
• The user chooses to stop autopilot in the webui
  • The webui tells the rover to stop
  • The webui exits the autopilot loop