Convnet Regressor

What is it?

ConvnetRegressor is an openFrameworks application, part of the ml4a-ofx collection, which lets you train a neural network to control one or more continuous variables (sliders) with your webcam. The app learns a regression to predict the slider values, from a set of example input/output pairs collected via webcam stream. It is closely related to ConvnetClassifier, which is the same as ConvnetRegressor, except that it does classification over discrete categories rather than continuous variables.

todo: ableton time trial

Training instructions

User interface for ConvnetRegressor
  1. One slider is loaded by default. Click ‘add slider’ if you wish to train more.

  2. Set the sliders to a particular set of values, and then click ‘record’ to begin recording images from the webcam. The images are analyzed by a convnet, and the feature vector extracted becomes associated with the set of slider values set.

  3. Repeat #2 for as many sets of values as you wish.

  4. Set training parameters, hidden neurons and epochs (training iterations). In general, the higher these are, the more accurate the training will be, but the longer it will take.

  5. Click train, and wait. Enjoy the flashing colors.

  6. Once trained, toggle Predict to have the sliders get set by the trained network.

  7. The slider prediction lerp to control the amount of easing in the predicted slider values. If set to 1.0, there is no easing, and the sliders are set to the predicted value exactly (this may be erratic). If set to a lower value, the sliders will fluctuate more gradually.

OSC Output

The OSC destination (IP address and port), and the message to bind the prediction values can be set by clicking ‘Change OSC settings’. All settings are saved on exit, and will be loaded back next time you open the application.

Setup and training considerations

ConvnetRegressor is a very versatile application that can be used in a wide variety of setups. However, to optimize accuracy, some simple rules of thumbs are: