Robustness Leaderboard

The robustness leaderboard compares the performance of different control methods by perturbing the simulation e.g. with noise or delay. The task for the controller is to swingup and balance the acrobot/pendubot.

The criteria can be found in

Controller Robustness

The scripts for the leaderboard calculation can be found in leaderboard/robustness/acrobot . for the acrobot and in leaderboard/robustness/pendubot . for the pendubot.

Creating the Leaderboard

For creating the leaderboard locally, you can run:

python create_leaderboard.py

This script will:

Check for controllers in all files starting with ‘con_’

Check if the robustness simulation data for these controllers already exists

Generate the data if it does not exist

Compute the leaderboard scores and save to a csv file

Leaderboard Parameters

The leaderboard uses a fixed model of the double pendulum and fixed simulation parameters. The parameters can be found in the sim_parameters.py file. The perturbations applied during the simulation can be checked in the benchmark documentation.

Evaluating your own controller

Note

For implementing your own controller see here

If you want to evaluate your own controller and compare it to the listed controllers on the leaderboard, you have to create a file with the name con_controllername.py, where controllername should be the name of the method your controller uses. In that file you should create an instance of your controller with the name controller (will be imported under this name from the other scripts). Additionally, yout con_controllername.py file should contain a dictionary:

leaderboard_config = {"csv_path": name + "/sim_swingup.csv",
                      "name": name,
                      "simple_name": "simple name",
                      "short_description": "Short controller description (max 100 characters)",
                      "readme_path": f"readmes/{name}.md",
                      "username": username}

where name is the controllername and username is your github username. Please add a simple name to display in the leaderboard and a short description of your controller with max. 100 characters. Do not use commas (,) in your description as they are used as separators in the data table! For participating on the official leaderboard, i.e. when you create a pull request, please also add a readme markdown file which describes your controller in readmes/controllername.md. Add references if applicable.

Feel free to import the model and simulation parameters from sim_parameters.py if you need them to instantiate your controller.

You can now generate the robustness data by calling:

python benchmark_controller.py con_controllername.py

This will simulate the double pendulum with all defined perturbations under the control of your controlled and save all relevant data in data/controllername.

Note

Depending on the speed of your controller and your PC this may take a while (1h-6h).

If you create a pull request and your controller and the con_controllername.py is added to the main repository your controller will appear on the official leaderboard .