Experiments

We now describe how to run the experiments used in this work and how to evaluate the forecasts. All training runs of the transformer can be visualized here.

To compare the transformer and the VAR model, you can run this script.

4 experiments with the transformer were conducted:

  1. Learning rate effect

  2. Forecast mask effect

  3. Effect of air measuring station

  4. Ablation of variables from input

The commands to train/test the appropriate transformer models can be found in the aforementioned location.

The same script that compares the transformer and the VAR model also generates the learning rate experiment.

All tables and figures found generated by these scripts can be found in the thesis document itself.

If you would like to look at an interactive report detailing the training and validation loss from experiments 1, 2, and 4, we have made a report available here.

Evaluation

The evaluation of the models can be conducted using these scripts.

You can generate the results for each experiment using the standalone R scripts. Furthermore, you can generate all experimental results in the RMarkdown script. This also includes all text found in the results section of the thesis.