Running experiments systematically in the cluster

[sfmig]

There are a few things to polish before we are able to systematically run experiments in the cluster.

Experiments = training a detector with different (hyper)parameters, for at least 3 dataset splits, and comparing them.

Some bits noted so far:

• write a bash script for training in the cluster.
  • fix OOM issues
  • agree on handling of environments
• Save checkpoints every (n?) epochs (right now, just saving the final model).
• Save loss vs epoch, to monitor training.
• Use validation set during training.
  • Compute val loss ---> see Implement validation loss during training #159
  • Compute eval metrics for val set
  • maybe in the same context: change evaluation script to use lightning syntax/approach.
• Link slurm job ID - MLflow run - final model (right now it's tricky to link them all)
• Verify that we can overfit the training dataset. Right now the loss seems a bit high after ~100 epochs.
  • check first without data augm