Using hyperparameters from a checkpoint that is "weight-only"

[sfmig]

If we merge #182 with the suggested fix, we will be able to restart training from a checkpoint. However, we will always use the hparams as specified in the config .yaml file.

Would we want to have the option to use the hparams from the checkpoint?
Checkpoints in pytorch lightning include hyperparameters, but it is not clear to me when these are loaded.

If we don't pass config to load_from_checkpoint we would in principle use the hparams from the checkpoint. However, this leads to a mismatch between the logged hparams in MLflow and the actual hparams used.

lightning_model = FasterRCNN.load_from_checkpoint(
	self.checkpoint_path,
	config=self.config,
)

To reproduce this bug:

1. Remove the config argument we pass to FasterRCNN.load_from_checkpoint()
2. Train a model for one epoch (specifying n_epochs=1 in the yaml file) and save a weights_only checkpoint.
   • the checkpoint is at the path_to_checkpoints parameter logged in MLflow (the name is last.ckpt).
3. then launch a training job that starts from that checkpoint . Before I launch it, I edit the config file to have n_epochs=3.
4. In MLflow, this second training job has the same hyperparameters as the job that produced the training (so it has n_epochs=1 etc), but in reality the job runs for as many epochs as in the yaml file. So it logs n_epochs=1, but runs for n_epochs=3.