Misc. worries to check or decide if they're issues

[sfmig]

1. OoM is a big issue? 😔
   • Meeting with IT 30-Apr-2024. @sfmig to attend.
   • Discussed in the retro.
2. Can't get the loss to decrease. Even with the augmentation we have we should get it low.
3. @sannatitus noticed that "A handful" crabs are not detected.
   • PI's focus for now is precision so... to first order "no" not an issue.
4. eval metrics for detector computed in two ways don't match. Eval metrics for detection #99

[sfmig]

Re GPU issue:
I attended the hpc meeting today, and someone raised that some SLEAP jobs seemed to be using the GPU (as in, they could see the GPU memory filled in with data) but actually the rate at which frames were processed suggested the CPU was used instead (compared to running things locally). This made me suspicious of our 3 day-long jobs...

Note to self: inspect this further next term and profile and inspect things in my local machine.

[sfmig]

Some things to keep in mind for the GPU issue:

• check profiling tools
• check CPU usage
• could be an IO issue