Integration test for detect+track video

[sfmig]

Description

What is this PR?

• Bug fix
• Addition of a new feature
• Other

Why is this PR needed?
To ensure we maintain the functionality of the detector and tracker when refactoring related parts of the codebase.

What does this PR do?

• Adds an integration test for the detect-and-track-video command, when ground-truth is passed.
  • Downloads sample data from GIN using pooch
  • Parametrises the test for different optional outputs (save frames or not, save tracked video or not).
• Adds a caching step to the Github actions workflow in CI
• Additionally, the CLI help for the --output-dir is edited, to clarify it refers to the root name of the output directory rather than the full path to it (I opened an issue Support specifying output directory path for detect-and-track-video #246 to support this other functionality)

Questions

I initially was parametrising possible flags for --output-dir too, but decided to remove them to reduce the run time of these tests and thinking this would be better covered in a unit test. But opinions are welcome on this!

References

For now, the integration test checks the command runs successfully and the expected outputs are created.

After reviewing and fixing the tracking evaluation, the tests should be expanded to confirm the output csv files contain the expected metrics (PR #243).

How has this PR been tested?

Existing and new tests pass locally and in CI.

Does this PR require an update to the documentation?

N/a

Checklist:

• The code has been tested locally
• Tests have been added to cover all new functionality
• [ n/a ] The documentation has been updated to reflect any changes
• The code has been formatted with pre-commit

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.18%. Comparing base (ba60fa9) to head (f65dfb2).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #242      +/-   ##
==========================================
+ Coverage   47.98%   48.18%   +0.19%     
==========================================
  Files          24       24              
  Lines        1567     1567              
==========================================
+ Hits          752      755       +3     
+ Misses        815      812       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sfmig]

as @niksirbi made me realise, if knonw_hash = None, the file is always downloaded.

So there is no need to set it to a different name

[niksirbi]

Very nice @sfmig! The integration test is very understandable and readable (even for me, having my first contact with this repo), and it makes sense.

The pooch bits we have already discussed intensely, and I think you can get rid of the time stamp.

FYI I timed the tests on my Macbook and they took:

• ~11 min the first time they were run (so including pooch download time)
• ~2.5 minutes for subsequent runs

Most time is spent on the new test_inference.py test.

This PR is good to go as is, but I'm wondering if you've considering running this on a smaller dataset (mostly in the interest of speeding up CI, if that's not so important ignore me).

[sfmig]

thanks for the feedback @niksirbi !

The main bottleneck is not the size of the sample data, but the download of the trained model (which is ~160MB). I checked if it was possible to compress the data further to speed this up (increasing the level of compression with zip -r -9) but there was not much further compression gain. I think it is because the .ckpt file itself is already a compressed directory.

For now I have marked the integration test as slow, so that it can be skipped locally if desired with pytest -m "not slow". I also skip the slow tests in CI if the runner is macos-13 as it is very often the slowest of all runners by quite a bit (see others discussing this too).

One thing I didn't know that was very useful in this process is that pytest can return setup and call times using the --durations=0 flag