Initial refactor / restructure of the codebase

[sfmig]

The codebase is getting a bit wild 🐆 , and I think some steps are consolidated enough now to make them a bit more established.

Roughly the pipeline would involve the following steps:

• Labelling prep: frames extraction and video reencoding. Maybe also checking the parameters of the raw videos (frame rate, image size etc)
• Labels postprocessing: combine existing labels and putting them in the correct format (COCO)
• Model training
• Evaluation
• Inference
  The last three steps are less well defined at this point.

@nikk-nikaznan and I chatted a bit today and some ideas came up:

• one option could be to follow a more functional programming approach: we have config files for each of the pipeline steps holding the main parameters, and scripts that take these config files as CLI arguments. The the steps are run using bash scripts. I am not very keen on more config files but I think this would be the easiest to transform to atm. This is sort of what we are doing now with the frame extraction step.

• Would this functional programming option be similar to the chain of responsibility @samcunliffe suggested?

• Another option would be a more OOP approach, I was thinking maybe similar to SLEAP's pipelines?

• Nik and Matt also suggested having a look at DVC - seems very well suited for ML but still flexible, might be a good investment to learn about it.

Any thoughts more than welcome, happy to discuss further at our next gemba.

[samcunliffe]

Spoke a bit about this on Thursday (@nikk-nikaznan).

My CoR-like idea is something closer to SLEAP's. Although their objects don't have an inheritance structure. It'd be something in code on our side. I was thinking something like:

class PipelineStep(abc.ABC):
    @abc.abstractmethod
    def run():
        pass

class VideoConversionStep(PipelineStep):
    def run():
        # run the actual encoder code
        self.encoder.encode(list_of_videos)

class FrameExtractionStep(PipelineStep):
    def run():
        ...


class Pipeline:
    def __init__(steps: list[PipelineStep]):
        self.steps = steps

    def run():
        for step in steps:
            output = step.run()
            if not output.is_ok():
                raise OhDearSomethingWentWrongError()

Other options we discussed:

• simple bash script to run steps in order
• snakemake
• airflow
• nextflow

My kneejerk would be to try snakemake first.

[sfmig]

This should deal with the following issues:

• Add additional tox support and features #63
• Add flake8 support for additional linting #62
• Add SWC submission scripts to mainbranch of some version control, please 🥺 #34

[sfmig]

After first chunk of light refactoring (#86), we merged the new structure to main, and then went on to close currently open PRs:

• add stereo-calibration work (as it is in current state)
  • PR Timecodes matching #2
• add bash scripts:
  • to run video reencoding (PR Bash script to reencode videos #31)
  • to run reencoding+frame extraction (PR Frame extraction with reencoding #78)
• add/update guides
  • PRs How to create GIN private repo for crab data #81,
  • PR Update extract frames guide #75
• add exclude pattern feature to combine annotations, with tests
  • PR Add exclude pattern to combining annotations #77
• finish test for frame extraction
  • PR Tests for frame extraction #24