Add exclude pattern to combining annotations

[sfmig]

Add option to exclude files from the combination operation.

It allows us to use a regex pattern to exclude annotation files we don't want to be part of the combined annotation file.

[codecov-commenter]

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (aa2fca4) 29.81% compared to head (ac906bd) 29.37%.

Files	Patch %	Lines
...bboxes_labelling/combine_and_format_annotations.py	0.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #77      +/-   ##
==========================================
- Coverage   29.81%   29.37%   -0.45%     
==========================================
  Files           6        6              
  Lines         332      337       +5     
==========================================
  Hits           99       99              
- Misses        233      238       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[samcunliffe]

(crabs-env) ➤ for i in {1..10}; do touch test_fake_jsons/good-$i.json; done
(crabs-env) ➤ for i in {1..10}; do touch test_fake_jsons/bad-$i.json; done
(crabs-env) ➤ ls test_fake_jsons
bad-1.json	bad-4.json	bad-8.json	good-2.json	good-6.json
bad-10.json	bad-5.json	bad-9.json	good-3.json	good-7.json
bad-2.json	bad-6.json	good-1.json	good-4.json	good-8.json
bad-3.json	bad-7.json	good-10.json	good-5.json	good-9.json
(crabs-env) ➤ combine-annotations test_fake_jsons --exclude-pattern "bad-*.json"

Obviously, my files are empty so I expect it to fail when opening, but it seems you don't exclude the bad guys...

│ ╭──────────────────────────────── locals ────────────────────────────────╮                       │
│ │       exclude_pattern = 'bad-*.json'                                   │                       │
│ │ list_input_json_files = [                                              │                       │
│ │                         │   PosixPath('test_fake_jsons/good-10.json'), │                       │
│ │                         │   PosixPath('test_fake_jsons/good-3.json'),  │                       │
│ │                         │   PosixPath('test_fake_jsons/bad-2.json'),   │                       │
│ │                         │   PosixPath('test_fake_jsons/bad-3.json'),   │                       │
│ │                         │   PosixPath('test_fake_jsons/good-2.json'),  │                       │
│ │                         │   PosixPath('test_fake_jsons/bad-4.json'),   │                       │
│ │                         │   PosixPath('test_fake_jsons/good-9.json'),  │                       │
│ │                         │   PosixPath('test_fake_jsons/bad-8.json'),   │                       │
│ │                         │   PosixPath('test_fake_jsons/good-5.json'),  │                       │
│ │                         │   PosixPath('test_fake_jsons/bad-10.json'),  │                       │
│ │                         │   ... +10                                    │                       │
│ │                         ]                                              │                       │
│ │  parent_dir_via_jsons = 'test_fake_jsons'                              │                       │
│ │       via_default_dir = None                                           │                       │
│ │      via_project_name = None                                           │                       │
│ ╰────────────────────────────────────────────────────────────────────────╯                       │

... am I doing something wrong?

[samcunliffe]

Looks good, it's possible I just can't get it to work because I'm being dim.

[sfmig]

thanks @samcunliffe for having a look!

Re the fake jsons test, I reproduced exactly your commands and I got: ValueError: Error decoding JSON data from file: test_fake_jsons/bad-1.json - which makes sense since there is nothing in the json file right? We are also expecting json files in the VIA format.

If you are wondering why the list_input_json_files includes the "bad" ones, I would still expect that because the filtering occurs when you combine them (rather than when you read the CLI arguments).

This test in which we copy a sample annotation file should work:

$ mkdir test_fake_jsons
$ for i in {1..10}; do cp tests/data/COCO_VIA_JSONS/VIA_JSON_1.json test_fake_jsons/good-$i.json; done
$ for i in {1..10}; do cp tests/data/COCO_VIA_JSONS/VIA_JSON_1.json test_fake_jsons/bad-$i.json; done
$ combine-annotations test_fake_jsons --exclude-pattern "bad-*.json"

Let me know if you think something should be changed about this.

[samcunliffe]

If you are wondering why the list_input_json_files includes the "bad" ones, I would still expect that because the filtering occurs when you combine them (rather than when you read the CLI arguments).

Ah right. I assumed you'd filter the files before you opened them.

... but OK. Not a blocker.

[sfmig]

Ah right. I assumed you'd filter the files before you opened them.

@samcunliffe I do, the error message just shows what you passed as a CLI argument right?

[samcunliffe]

Ah right. I assumed you'd filter the files before you opened them.

@samcunliffe I do, the error message just shows what you passed as a CLI argument right?

Ahhh sorry. I'm being slow. So the error is thrown after filtering, before opening the files ✅. But the fact that it threw an error at all makes typer dump the CLI arguments for your information.

[samcunliffe]

Confirmed that it works with toy Jsons containing actual content. 👍