MOTA revisited

[nikk-nikaznan]

I am trying to add more definition to what we consider as ID switches. This PR consider ID switch as a crab been assigned a new ID (re-ID)

• This is a bug : The old way of getting the ID switch is by finding the symmetric_difference between two list. The problem with this if we have prev_id = [1, 2, 3] and current_id = [1, 2, 4] the n_switches =2. The new approach will be 1.

After some help from @sfmig and @samcunliffe, this is currently what we have:

• We create a dict to map the ground_truth ID to tracked ID in every frame.

  • during count_identity_switches we have prev_frame_id_map and current_frame_id_map.
  • for each gt ID and tracked ID of current frame, we retrieve the tracked ID from previous frame that associated with a particular ground truth ID in current frame.
  • we also do the inverse, we retrieve the ground truth ID from previous frame that associate with a particular predicted ID in current frame.
  • we only check once, if the previous ID either from ground truth or prediction exist, if they exist, we check if the value is the same.

• Some changes for the test -- feel free to play around with the value in the test to see any other possibility.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 72.72727% with 33 lines in your changes missing coverage. Please review.

Project coverage is 46.16%. Comparing base (ff59d84) to head (1d45dd6).

Files	Patch %	Lines
crabs/tracker/evaluate_tracker.py	80.43%	18 Missing ⚠️
crabs/tracker/track_video.py	40.00%	15 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #181      +/-   ##
==========================================
+ Coverage   39.52%   46.16%   +6.64%     
==========================================
  Files          24       24              
  Lines        1417     1460      +43     
==========================================
+ Hits          560      674     +114     
+ Misses        857      786      -71     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sfmig]

hey @nikk-nikaznan, I was just having a look at this and I get a different result:

Similarly, if prev_id = [1, 2, 3] and current_id = [1, 2, 3, 4] with current_gt = 4, n_switches = 0. (A new ID is appear in the current frame)

For me this returns n_switches=1

I think this is not intended right? Otherwise whenever a crab appears in view (exiting a burrow for example) it will be counted as an ID switch.

[sfmig]

I think there is some confusion, because as I understand it from the HOTA paper we only check ID switches for the true positives. So we shouldn't be filtering out missed detections or false negatives inside this function, right?

I got this from sections 3 and 4.1 (included). I recommend it, it is very well explained.

I think we could rewrite the count_identity_switches function to count ID switches as they say in the paper. Following their approach, we could fix the error above and also detect if IDs are swapped between crabs (right now, we don't detect this if all IDs are present in frame f and frame f-1).

[sfmig]

I think it would be something like follows:

1. For the bounding boxes that match the ground truth in frame f (i.e., that overlap sufficiently with the ground truth box), compute the map (dictionary) from ground truth IDs to predicted IDs.
2. Get the same map for the previous frame f-1 (we would have stored it).
3. The number of ID switches is the number of times the following thing occurs: for every ground-truth ID (keys of the dictionary), when do the predicted IDs disagree?

Let me know thoughts, happy to chat offline if you prefer.

[sfmig]

I think the count_identity_switches is not behaving as expected 🤔

I explained in the comments above ☝️

Maybe a good one to pair program? Let me know thoughts.

[samcunliffe]

@sfmig I think there is some confusion, because as I understand it from the HOTA paper we only check ID switches for the true positives. So we shouldn't be filtering out missed detections or false negatives inside this function, right?

https://github.com/JonathonLuiten/TrackEval/blob/12c8791b303e0a0b50f753af204249e622d0281a/trackeval/metrics/clear.py#L93-L97

[sfmig]

thanks Nik! This is great 🌟

I reviewed the rest. Apologies that it's a big one 🙏

The bigger points of this PR are:

• suggested reimplementation of count_identity_switches function.
• tracked_list should be a dictionary (following gt_data) rather than a list (otherwise if frames are not valid Python indices it will break).
• dictionaries that map from ground truth ID to predicted ID to include missed detections.
• suggestions to rearrange and remove duplicates in the parametrisation of the ID switches test.

Re the first point, I think you are right and with the latest fixes the suggested algorithm doesn't cover any additional cases compared to the current one (contrary to what we thought initially).

However, I think it is a bit easier to read and conceptualise, and it allows us to structure the cases to test. Plus if in the future we want to cover that case of reID after a missed detection gap, I think the suggested algorithm would make it easier.

[sfmig]

thanks Nik, it looks great!

Just a few frame_idx to frame_number to avoid confusion when it the frame number is not a valid index to a list / array.