2023-09-13 Data Architecture Meeting

[jkbhagatio]

• Merging PRs: Update pose reader #253, Refactored tests to use absolute file paths #242, Blackened and ruffed #244
  (https://teams.microsoft.com/l/message/19:meeting_MjFmZjJkM2YtZDhmMS00Yzg5LWI5ODctOTM3ZmE0N2NjYjIz@thread.v2/1694461081798?context=%7B%22contextType%22%3A%22chat%22%7D)
• Where / How to log metadata that is not immediately analysis specific aeon_experiments#150
• Consider moving rig testing and calibration workflows to reusable acquisition package aeon_experiments#308
• Example 'aeon/analysis/' refactor #245
• Wheel build for Datajoint failing on macOS 12 using mamba #249
• Exclude dj_pipeline from precommit hooks #247
  • [dj pipeline CI checks error log]
• Datajoint pipeline #199
• SOW for Y3-M2 review

[jkbhagatio]

Some points for for future reference / follow-up:

• We discussed whether manually allowing new insertions to the Colony db table should be allowed: I think we settled on having an approach where, on PyRat, an experimenter specifies which animals are Aeon animals via some sort of tag.

  • On the sciviz side, the experiment can add their name to an 'Experimenter' table, and upon new entries this could trigger a job that looks for all animals belonging to that experimenter that have the Aeon tag, and those animals get added to the Colony db.
  • There could also be a column in the Colony db table that is called 'Experiment' - and whenever data from a particular experiment is ingested into the db, a job could be run to see which animals were part of that data ingestion, and find those corresponding animal keys in the Colony db table, and add that experiment to the 'Experiment' column for those animals, if not already there.

• We tabled the discussion of the Example 'aeon/analysis/' refactor #245 for future. I think I'm currently more in favor of including an 'analysis' subpackage within 'aeon_mecha' (nice to have everything in one place, avoid issues with things like circular imports), rather than have a separate 'aeon_analysis' repo (but I think others like @lochhh are in favor of this latter approach). In either case, the current 'aeon_analysis' repo could be renamed 'aeon_scratchpad'.