Running full-pose (and in future other) post-processing on hpc, in close-to-real-time #385
jkbhagatio
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
@ttngu207 @lochhh @anayapouget @OrsiFolsz today we discussed options for running close-to-real-time postprocessing on raw data on the hpc (i.e., after each chunk's worth of data has been copied over to ceph).
Specifically, for sleap data, to get full-pose data for social periods in the social experiment, we discussed running a python script that looks for new sleap raw bin files, and then calls submitit to submit a job to the hpc for that file.
We then discussed naming conventions for this process, and whether each processed file should contain in its filename each separate hpc job id (and thus a separate config file would be created for each job), or whether we should put in another "fake" job id for all jobs for a given computer-experiment combo.
@anayapouget preferred this latter approach, and will look more into this.
Beta Was this translation helpful? Give feedback.
All reactions