Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Start thinking about parallelising over sessions / runs #88

Open
JoeZiminski opened this issue Aug 11, 2023 · 1 comment
Open
Labels
enhancement New feature or request

Comments

@JoeZiminski
Copy link
Member

Now that #72 is beggining to handle full project organisation with subject > session > run levels, it is worth thinking about parallelisation.

The top level will always be the subject. However, if running 3 subjects sorting separately, we will want to parallalise. This will be over nodes using SLURM if GPU required, or other cores otherwise.

For now, keep the code structured as for loops over subject, runs. This will be the easiest to play around with parallelisation strategies. Following this, if the jobs of per-session processing and per-run processing are overlapping, create class scheme.

@JoeZiminski JoeZiminski added the enhancement New feature or request label Aug 11, 2023
@JoeZiminski
Copy link
Member Author

This is very relevant to #64 because SI also implements dask / multiprocessing backends for sorting which it would be great to leverage, and determine how these will play if we are already parallelising over subject / runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant