Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More linear pipeline structure for supporting multiple file types #5

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

adamrtalbot
Copy link
Contributor

@adamrtalbot adamrtalbot commented Oct 2, 2024

This PR changes the pipeline structure so that it uses a more linear set of steps and can be controlled.

Changes:

  • Create FASTQ and create files composed of zeros are performed at the start of the pipeline
  • They are fed into subsequent processes for file manipulations
  • The two file types are manipulated together but can be switched on or off based on flags

Advantages:

  • This is a more flexible structure which means we can extend it to new file types when they are necessary
  • We can perform the file manipulations on larger or smaller files by enabling or disabling features
  • We have granular control over more aspects of the pipeline

Disadvantage

  • Separate file operations cannot be performed on each file type, since they are mixed. We can disable individual processes though.
  • It may be necessary to launch multiple pipelines to test multiple things instead of one test with multiple hypotheses
  • If statements (yuck)

Supercedes #3, closes #4

@adamrtalbot adamrtalbot changed the title Flexible pipeline structure More linear pipeline structure for supporting multiple file types Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[feature] Add ability to configure which file types are included
1 participant