Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Neglect aspects of the pickle state file if workflow, parameters or inputs files change their md5 #28

Open
pcm32 opened this issue Jul 20, 2022 · 1 comment

Comments

@pcm32
Copy link
Member

pcm32 commented Jul 20, 2022

Currently on changes of the execution of a workflow on a directory, some changes in the workflow, parameters or other files might not be picked up in favour of the pickle state saved. Elements of the pickle state should be made obsolete if the originated element on disk shows a change. This could be either checking the MD5s of the files (and keeping that) or relying on the modification dates of the input files in general (although this could lead to issues if an older file is moved into the directory, so maybe not the best approach).

Most likely the change should happen here:

https://github.com/ebi-gene-expression-group/galaxy-workflow-executor/blob/develop/run_galaxy_workflow.py#L152

so that if the state is None or the md5 for the workflow file is different (which would need to be stored in the state before hand), then this would need to be read from disk (because the file might have changed).

@pcm32
Copy link
Member Author

pcm32 commented Jul 20, 2022

Alternatively, one could just warn the user that the files seemed changed and that he/she should either delete the state file or re-instate the older files.

But it still implies keeping some record of the checksum of the files to be able to detect this.

Claptar added a commit that referenced this issue Jul 28, 2022
* Added saving md5 of workflow and parameters files inside of ExecutionState class
* Added comparison of new and previous md5's
Claptar added a commit that referenced this issue Jul 28, 2022
This was referenced Aug 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant