Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/scanpy_mtx_compression #259

Open
wants to merge 20 commits into
base: develop
Choose a base branch
from

Conversation

irisdianauy
Copy link
Contributor

Add compression options to scanpy galaxy tools

@irisdianauy irisdianauy changed the title Feature/scanpy_mtx_compression [WIP] Feature/scanpy_mtx_compression Mar 3, 2022
@irisdianauy
Copy link
Contributor Author

Tests using the option have not yet been added

</data>
<data name="barcodes_10x" format="tsv" from_work_dir="barcodes.tsv" label="${tool.name} on ${on_string}: 10x barcodes">
<data name="barcodes_10x" label="${tool.name} on ${on_string}: 10x barcodes">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have multiple outputs, each attached to a potential value of the format.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 85534a6

<filter>export_mtx</filter>
<actions>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be that these actions require a higher profile than 18.01?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I'll check this.

@irisdianauy
Copy link
Contributor Author

Locally passing tests, but still getting the warning below for compression types gz, bz2, and zst.
WARNING:galaxy.model:Datatype class not found for extension 'zst'

@irisdianauy irisdianauy changed the title [WIP] Feature/scanpy_mtx_compression Feature/scanpy_mtx_compression Mar 10, 2022
@irisdianauy irisdianauy marked this pull request as ready for review March 10, 2022 06:48
@pcm32
Copy link
Member

pcm32 commented Aug 9, 2022

Great work @irisdianauy . I suspect that Galaxy doesn't support yet the zst compressed format. Do we need it forcefully?

<data name="genes_10x_bz2" label="${tool.name} on ${on_string}: 10x genes (bz2)" from_work_dir="genes.tsv.bz2"/>
<data name="barcodes_10x_bz2" label="${tool.name} on ${on_string}: 10x barcodes (bz2)" from_work_dir="barcodes.tsv.bz2"/>
</collection>
<collection name="mtx_zstd" type="list" format="zst">
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot find zst support in Galaxy, I would suggest that we comment this part. If we really need it, we could add it as a format to the galaxy codebase (it is called a datatype in Galaxy-speak).

@@ -1,10 +1,12 @@
<macros>
<token name="@TOOL_VERSION@">1.8.1+3</token>
<token name="@TOOL_VERSION@">1.8.1+4</token>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to move out of this pattern as it confuses the tools sorting. Please remove the + here and simply bump the Galaxy build number for the tools.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #266

Comment on lines +8 to 12
1.8.1+4+galaxy0: Upate to scanpy-scripts 1.1.5 (running scanpy ==1.8.1), including an option to compress MTX outputs.

1.8.1+3+galaxy0: Upate to scanpy-scripts 1.1.3 (running scanpy ==1.8.1), including a fix to MTX output and a bugfix for the Scrublet wrapper.

1.8.1+2+galaxy0: Upate to scanpy-scripts 1.1.2 (running scanpy ==1.8.1), including improved boolean handling for mito etc.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
1.8.1+4+galaxy0: Upate to scanpy-scripts 1.1.5 (running scanpy ==1.8.1), including an option to compress MTX outputs.
1.8.1+3+galaxy0: Upate to scanpy-scripts 1.1.3 (running scanpy ==1.8.1), including a fix to MTX output and a bugfix for the Scrublet wrapper.
1.8.1+2+galaxy0: Upate to scanpy-scripts 1.1.2 (running scanpy ==1.8.1), including improved boolean handling for mito etc.
1.8.5+galaxy0: Update to scanpy-scripts 1.1.5 (running scanpy ==1.8.1), including an option to compress MTX outputs.
1.8.1+3+galaxy0: Update to scanpy-scripts 1.1.3 (running scanpy ==1.8.1), including a fix to MTX output and a bugfix for the Scrublet wrapper.
1.8.1+2+galaxy0: Update to scanpy-scripts 1.1.2 (running scanpy ==1.8.1), including improved boolean handling for mito etc.

or perhaps just change the minor version.

<option value="zip">zip</option>
<option value="gzip">gzip</option>
<option value="bz2">bz2</option>
<option value="zstd">zstd</option>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if zst is not supported in Galaxy then we should remove it for now.

@pcm32
Copy link
Member

pcm32 commented Aug 2, 2023

There are still a number of comments from my last review that haven't been addressed.

@pcm32
Copy link
Member

pcm32 commented Aug 2, 2023

In particular the versioning pattern is a no go. Also, this should be rebased with Dev please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants