Fix post on PyTables+Blosc2 NDim slicing #9
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
And slightly rejuggle the sentence for readability.
They were a result of a bug in the benchmark and differing versions of Blosc2 HDF5 support software.
To hint about them using equivalent versions of Blosc2 HDF5 filter code.
posts/pytables-b2nd-slicing.rst
Outdated
|
|
||
| Conclusions and future work | ||
| --------------------------- | ||
|
|
||
| The benchmarks above show how optimized Blosc2 NDim's two-level partitioning combined with direct HDF5 chunk access can yield considerable performance increases when slicing multi-dimensional Blosc2-compressed arrays under PyTables. However, the usual advice holds to invest some effort into fine-tuning some of the parameters used for compression and chunking for better results. We hope that this article also helps readers find those parameters. | ||
|
|
||
| It is worth noting that these techniques still have some limitations: they only work with contiguous slices (that is, with step 1 on every dimension), and on datasets with the same byte ordering as the host machine. Also, although results are good indeed, there may still be room for implementation improvement: for instance, the case of PyTables flat slicing via HDF5 filters (no b2nd) still looks strangely slow in comparison with the equivalent h5py's access; these future enhancements might as well carry over to the b2nd case for even better results. | ||
| It is worth noting that these techniques still have some limitations: they only work with contiguous slices (that is, with step 1 on every dimension), and on datasets with the same byte ordering as the host machine. Also, although results are good indeed, there may still be room for implementation improvement, for instance with extra code profiling and parameter adjustments. |
There was a problem hiding this comment.
suggestion: "for instance with extra code profiling and parameter adjustments" -> "but that will require extra code profiling and parameter adjustments"
FrancescAlted
approved these changes
Nov 23, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The benchmark shown in
posts/pytables-b2nd-slicing.rsthad some issues (see PyTables/PyTables:a0124f5) , this fix includes updated benchmark results and related fixes to the text and images.