fix: refine filetype detection #3828
Open
+211
−478
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The eight file-types based on CFB or ZIP compound files can be detected with 100% accuracy (or at least 99.999%). Try this strategy first, ignoring the unreliable content-type for these file-types. This fixes a problem where a CSV file with an asserted XLS file-type was mistakenly typed as XLS and failed partitioning.
Looks like we need some work to upgrade ruff past 0.4.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a bug where a CSV file with asserted content-type
application/vnd.ms-excelwas incorrectly identified as an XLS file and failed partitioning.Additional Context
The
content_typeargument to partitioning is often authored by the client system (e.g. Unstructured SDK) and is both unreliable and outside the control of the user. In this case the.csv -> XLSmapping is correct for certain purposes (Excel is often used to load and edit CSV files) but not for partitioning, and the user has no readily available way to override the mapping.XLS files as well as seven other common binary file types can be efficiently detected 100% of the time (at least 99.999%) using code we already have in the file detector.
content_typewhen it matches any of those types. This prevents the problem seen in the bug where the asserted content type was used to determine the file-type.