Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add preprocessed as remove intermediate file option #79

Open
JoeZiminski opened this issue Aug 10, 2023 · 0 comments
Open

[Feature] Add preprocessed as remove intermediate file option #79

JoeZiminski opened this issue Aug 10, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@JoeZiminski
Copy link
Member

preprocessed was an option to delete_intermediate_files but becase the sorting_data class holds Spikeinterface recording objects that are not closed, an error occurs when trying to delete e.g.:

[WinError 32] The process cannot access the file because it is being used by another process: 'C:\\data\\ephys\\test_data\\steve_multi_run\\1119617\\time-short\\derivatives\\1119617\\1119617_LSE1_shank12_g0\\preprocessed\\si_recording\\traces_cached_seg0.raw'      ```

As a workaround, the hacky below code was used however it is proving unstable

if "preprocessed" in delete_intermediate_files:
    # remove the existing link to the preprocessed data binary.
    # wait time of 5 s is arbitrary.
    # TODO: this feels very hacky. Can unlink the memory map from the segment?
    # Ask SI.
    del sorting_data.data
    gc.collect()
    time.sleep(5)

    if sorting_data.get_preprocessing_path(run_name).is_dir():
        shutil.rmtree(sorting_data.get_preprocessing_path(run_name))

The correct solution is to use SI to close the memmaped file link. However, I can't find on their API - make an issue.
@JoeZiminski JoeZiminski added the enhancement New feature or request label Aug 10, 2023
@JoeZiminski JoeZiminski modified the milestone: 0.0.1 Aug 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant