Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

load_datasets fails when passed paths containing revisions #123

Open
k1o0 opened this issue Jun 17, 2024 · 1 comment
Open

load_datasets fails when passed paths containing revisions #123

k1o0 opened this issue Jun 17, 2024 · 1 comment
Assignees

Comments

@k1o0
Copy link
Collaborator

k1o0 commented Jun 17, 2024

Related to #65

from one.api import ONE
one_cache = ONE()
eid = '004d8fd5-41e7-4f1b-a45b-0d4ad76fe446'
one = ONE(base_url='https://alyx.internationalbrainlab.org/', mode='remote', cache_dir=one_cache.cache_dir.parent / 'foo')
dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', revision='2024-05-06')
files = one.load_datasets(eid, dsets, download_only=True)

This chains a call to list_datasets that in turn filters out the revision as it is set to None, which causes an object not found exception.

@oliche oliche self-assigned this Jul 1, 2024
@oliche
Copy link
Member

oliche commented Jul 3, 2024

Analysis and todos from today w/ Miles:

from one.api import ONE
from one.util import filter_datasets
one_cache = ONE()
eid = '004d8fd5-41e7-4f1b-a45b-0d4ad76fe446'
one = ONE(base_url='https://alyx.internationalbrainlab.org/', mode='remote', cache_dir=one_cache.cache_dir.parent / 'foo')

dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', revision='2024-05-06', filename='channels*')
files = one.load_datasets(eid, dsets, download_only=True, revisions='2024-05-06')  # this should fail if there is no dataframe
files = one.load_datasets(eid, dsets, download_only=True)  # this should work
files = one.load_datasets(eid, dsets, download_only=True, revisions='2024-10-10')  # this should fail as no dataset matches now



df_dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', revision='2024-05-06', filename='channels*', details=True)
assert df_dsets.revision.unique() == '2024-05-06'
files = one.load_datasets(eid, df_dsets, download_only=True)  # this should work
# todo raise if any of revisions / collections are provided with a dataframe (any filter)
# todo deactivate the filter datasets if a dataframe OR a relative path is provided
# todo raise if revision singular is provided as kwarg

dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', filename='channels*')   # TODO raise warning with mixed revisions  / mixed defaults (check existing warnings)
dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', revision='*', filename='channels*')  # doing the last before and then restricts to this revision


# mock checkfilesystems for tests mock.patch.object(one, 'download_datasets')
# check that behaviour is consistent with load_dataset
# todo load_dataset should raise if kwarg filters are provided with a dataframe record

k1o0 added a commit that referenced this issue Sep 25, 2024
k1o0 added a commit that referenced this issue Sep 26, 2024
k1o0 added a commit that referenced this issue Sep 26, 2024
k1o0 added a commit that referenced this issue Sep 26, 2024
k1o0 added a commit that referenced this issue Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants