Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ONE tag specific caches for datasets that are not associated to sessions #43

Closed
juhuntenburg opened this issue Dec 13, 2023 · 1 comment
Assignees

Comments

@juhuntenburg
Copy link
Contributor

juhuntenburg commented Dec 13, 2023

When trying to create the cache for Sebastian's release tag I get the error below which seems to be due to the fact that none of his datasets are Session associated (they are all Subject aggregates), so the dataframe in this line is empty

https://github.com/cortex-lab/alyx/blob/master/alyx/misc/management/commands/one_cache.py#L328

We should probably change the code so that datasets that are not associated to a Session will be picked up too. I am unsure if that requires changes to the cache itself, or just to the code generating the cache

(venv) root@e0966ff6424f:/var/www/alyx/alyx# python manage.py one_cache -v 2 --compress --qc --tag 2023_Q4_Bruijns_et_al -D s3://ibl-brain-wide-map-public/caches/openalyx/2023_Q4_Bruijns_et_al
Traceback (most recent call last):
  File "/var/www/alyx/venv/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'all_projects'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/var/www/alyx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/var/www/alyx/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/var/www/alyx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/var/www/alyx/venv/lib/python3.8/site-packages/django/core/management/base.py", line 448, in execute
    output = self.handle(*args, **options)
  File "/var/www/alyx/alyx/misc/management/commands/one_cache.py", line 132, in handle
    self.generate_tables(tables, export_qc=qc, tags=options.get('tag'))
  File "/var/www/alyx/alyx/misc/management/commands/one_cache.py", line 150, in generate_tables
    tbl, filename = self._save_table(generate_sessions_frame(**kwargs), table, dry=dry)
  File "/var/www/alyx/alyx/misc/management/commands/one_cache.py", line 40, in wrapper
    res = func(*arg, **kwargs)
  File "/var/www/alyx/alyx/misc/management/commands/one_cache.py", line 331, in generate_sessions_frame
    df['all_projects'] = df['all_projects'].map(lambda x: ','.join(filter(None, set(x))))
  File "/var/www/alyx/venv/lib/python3.8/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/var/www/alyx/venv/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'all_projects'
@k1o0 k1o0 self-assigned this Feb 26, 2024
@chris-langfield chris-langfield self-assigned this Sep 9, 2024
@chris-langfield chris-langfield removed their assignment Oct 7, 2024
@mayofaulkner
Copy link
Collaborator

PR cortex-lab/alyx#873

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants