Skip to content

Commit

Permalink
Do not remove uncompressed images; compression no longer depends on s…
Browse files Browse the repository at this point in the history 
…ync; project_mlapdv
  • Loading branch information
@k1o0
k1o0 committed Jul 25, 2023
1 parent 9ca107f commit 921167f
Show file tree
Hide file tree
Showing 6 changed files with 856 additions and 56 deletions.
333 changes: 316 additions & 17 deletions examples/loading_data/loading_multi_photon_imaging_data.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,15 @@
"* mpciMeanImage\n",
"* mpciStack\n",
"\n",
"## Finding sessions with imaging data"
"## Terminology\n",
"* **ROI** - A region of interest, usually a neuron soma, detected using an algorithm such as Suite2P.\n",
"* **FOV** - A field of view is a plane or volume covering a region of the brain.\n",
"* **Imaging stack** - Multiple FOVs acquired at different depths along the same z axis.\n",
"\n",
"## Finding sessions with imaging data\n",
"Sessions that contain any form of imaging data have an 'Imaging' procedure. This includes sessions\n",
"photometry, mesoscope, 2P, and widefield data. To further filter by imaging modality you can query\n",
"the imaging type associated with a session's field of view."
],
"metadata": {
"collapsed": false
Expand All @@ -35,23 +43,78 @@
"execution_count": null,
"outputs": [],
"source": [
"# Find mesoscope imaging sessions in V1\n",
"# Find mesoscope imaging sessions\n",
"import numpy as np\n",
"\n",
"from one.api import ONE\n",
"one = ONE()\n",
"assert not one.offline, 'ONE must be connect to Alyx for searching imaging sessions'\n",
"\n",
"# atlas_id, atlas_name\n",
"res = one.alyx.rest('sessions', 'list', procedures='Imaging', django='field_of_view__imaging_type__name,mesoscope')\n",
"eids = one.search(procedures='Imaging', django='field_of_view__imaging_type__name,mesoscope', atlas_acronym='AIp6b')\n",
"res = one.alyx.rest('fields-of-view', 'list', imaging_type='mesoscope', atlas_acronym='AIp6b')\n",
"fov00 = one.load_collection(res[0]['session'], '*' + res[0]['name'])\n",
"eid = 'a5550a8e-2484-4539-b7f0-8e5f829d0ba7'\n",
"fovs = one.alyx.rest('fields-of-view', 'list', imaging_type='mesoscope', atlas_id=187, session=eid)\n",
"query = 'field_of_view__imaging_type__name,mesoscope'\n",
"eids = one.search('sessions', 'list', procedures='Imaging', django=query, query_type='remote')"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"Sessions can be further filtered by brain region. You can filter with by Allen atlas name, acronym\n",
"or ID, for example:\n",
"\n",
"# Search by layer\n",
"# Only sessions with GCaMP mice\n",
"# Only sessions with single plane / N planes\n",
"JSON field = slice_id\n",
"field = part_of_stack\n"
"* `atlas_name='Primary visual area'`\n",
"* `atlas_acronym='VISp'`\n",
"* `atlas_id=385`"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"# Find mesoscope imaging sessions in V1, layer 2/3\n",
"query = 'field_of_view__imaging_type__name,mesoscope'\n",
"eids = one.search(procedures='Imaging', django=query, query_type='remote', atlas_acronym='VISp2/3')"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"The 'details' flag will return the session details, including a `field_of_view` field which contains\n",
"a list of each field of view and its location. All preprocessed mpci imaging data is in `alf/FOV_XX`\n",
"where XX is the field of view number. The `FOV_XX` corresponds to a field of view name in Alyx."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"eids, det = one.search(procedures='Imaging', django=query, query_type='remote', atlas_acronym='VISp2/3', details=True)\n",
"FOVs = det[0]['field_of_view']\n",
"print(FOVs[0])\n"
],
"metadata": {
"collapsed": false,
Expand All @@ -63,7 +126,126 @@
{
"cell_type": "markdown",
"source": [
"# List the number of fields of view (FOVs) recorded during a session"
"The ibllib AllenAtlas class allows you to search brain region descendents and ancestors in order to\n",
"find the IDs of brain regions at a certain granularity."
],
"metadata": {
"collapsed": false
}
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"# Search brain areas by name using Alyx\n",
"V1 = one.alyx.rest('brain-regions', 'list', name='Primary visual area')\n",
"for area in V1:\n",
" print('%s (%s: %i)' % (area['name'], area['acronym'], area['id']))\n",
"\n",
"\n",
"from ibllib.atlas import AllenAtlas\n",
"atlas = AllenAtlas()\n",
"\n",
"# Interconvert ID and acronym\n",
"V1_id = atlas.regions.acronym2id('VISp')\n",
"V1_acronym = atlas.regions.id2acronym(V1_id)\n",
"\n",
"# Show all descendents of primary visual area (i.e. all layers)\n",
"atlas.regions.descendants(V1_id)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"For more information see \"[Working with ibllib atlas](../atlas_working_with_ibllib_atlas.html)\"."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"## Loading imaging data for a given field of view\n",
"\n",
"For mesoscope sessions there are likely more than one field of view, not all of which cover the\n",
"area of interest. For mesoscope sessions it's therefore more useful to search by field of view instead.\n",
"Each field of view returned contains a session eid for loading data with."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"# Search for all mesoscope fields of view containing V1\n",
"FOVs = one.alyx.rest('fields-of-view', 'list', imaging_type='mesoscope', atlas_acronym='VISp')\n",
"# Download all data for the first field of view\n",
"FOV_00 = one.load_collection(FOVs[0]['session'], '*' + FOVs[0]['name'])\n",
"\n",
"# Search the fields of view for a specific session that took place in a given brain region\n",
"eid = 'a5550a8e-2484-4539-b7f0-8e5f829d0ba7'\n",
"FOVs = one.alyx.rest('fields-of-view', 'list', imaging_type='mesoscope', atlas_id=187, session=eid)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"## Loading imaging stacks\n",
"For mesoscope sessions the same region may be acquired at multiple depths. The plane at each depth\n",
"is considered a separate field of view and are related to one another through the stack object.\n",
"If a field of view was acquired as part of a stack, the `stack` field will contain an ID. You can\n",
"find all fields of view in a given stack by querying the 'imaging-stack' endpoint:"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"stack = one.alyx.rest('imaging-stack', 'read', id=FOVs[0]['stack'])\n",
"FOVs = stack['slices']\n",
"print('There were %i fields of view in stack %s' % (len(FOVs), stack['id']))\n"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"### List the number of fields of view (FOVs) recorded during a session"
],
"metadata": {
"collapsed": false
Expand Down Expand Up @@ -104,11 +286,11 @@
"outputs": [],
"source": [
"# Loading ROI activity for a single FOV\n",
"ROI_data_00 = one.load_collection(eid, 'alf/FOV_00', object=['mpci', 'mpciROI', 'mpciROITypes'])\n",
"ROI_data_00 = one.load_collection(eid, 'alf/FOV_00', object=['mpci', 'mpciROIs', 'mpciROITypes', 'mpciStack'])\n",
"print(ROI_data_00.keys())\n",
"\n",
"# Loading ROI activity for all FOVs\n",
"all_ROI_data = one.load_collection(eid, 'alf/FOV_*', object=['mpci', 'mpciROI', 'mpciROITypes'])\n",
"all_ROI_data = one.load_collection(eid, 'alf/FOV_*', object=['mpci', 'mpciROIs', 'mpciROITypes', 'mpciStack'])\n",
"print(all_ROI_data.keys())\n",
"print(all_ROI_data.FOV_00.keys())"
],
Expand All @@ -119,6 +301,123 @@
}
}
},
{
"cell_type": "markdown",
"source": [
"### Get the brain location of an ROI\n",
"The brain location of each ROI are first estimated using the surgical coordinates of the imaging window.\n",
"These datasets have an '_estimate' in the name. After histological alignment, datasets are created\n",
"without '_estimate' in the name. The histologically aligned locations are most accurate and should be\n",
"used where available."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"roi = 0 # The ROI index to lookup\n",
"final_alignment = 'brainLocationsIds_ccf_2017' in ROI_data_00['mpciROI']\n",
"key = 'brainLocationsIds_ccf_2017' if final_alignment else 'brainLocationsIds_ccf_2017_estimate'\n",
"\n",
"atlas_id = ROI_data_00['mpciROI'][key][roi]\n",
"print(f'ROI {roi} was located in {atlas.regions.id2acronym(atlas_id)}')"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"## Loading times\n",
"Timestamps for each frame are in seconds from session start and represent the time when frame acquisition started.\n",
"Typically a laser scans each voxel in the field of view in a line by line fashion (this may vary across apparatus and\n",
"in configuarations such as dual plane mode). Thus there is a fixed time offset between regions of interest.\n",
"The offset can be found in the mpciStack.timeshift.npy dataset and depending on its shape, may be per voxel or per\n",
"scan line."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"frame_times = ROI_data_00['mpci']['times']\n",
"roi_xyz = ROI_data_00['mpciROIs']['stackPos']\n",
"timeshift = ROI_data_00['mpciStack']['timeshift']\n",
"roi_offsets = timeshift[roi_xyz[:, len(timeshift.shape)]]\n",
"# An array of timestamps of shape (n_roi, n_frames)\n",
"roi_times = np.tile(frame_times, (roi_offsets.size, 1)) + roi_offsets[np.newaxis, :].T\n",
"\n",
"import matplotlib.pyplot as plt\n",
"roi_signal = ROI_data_00['mpci']['ROIActivityF'].T\n",
"roi = 2 # The ROI index to lookup\n",
"plt.plot(roi_times[roi], roi_signal[roi])\n",
"plt.xlabel('Timestamps / s'), plt.ylabel('ROI activity / photodetector units')"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"### Search for sessions with multi-depth fields of view (imaging stacks)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"query = 'field_of_view__stack__isnull,False'\n",
"eids, det = one.search(procedures='Imaging', django=query, query_type='remote', details=True)"
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%%\n"
}
}
},
{
"cell_type": "markdown",
"source": [
"### Search sessions with GCaMP mice\n",
"..."
],
"metadata": {
"collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
}
},
{
"cell_type": "markdown",
"source": [
Expand Down
Loading

0 comments on commit 921167f

Please sign in to comment.