Skip to content

Latest commit

 

History

History
1149 lines (790 loc) · 35.8 KB

release_notes.md

File metadata and controls

1149 lines (790 loc) · 35.8 KB

Release Note 2.40.0

features

  • iblsorter >= 1.9 sorting tasks with waveform extraction and channel sorting
  • s3 patcher prototype

Release Note 2.39.0

features

  • register inputs without rename in registerSnapshots
  • iterate only plane folders in Suite2P
  • removed old pipeline

bugfixes

  • NeuromodulatorCW is a subclass of Advanced CW regarding debiasing
  • Behaviour QC: StimFreeze QC due to indexing errors, ITI duration and negative feedback delay for nogo trials
  • Camera QC fixes

2.39.1

  • Bugfix: brainbox.metrics.single_unit.quick_unit_metrics fix for indexing of n_spike_below2

2.39.2

  • Bugfix: routing of protocol to extractor through the project repository checks that the target is indeed an extractor class.

Release Note 2.38.0

features

  • Support running large pipeline jobs within multiple python environments
  • Extraction and QC of of task pause periods
  • Register animated GIF snapshots and tiff images
  • Load waveforms low level changes in spike sorting loader
  • Loading of additional channel object attributes (qc labels) in spike sorting loader
  • Stimulus times now extracted in a more robust manner

bugfixes

  • Fix in training plots when no trials in block

Release Note 2.37.0

features

  • Add code in preparation for retirement of old training and ephys local server pipelines

bugfixes

  • Change handling of trials extraction of iblrigv8 sessions such that pregenerated session is not used

Release Note 2.36.0

features

  • Adding spike sorting iblsort task

Release Note 2.35.0

features

  • Adding LightningPose task

bugfixes

  • SessionLoader can now handle trials that are not in the alf collection
  • Extraction of trials from pre-generated sequences supports iblrigv8 keys

2.35.1

  • Ensure no REST cache used when searching sessions in IBLRegistationClient

2.35.2

  • Flag to allow session registration without water administrations
  • Support extraction of repNum for advancedChoiceWorld
  • Support matplotlib v3.9; min slidingRP version now 1.1.1

2.35.3

  • Use correct task for timeline acquisitions in make_pipeline

Release Note 2.34.0

features

  • Task assert_expected_input now take into account revisions
  • Camera qc and video motion now take into account dynamic pipeline folder

bugfixes

  • Typo in raw_ephys_data documentation
  • oneibl.register_datasets accounts for non existing sessions when checking protected dsets

2.34.1

  • Ensure mesoscope frame QC files are sorted before concatenating
  • Look for SESSION_TEMPLATE_ID key of task settings for extraction of pre-generated choice world sequences
  • Download required ap.meta files when building pipeline for task_qc command

Release Note 2.33.0

features

  • Datasets no longer registered by default if protected, need to force registration
  • Tasks now allows datasets to be registered before qc computation
  • Histology channel upload now reads in channel map from data when available

bugfixes

  • PostDLC tasks looks in alf folder for lick datasets

Release Notes 2.32

features

  • SDSC patcher automatically support revisions

other

  • Add extra key to alignment qc with manual resolution for channel upload

2.32.3

  • FpgaTrials supports alignment of Bpod datasets not part of trials object
  • Support chained protocols in BehaviourPlots task

2.32.4

  • Add support for variations of the biaseCW task in the json task description

2.32.5

  • Minor fixes to IBL registration client, including use of SESSION_END_TIME key

Release Notes 2.31

features

  • Training status uses new extractor map
  • Refactor neurodsp to ibldsp
  • ITI qc check for iblrig v8
  • Support habituationChoiceWorld extraction in iblrig v8.15.0

bugfixes

  • NP2 waveforms extracted with correct dtype
  • Sorted cluster ids in single unit metrics

Release Notes 2.30

features

  • Task QC viewer
  • Raw ephys data loading documentation

other

  • Pandas 3.0 support

Release Notes 2.29

features

  • Added raw data loaders and synchronisation tools in brainbox.io.one.SpikeSortingLoader, method ssl.raw_electrophysiology()

Release Notes 2.28

features

  • Added ibllib.pipes.dynamic_pipeline.get_trials_tasks function

bugfixes

  • Fix ibllib.io.extractors.ephys_fpga.extract_all for python 3.8

other

  • Change behavior qc to pass if number of trials > 400 (from start) can be found for which easy trial performance > 0.9

2.28.1

  • Typo in ibllib.pipes.video_tasks.EphysPostDLC class
  • ibllib.io.raw_data_loaders.patch_settings works with iblrigv8 settings files

2.28.2

  • Fix loading of personal projects extractor map

Release Notes 2.27

features

  • Add full video wheel motion alignment code to ibllib.io.extractors.video_motion module
  • Change FPGA camera extractor to attempt wheel alignment if audio alignment fails
  • Flexible FpgaTrials class allows subclassing for changes in hardware and task
  • Task QC thresholds depend on sound card
  • Extractor classes now return dicts instead of tuple
  • Support extraction of habituationChoiceWorld with FPGA
  • New IBLGlobusPatcher class allows safe and complete deletion of datasets

bugfixes

  • Fix numpy version dependent error in io.extractors.camera.attribute_times
  • Fix for habituationChoiceWorld stim off times occuring outside of trial intervals
  • Improvements to Timeline trials extractor, especially for valve open times
  • trainingPhaseChoiceWorld added to Bpod protocol extractor map fixture
  • Last trial of FPGA sessions now correctly extracted
  • Correct dynamic pipeline extraction of passive choice world trials

2.27.1

  • Correct handling of missing TTLs in FpgaTrialsHabituation

other

  • Removed deprecated pyschofit module
  • Deprecated oneibl.globus module in favour of one.remote.globus
  • Deprecated qc.task_extractors in favour of behaviour pipeline tasks

Release Notes 2.26

features

  • Deprecate ibllib.atlas. Code now contained in package iblatlas

bugfixes

2.26.1

  • fix typo in criterion_ephys where lapse high for 0.8 blocks only passed when non-zero.

Release Notes 2.25

features

  • Training status pipeline now compatible with dynamic pipeline
  • Dynamic DLC task using description file
  • Full photometry lookup table

bugfixes

  • fix for untrainable, unbiasable don't repopulate if already exists

2.25.1

  • relax assertion on Neuropixel channel mappings to allow for personal projects

2.25.2

  • listing of all collections does not skip repeat task protocols anymore for copy/extraction

Release Notes 2.23

Release Notes 2.23.1 2023-06-15

features

  • split swanson areas

bugfixes

  • trainig plots
  • fix datahandler on SDSC for ONEv2

Release Notes 2.23.0 2023-05-19

  • quiescence period extraction
  • ONEv2 requirement

Release Notes 2.22

Release Notes 2.22.3 2023-05-03

same as 2.22.2 (I messed up the release sorry)

Release Notes 2.22.2 2023-05-03

bugfixes

  • training plots

features

  • can change download path for atlas

Release Notes 2.22.1 2023-05-02

bugfixes

  • use session path in training status task

Release Notes 2.22.0

features

  • trials extractors support omissions trials from neuromodulator tasks
  • SpikeSortingLoader and EphysSessionLoader utils functions to load related objects such as drift
  • Training status tasks upload training date and session to subject json
  • Query training criterion function added to brainbox.training
  • Swanson vector - option to annotate top/bottom 5 regions based on values, or pass in a list of acronyms
  • EphysQC can run without connection to ONE

Release Notes 2.21

Release Notes 2.21.3 2023-03-22

features

  • show atlas names in swanson plot
  • allow user to define mask regions in swanson plot

bugfixes

  • register_session: handle lack of taskData in passive sessions

Release Notes 2.21.2 2023-02-24

bugfixes

  • get_lab function now gets lab name from session path subject name
  • create_jobs now returns pipelines

Release Notes 2.21.1 2023-02-21

bugfixes

  • remove unused jupyter and jupyterhub requirements
  • fix mapping for atlas svg plots

Release Notes 2.21.0 2023-02-14

features

  • support for multiple task protocols within a session
  • extract protocol period from spacer signals
  • function for changing subject, collection, number or date in a settings file
  • function to retrieve task protocol number from experiment description
  • dataset type validation supports null filename patterns
  • oneibl.registration uses one.registration client as superclass
  • experiment description files are created and registered in legacy pipelines
  • QC sign-off keys are added to session JSON field
  • class-based note template API for attaching session notes for QC sign-offs
  • protocol and procedures now taken from experiment description file
  • handle truncated first spacer in passive extraction
  • fix the Perirhinal area layer color in Allen Atlas
  • fix region volume computation
  • vectorised swanson flatmap
  • brainbox trial ITI loader
  • vectorised atlas plots

Release 2.20

Release Notes 2.20.0 2023-01-23

features

  • ephys session loader
  • bwm release documentation

bugfixes

  • mock gpu lock in tests
  • use cluster_ids in metrics

Release Notes 2.19

Release Notes 2.19.0 2022-11-30

features

  • camera qc allows for string values
  • deprecate brainbox.io.spikeglx.stream instead use Streamer
  • copy logics take into account presence of experiment description files
  • option for error bars in training plots

bugfixes

  • cam qc for dynamic pipeline uses sync type to determine number of cameras

Release 2.18

Release Notes 2.18.0 2022-11-23

features

  • brainbox.modeling removed
  • brainbox.io.one.load_trials_df removed, use one.load_object(...).to_df() instead
  • AWS DataHandler refactored (ibllib.oneibl.aws removed)
  • raise error when saving an empty dataset during extraction
  • removed deprecated dsp packages (ibllib.dsp, ibllib.ephys.neuropixel, ibllib.io.spikeglx)
  • register datasets as a revision if protected on Alyx

bugfixes

  • frameData and timestamps both optional datasets for camera QC
  • for 3B probes don't glob the sync files as fails for NP2 case
  • support iblutil 1.4 (iblutil.util.get_logger -> iblutil.util.setup_logger)

Release Notes 2.17

Release Notes 2.17.2 2022-10-14

  • Compute wheel velocity using a low-pass filter instead of a Gaussian window smoothing
  • Save widefield output plots in png format
  • Widefield Sync task uses sync_collection argument
  • Improve boundary display for atlas slice plots

Release Notes 2.17.1 2022-10-04

  • adjust ONE-api requirement to redownload on AWS backend when md5sum mismatch

Release Notes 2.17.0 2022-10-04

  • units quality metrics use latest algorithms for refractory period violations and noise cut-off

Release Notes 2.16

Release Notes 2.16.1 2022-09-28

bugfixes

  • photometry extraction: recover from corrupt DAQ signal and reversed polarity of voltage pulses

Release Notes 2.16.0 2022-09-27

features

  • swanson flatmap: the algorithm to propagate down the hierarchy has been refined

bugfixes

  • set exists flag to false for all data repos when registering datasets with tasks

Release Notes 2.15

Release Notes 2.15.3 - 2022-09-26

  • SessionLoader error handling and bug fix

Release Notes 2.15.2 - 2022-09-22

  • extraction pipeline: fix unpacking of empty arguments field from alyx dict that prevents running task

Release Notes 2.15.1 - 2022-09-21

  • atlas: gene-expression backend and MRI Toronto atlas stretch and squeeze factors (Dan/Olivier)
  • FDR correction (Benjamin-Hochmann) to correct for multiple testing optional (Guido)
  • SpikeSortingLoader can be used with ONE local mode (Julia)

Release Notes 2.15.0 2022-09-20

features

  • dynamic pipelines creation: if the acquisition description file exists, task sequences get created
  • session registration procedures and projects, only add a couple of fields in the json by default
  • new modalities:
    • photometry extraction (Mainen lab)
    • widefield extraction (Churchland lab)

bugfixes

  • Spike sorting task: parse new pykilosort log format
  • Session loader

Release Notes 2.14

Release Notes 2.14.0 2022-08-17

  • Adding brainbox.io.one.SessionLoader for standardized loading of session data
  • Changes to TaskQC, VideoQC and DLCQC thresholds and aggregation
  • Unfreezing numpy version
  • Adding probabilistic brain regions lookup in ibllib.atlas.atlas.BrainAtlas.get_labels
  • Removing two voxelless areas from beryl atlas
  • Minor updates to audio events detection and audio welchogram

Release Notes 2.13

Release Notes 2.13.6 2022-08-02

  • Hotfix: don't overwrite full settings if iblrig on untagged version

Release Notes 2.13.5 2022-07-27

  • Hotfix: pseudo session biased generation contrast distribution

Release Notes 2.13.4 2022-07-22

  • Hotfix: Density displays had non-existing colormaps in snapshots QC tasks
  • Generate extra training plots based on psychometric curves

Release Notes 2.13.3 2022-07-01

  • Hotfix: fix previous hotfix with incorrect package version number

Release Notes 2.13.2 2022-07-01

  • Hotfix: datahandler sets local paths in init

Release Notes 2.13.1 2022-07-01

  • Hotfix: globus imports were mixed one.globus and one.remote

Release Notes 2.13.0 2022-06-30

  • Deprecated ibllib.version
  • Fix Globus patcher
  • Add SpikeSorting Loader samples2times function
  • Fix atlas.BrainCoordinate.xyz2i functions to not quietly wrap indices out of volume bounds.
  • Set jobs to Held if parent jobs are Started or Abandoned as well
  • Reverse matplotlib colorbars in density displays

Release Notes 2.12

Release Notes 2.12.2 2022-05-27

  • Fixes to plotting in training_status

Release Notes 2.12.1 2022-05-26

  • ibllib.pipes.training_status: pipeline to compute training status of mice on local servers, new TrainingStatus task (Mayo)
  • Fix swanson regions (Olivier)
  • Swanson to Beryl mapping

Release Notes 2.12.0 2022-05-10

  • ibllib.atlas: add the Swanson flatmap backend (Olivier)
  • ibllib.io.extractors: output of task extractions are trial tables, not individual datasets (Miles)
  • Documentation: data release examples (Mayo)
  • ibl-neuropixel new repository contains ibllib.dsp, illlib.ephys.neuropixel and ibllib.io.spikeglx modules (Olivier)
  • brainbox.task.closed loop get impostor targets to evaluate null distribution (Brandon)
  • minimum supported version of Python is 3.8 (Michele)

Release Notes 2.11

Release Notes 2.11.1 2022-04-12

  • Set video compression jobs to priority 90
  • Set jobs to Held if parents are Waiting

Release Notes 2.11.0 2022-04-08

  • brainbox.io.one.SpikeSortingLoader: option to load using collection argument
  • Restructuring of how jobs are run on local servers, run large jobs as service

Release Notes 2.10

Release Notes 2.10.6 2022-03-15

  • Allow parent tasks to be 'Incomplete' to run task on local server
  • Change one base_rul for dlc_qc_plot on cortexlab

Release Notes 2.10.5 2022-03-11

  • Fix moot release accident

Release Notes 2.10.4 2022-03-11

  • Data handler connects to correct alyx database on cortexlab

Release Note 2.10.3 2022-03-09

  • Fixes to EphysPostDLC
  • Small change to storing in dsp.voltage.decompress_destripe_cbin function

Release Notes 2.10.2 2022-02-28

  • Small fixes to local server task queues

Release Notes 2.10.1 2022-02-22

  • Authenticate alyx user in Task class
  • Some fixes to make dlc_qc_plot in EphysPostDLC more reliable
  • SpikeGlx:
    • supports reading of flat binary files without meta data
  • atlas.Regions:
    • add support for region ordering in BrainRegions object
    • bugfix in transposing of RGB image data for coronal slice plots
  • voltage: decompress cbin
    • add support for custom spikeglx.Reader

Release Notes 2.10.0 2022-02-11

  • Fix in EphysDLC task to not return multiple copies of outputs
  • Loading examples for different IBL data types
  • Fix for probe syncing when Nidq and probe pulses don't match
  • Account for new ONE tables in ond datahandler
  • Add bad channels plots into RawEphysQc task

Release Notes 2.9

Release Notes 2.9.1 2022-01-24

  • deprecation warnings and documentation for spike sorting loading method
  • bugfix: remove lru_cache on AllenAtlas class for iblviewer

Release Notes 2.9.0 2022-01-24

  • Adding EphysDLC task in ephys_preprocessing pipeline
  • NOTE: requires DLC environment to be set up on local servers!
  • Fixes to EphysPostDLC dlc_qc_plot

Release Notes 2.8

Release Notes 2.8.0 2022-01-19

  • Add lfp, aprms, spike raster and behaviour report plots to task infastructure
  • Computation of apRMS in decompress_destripe_cbin before conversion to normalised units
  • Add SpikeSortingLoader class in brainbox.io.one

Release Notes 2.7

Release Notes 2.7.1 2022-01-05

  • Fixes and better logging for EphysPostDLC task

Release Notes 2.7.0 2021-12-20

  • Remove atlas instantiation from import of histology module
  • Write electodeSites dataset at end of histology and spikesorting pipelines if insertion resolved
  • Removal of native test train support and improved documentation / tests for GLMs
  • LFP destriping functions
  • Version number is now obtained from ibllib/init.py

Release Notes 2.6

Release Notes 2.6.0 2021-12-08

  • New ReportSnapshot class
  • DLC QC plots, as part of EphysPostDLC task
  • BadChannelsAP plots for ephys QC
  • Fix typo in camera extractor

Release Notes 2.5

Release Notes 2.5.1 2021-11-25

  • SpikeSorting task overwrites old tar file on rerun

Release Notes 2.5.0 2021-11-24

  • Snapshot class to register images as notes in Alyx
  • Ephys pipeline: RawEphysQC: outputs channels labels as part of pipeline
  • Ephys pipeline: spike sorting
    • dsp.voltage: destripe detects bad channels and interpolate them
    • synchronisation 3B: gives a 10% leeway to throw exception
    • spikes sorting task outputs the RMS plot

Release Notes 2.4

Release Notes 2.4.0

  • Setting tasks to Waiting if they encountered lock (status -2)
  • Setting tasks to Incomplete if they return status -3
  • Completed tasks set held dependent tasks to waiting
  • Adding PostDLC task to compute pupil diameters, licks and DLC QC

Release Notes 2.3

Release Notes 2.3.2 2021-11-08

  • Trial wheel extraction: use alternative sync method when first on fails
  • bugfix: timer for tasks was returning None

Release Notes 2.3.1 2021-11-5

  • Add setup method in tasks.py into try except to catch globus download errors

Release Notes 2.3.0 2021-11-4

  • Add input and output signatures to all ephys tasks
  • Add datahandler to task to download and upload data based on location where task is run
  • Spike sorting and EphysVideoSyncQc download data on local servers if not available
  • brainbox.io.one load_spike_sorting_fast: bugfix returns acronyms
  • creates sequence files for spikesorting
  • GPU tasks have a lock - local data handler doesn't instantiate one

Release Notes 2.2

Release Notes 2.2.1 2021-11-02

  • allows more than 2 probes in ephys computer probe creation

Release Notes 2.2.0 2021-10-29

  • brainbox.io.one load_spike_sorting fast: merge channel info in clusters
  • brainbox.io.one generic function to interpolate channels after alignment
  • dsp: spike detection by voltage thresholding and cadzow filtering docs
  • ibllib.io.spikeglx: get geometry from metadata
  • RawEphysQC outputs basic detection spike rates
  • ibllib.atlas.regions: add new mapping cosmos and revise Beryl

Release Notes 2.1

Release Notes 2.1.0 2021-10-05

  • destriping as pykilosort internal pre-processing
  • NP2 probe framework for splitting shanks and LFP band
  • Extension of task module to rerun from different locations

Release Notes 2.1.1 2021-10-06

  • RawEphysQC tasks computes median RMS from samples for .ap data (stored in _iblqc_ephysChannels.RMS)
  • New EphysQC class

Release Notes 2.1.2 2021-10-14

  • Fix issue with RawEphysQC that was not looking in local Subjects folder for data
  • Fix ensure_required_data in DlcQc

Release Notes 2.1.3 2021-10-19

  • Split jobs.py run function in two, one running large tasks (video compression, dlc, spike sorting), one the rest
  • Ensure RawEphysQC runs both probes if one fails

Release Notes 2.0

Release Notes 2.0.1 2021-08-07

  • pykilosort error handling

Release Notes 2.0.2 2021-08-31

  • passive extraction robust to frame2ttl flickers

Release Notes 2.0.3 2021-09-03

  • pykilosort bugfix after low yield results

Release Notes 2.0.4 2021-09-10

  • ephys trials extraction when audio FPGA starts on up state

Release Notes 2.0.5 2021-09-13

  • pykilosort pipeline: output correct version number / fix log file name

Release Notes 2.0.6 2021-09-14

  • extraction fixes: passive extraction spacers and camera times format
  • small bugfix for the sequential selection

Release Notes 2.0.7 2021-09-15

  • ephys trials extraction: audio from FPGA: up-state and TTLs cleanup from camera wiring
  • passive extraction: improved spacers detection
  • ephyscellsqc: bugfix on probe creation for pykilosort subfolder
  • task version 6.2.5 specific audio extractor
  • camera times: index by video length when GPIO unstable

Release Notes 2.0.6 - 2021-09-14

  • extraction fixes: passive extraction spacers and camera times format
  • small bugfix for the sequential selection

Release Notes 2.0.5 - 2021-09-13

  • pykilosort pipeline: output correct version number / fix log file name

Release Notes 2.0.4 - 2021-09-10

  • ephys trials extraction when audio FPGA starts on up state

Release Notes 2.0.3 - 2021-09-03

  • pykilosort bugfix after low yield results

Release Notes 2.0.2 - 2021-08-31

  • passive extraction robust to frame2ttl flickers

Release Notes 2.0.1 - 2021-08-07

  • pykilosort error handling

Release Notes 2.0.0 - 2021-08-04

  • ONE2 released on the master branch
  • Pykilosort is the new default spike sorter in the pipeline

Release Notes 1.12 (not released yet)

Release Notes 1.12.0

  • oneibl and alf now deprecated; moved to separate repository
  • other ibllib miscellanea moved to iblutil repository
  • test database is now configurable with an env var

Release Notes 1.11

Release Notes 1.11.0

  • brainbox.modeling: New API for using linear and poisson models, doing sequential feature selection.
  • bugfix: spike sorting continues even if probe information not entered (personal projects)
  • TaskQC iteration: aggregation excludes iti delays, frame2ttl clean signal used to compute QC instead of raw

Release Notes 1.10

Release Notes 1.10.4 - 2021-05-15

  • optogenetics for UCL task variant
  • check Nvidia status before launching spike sorting

Release Notes 1.10.2 / 1.10.3

  • fix public one params

Release Notes 1.10.1

  • fix ks2 logging output
  • set default one params to public credentials

Release Notes 1.10.0 - 2021-05-04

  • TaskQC: exclude stim_freeze from overall session qc
  • Get modality from task type to include personal projects
  • Bugfix Atlas ccf coordinates order
  • Tool to label Critical sessions / insertions reasons

Release Notes 1.9

Release Notes 1.9.1 - 2021-04-19

  • Successful ONE setup on instantiation when params file doesn't exist

Release Notes 1.9.0 - 2021-04-15

  • Video QC optimization
  • New task types for widefield imaging
  • Add revision and default dataset to register dataset
  • Fix for camera extraction failure for novel protocols
  • CameraQC wheel alignment check more robust to short videos and large alignment failures
  • Parameters raises FileNotFound error if no defaults are provided

Release Notes 1.8

Release Notes 1.8.0 - 2021-03-30

  • opto-ephys tasks
  • DLC QC
  • Do not block session creation if error

Release Notes 1.7

Release Notes 1.7.0 - 2021-03-16

  • Removed deprecated numpy dtypes
  • Fix trajectory query bug
  • GPIO pin states loaded as bool array
  • Machine info and log append in Task class
  • Only delete session flag file after processing
  • Extractor support for widefield imaging protocol

Release Notes 1.6

Release Notes 1.6.2 - 2021-03-18

  • hotfix: fix test_modelling error after removing pytorch

Release Notes 1.6.1 - 2021-03-17

  • hotfix: maintain compatibility with Python 3.7

Release Notes 1.6.0 - 2021-03-16

  • ibllib.atlas: backend region mapping is a lateralized version of the Allen atlas "Allen-lr" The default behaviour in the Atlas is to remap according to the original mapping "Allen"
  • camera timestamps: extraction of the exact camera time stamps by cross-examination of bonsai timestamps, audio pulses and camera pulses

Release Notes 1.5

Release Notes 1.5.39

  • ephys extraction: remove short TTL pulses of frame2ttl in task extraction

Release Notes 1.5.38

  • ibllib.atlas.regions remapping option
  • optogenetics pybpod: extraction of laser probabilities

Release Notes 1.5.37

  • MANIFEST.in fix
    • Added fixtures file for passive protocol extraction

Release Notes 1.5.36

  • Amplitudes fix:
    • sync_probes doesn't require raw binary file and looks for meta-data files instead
    • tar file of intermediate spike sorting results gets uploaded on flatiron
  • Ephys Task extraction: ephys extraction doesn't fail when bpod started before ephys

Release Notes 1.5.35

  • histology: brain atlas can handle insertion on the sides
  • optogenetics dataset type _ibl_trials.laser_stimulation for training sessions

Release Notes 1.5.34

  • spikeglx analog sync thresholding removes DC offset option
  • ephys FPGA behaviour extraction: bpod sync with FPGA assumes possible missing FPGA fronts

Release Notes 1.5.32

  • Ephys pipeline:
    • passive extractor
    • units QC

Release Notes 1.5.31/1.5.33 Hotfix

  • add the available space of system and raid volumes in local servers reports.

Release Notes 1.5.30

  • following flatiron server change, add auto-redirect to https

Release Notes 1.5.29

  • add widefieldChoiceworld tasks for pipeline

Release Notes 1.5.28

  • add opto laser tasks for pipeline

Release Notes 1.5.27

  • register ch when mtscomp runs properly
  • probes_description runs even if .cbin doesn't exist

Release Notes 1.5.26 Hotfix

  • allows dataset registration on errored task

Release Notes 1.5.25 Hotfix

  • ks2task import conflict fix

Release Notes 1.5.24 Hotfix

  • Ks2 task does not depend on ephys pulses

Release Notes 1.5.23

  • Ephys mtscomp
    • ks2 task registers first probe even if one failing
    • mtscomp task register .ch and .meta even if .cbin doesn't exist
    • move ibllib tests to tests_ibllib
    • brainbox atlas plot functions

Release Notes 1.5.22

  • Ephys extraction:
    • synchronisation between probes computed in the ephysPulses job
    • spike sorting resync done directly after KS2 output
    • unit-based metrics have their own task

Release Notes 1.5.21: hotfix

  • create local server tasks only on raw_session.flag

Release Notes 1.5.20

  • ephys alignment QC
  • hotfix: ibl errors inherit Exception, not BaseException
  • hotfix: partial qc task extractor keeps FPGA stim times

Release Notes 1.5.19

  • create tasks looks for create_me.flags

Release Notes 1.5.18

  • add Karolina's optogenetics tasks for extractions

Release Notes 1.5.17

  • histology probe QC pipeline and final locations dataset export

Release Notes 1.5.16 Hotfix

  • numpy needs upgrading >= 1.18

Release Notes 1.5.15 Hotfix

  • session creation skips alyx procedure for unknown task protocol (custom projects)

Release Notes 1.5.14

  • task extraction:
    • Habituation QC
    • ephys extraction StimOffTimes fix

Release Notes 1.5.13

  • ibllib.atlas
    • allen csv Atlas part of package is not installed in dev mode
    • improved slicing performance
  • ephys extraction: mtscomp registers ch file on run and re-runs bis

Release Notes 1.5.12 Hotfix

  • mtscomp registers ch file on run and re-runs

Release Notes 1.5.11 Hotfix

  • ffmpeg nostdin option as jobs were stopped in background on a server

Release Notes 1.5.10

  • QC base class
  • Support for task QC on FPGA data
  • TaskQC run during task extraction

Release Notes 1.5.9

  • local server: catches error when subject is not registered in Alyx
  • ibllib.atlas.AllenAtlas
    • re-ordered the volumes in c-order contiguous ml-ap-dv efficient coronal shapes
    • top/bottom surface extraction

Release Notes 1.5.8 Hotfix

  • Ephys extraction SyncSpikeSorting: specify different dir for ks2 ouput and raw ephys data

Release Notes 1.5.7 Hotfix

  • Ephys extraction ks2: mkdir for scratch more robust

Release Notes 1.5.6

Ephys extraction bugfixes:

  • RawEphysQC: No object "ephysTimeRmsAP" found
  • EphysMtsComp,RawEphysQC, EphysPulses : ValueError: mmap length is greater than file size
  • Ks2: ks2 job cleans-up temp dir

Release Notes 1.5.5

  • ONE offline mode and cache dataset table to speed up reloading of large datasets (Olivier)
  • ALF io naming conventions on loading objects (Miles)
  • KS2 Matlab ephys pipeline tasks (Olivier)
  • Support for running QC on biased and training sessions (Nico)
  • metrics_df and passed_df properties in BpodQC obj for qcplots (Nico)
  • Added missing unittest to stim_move_before_goCue metric (Nico)

Release Notes 1.5.4 - 29/07/2020 hotfix

  • ibllib.pipes.training_preprocessing.TrainingAudio

Release Notes 1.5.3 - 28/07/2020

  • ibllib.pipes.training_preprocessing.TrainingAudio: returns files for registration and proper status. (Olivier)
  • ibllib.atlas: compute nearest region from probe trajectory (Mayo)

Release Notes 1.5.2 - 23/07/2020

  • Local server jobs:
    • fix wheel moves size mismatch extractor error
    • only look for raw_session.flag for ephys extraction to avoid race conditions

Release Notes 1.5.1 - 25/05/2020

  • Ephys extraction:
    • spike amplitudes in Volts
    • added waveforms samples dataset to use Phy from Flatiron datasets
  • ONE performance:
    • Metaclass implementation of UniqueSingletons for AlyxClient
    • Multi-threaded downloads
    • Added JSON fields methods to AlyxClient
  • QCs: Bpod and ONE QC features, basic plotting, examples

Release Notes 1.4

Release Notes 1.4.14 - 2020-03-09

  • Added permutation test, comparing a metric on two sets of datasets, by shuffling labels and seeing how plausible the observed actual difference is Sped up calculation of firing_rate
  • ephys extraction: updated extracted metrics, including a new contamination estimate and drift metrics.
  • ibllib.io.spikeglx

Release Notes 1.4.13 - 2020-03-07

  • Hotfix: rig transfer - create probes. One variable used before assignation.

Release Notes 1.4.12 - 2020-03-06

  • ONE.load overwrites local file if filesizes different or hash mismatch
  • ephys extraction:
    • registration sets the session.procedure field to acute recording
    • bugfix synchronization on re-extraction: always recompute spike.times from spike.samples
    • ephys registration sets the session.procedure field to acute recording
  • training extraction:
    • added biasedVisOffChoiceWorld as training extractor
  • wheel data
    • dropping support for wheel velocity, not extracted anymore

Release Notes 1.4.11 - 2020-02-14

  • bugfix: Include sessions data files for ephys mock
  • bugfix: Single probe 3B gets synchronized

Release Notes 1.4.10 - 2020-02-10

  • bugfix: Include sessions data files in pip package

Release Notes 1.4.9 - 2020-02-09

  • Big brainbox merge and release
  • bugfix: clusters.metrics spiking rates accurate
  • probability left for ephys choice world contain generative probabilities, not outcomes

Release Notes 1.4.8 - 2020-01-27

  • ONE Light Windows fixes
  • Installation documentation separates conda and virtualenv options
  • Conda yaml environement file for ibllib

Release Notes 1.4.7

  • ONE Light for behaviour paper data release
  • ONE() standard syntax matching the one light examples

Release Notes 1.4.6

  • Alyx registration: add md5, version and filesize to the pipeline registration
  • Data Patcher: allows to register data from anywhere through FTP/SSH/GLobus
  • ONE Light for behaviour paper data release

Release Notes 1.4.5 Hotfix

  • Ephys extraction: left probability bug when sequence was 0 - fixed

Release Notes 1.4.4

  • Ephys extraction:
    • left probability extracted properly
    • add robustness to audio fronts extraction in FPGA
  • Passive stimulus: raw data registered in pipeline
  • Training extraction: microphone extraction for habituation sessions
  • ALF: specific to_dataframe method for Bunch

Release Notes 1.4.3 Hotfix

  • Ephys extraction: handle fringe case where recording is interrupted in the middle
  • Wheel extraction: if rotary encoder version is outdated and stores data in the wrong unit, auto-detect and output in seconds even for new versions

Release Notes 1.4.2

  • FPGA/bpod events synchronization performed even when their counts do not match
  • Updated requirement versions for mtscomp and phylib

Release Notes 1.4.1

  • wheel extraction outputs a timestamps attribute, not times
  • make the wheel extraction more robust

Release Notes 1.4.0

  • Ephys extraction:
    • un-synchronized spike sortings not uploaded on flat-iron
    • reaction times extracted
    • valve-open times bugfix
  • Wheel extraction:
    • training wheel position and timing are now correct
    • ephys & training: units: radians mathematical convention
  • ONE:
    • Alyx client handles pagination seamlessly

Release Notes 1.3

Release Notes 1.3.10 Hotfix

  • cross-platform get of session folder for rig computer copy to server

Release Notes 1.3.9

  • spikeglx.verify_hash() method to check file integrity after transfers/manipulations
  • create wirings settings files on ephys computer transfer to server

Release Notes 1.3.8

  • Ephys Extraction:
    • duplicate probe.trajectories bugfix
    • extraction works with unoperational fram2ttl at beginning of ephys session
    • clusters.metrics.csv has consistent size with npy cluster objects
    • ephys transfer: create ephys extraction flags after the transfer is complete

Release Notes 1.3.7 hotfixes- 21-NOV-2019

  • Rename spike.times on failed sync to reflect the clock as per ALF convention
  • sync 3A fails if first cam event whithin 200ms of start
  • compress ephys goes through a tempfile to not interfere with transfers/globbing

Release Notes 1.3.6 - 20-NOV-2019

  • Ephys Extraction (phylib)
    • convert ks2 amplitudes to volts for spikes.amps, clusters.amps. templates.waveforms, clusters.waveforms to get uV
    • generates Cluster UUIDs file
    • individual spike depths computed from PC features
  • Ephys Synchronization
    • use frame2TTL split for 3A by default, if not found look for right_camera
    • output individual probe sync in ALF timestamps format

Release Notes 1.3.5 - 18-NOV-2019

  • registration ignores ks2alf probes subfolders
  • fix typo in raw qc dataset types

Release Notes 1.3.4 - 18-NOV-2019

  • Alyx registration adds the relative path to session root as dataset.subcollection
  • Ephys extraction:
    • split probe folders output alf/probe00 and alf/probe01 instead of merge
    • outputs templates.waveforms and clusters.waveforms in sparse arrays
    • outputs probes.description and probes.trajectory
    • renamed the raw ephys QC output
    • outputs clusters.metrics
  • Bugfixes:
    • 3B raw ephys QC output ap.file not found on nidq object
    • 3A sync probe threshold set to 2.1 samples

Release Notes 1.3.1 - 12-NOV-2019

  • transfer scripts from ephys/video/rig computers to local servers
  • bugfix spigeglx.glob_ephys_files when metadata file without ap.bin file

Release Notes 1.3.0 - 11-NOV-2019

  • Transfer rig data takes into account session type (ephys/training) to create flags
  • Ephys video compression in pipeline
  • Ephys audio compression in pipeline

Release Notes 1.2

Release Notes 1.2.9

  • Ephys extraction: provide full 3B default wirings if files do not exist.

Release Notes 1.2.8

  • Ephys extraction: merge sync ephys in the pipeline overwrites ks2_alf directory if it already exists

Release Notes 1.2.7

  • Added biasedScanningChoiceWorld task to biased extractor for Zador lab

Release Notes 1.2.6

feature/ephys_compression

  • spikeglx.Reader supports mtscomp ephys binaries
  • server pipeline for compression of ephys files

feature/peths

  • brainbox.singlecell.peths with tests
  • brainbox.processing.bincount2D supports aggregation on fixed scale
  • simple examples script and notebook

Release Notes 1.2.5

  • examples/brainbox/plot_peths.py: by Matt. W.
  • examples/brainbox/rasters by Michaël S.
  • Allen Atlas framework and probe registration base functions
  • server pipeline for audio extraction of training sessions