diff --git a/aeon/dj_pipeline/README.md b/aeon/dj_pipeline/README.md
index 545f0f35..7d898168 100644
--- a/aeon/dj_pipeline/README.md
+++ b/aeon/dj_pipeline/README.md
@@ -1,93 +1,91 @@
# DataJoint Pipeline for Project Aeon
-This pipeline models the data organization and data flow custom-built for Project Aeon. You can find Aeon acquisition system here: [aeon_aquisition](https://github.com/SainsburyWellcomeCentre/aeon_acquisition)
+This DataJoint pipeline models the data organization and data flow tailored to the project's Aeon requirements. You can access the Aeon acquisition system here: [aeon_aquisition](https://github.com/SainsburyWellcomeCentre/aeon_acquisition)
-## Pipeline architecture
+## Pipeline Architecture
-The diagram below shows the high level overview of the diagram (only the subset of the tables that are most relevant).
+The following diagrams provide a high-level overview of the pipeline's components and processes:
-
+The diagram below illustrates the structure of the **acquisition-related tasks within the pipeline**, focusing on the most relevant subset of tables.
+
-The diagram below shows the analysis portion of the pipeline (work in progress).
+The diagram below represents the **data stream flow within the pipeline**, highlighting the subset of tables critical to understanding the process.
-
+
+The diagram below illustrates the **Pyrat synchronization process within the pipeline**, highlighting the key tables involved in syncing data across different components.
-From the diagram above, we can see that the pipeline is organized in layers of
-tables, going top down, from `lookup`-tier (in gray) and `manual`-tier (in green) tables
-to `imported`-tier (in purple) and `computed`-tier (in red) tables.
+
+
+The diagram below shows the **analysis portion of the pipeline**.
+
+
+
+
+The pipeline is structured into hierarchical layers of tables, which are depicted in the diagrams above. These layers include:
+
++ `lookup`-tier tables (gray): Define static reference information
++ `manual`-tier tables (green): Contain user-inputted data
++ `imported`-tier tables (purple): Store data ingested from external sources
++ `computed`-tier tables (red): Represent results of automated computations
+
+Data flows through the pipeline in a top-down manner, driven by a combination of ingestion and computation routines. This layered organization facilitates efficient data processing and modular analysis.
-Such is also the way the data flows through the pipeline, by a combination of ingestion and
-computation routines.
## Core tables
#### Experiment and data acquisition
-1. `Experiment` - the `aquisition.Experiment` table stores meta information about the experiments
++ `Experiment` - The `aquisition.Experiment` table stores meta information about the experiments
done in Project Aeon, with secondary information such as the lab/room the experiment is carried out,
which animals participating, the directory storing the raw data, etc.
-2. `Epoch` - A recording period reflecting on/off of the hardware acquisition system.
++ `Epoch` - A recording period reflecting on/off of the hardware acquisition system.
The `aquisition.Epoch` table records all acquisition epochs and their associated configuration for
any particular experiment (in the above `aquisition.Experiment` table).
-3.`Chunk` - the raw data are acquired by Bonsai and stored as
++ `Chunk` - The raw data are acquired by Bonsai and stored as
a collection of files every one hour - we call this one-hour a time chunk.
The `aquisition.Chunk` table records all time chunks and their associated raw data files for
any particular experiment (in the above `aquisition.Experiment` table). A chunk must belong to one epoch.
-#### Devices
-
-5. `ExperimentCamera` - the cameras and associated specifications used for this experiment -
-e.g. camera serial number, frame rate, location, time of installation and removal, etc.
-
-6. `ExperimentFoodPatch` - the food-patches and associated specifications used for this experiment -
-e.g. patch serial number, sampling rate of the wheel, location, time of installation and removal, etc.
-
-7. `ExperimentWeightScale` - the scales for measuring animal weights, usually placed at the nest, one per nest
-
-#### Data streams
-
-8. `FoodPatchEvent` - all events (e.g. pellet triggered, pellet delivered, etc.)
-from a particular `ExperimentFoodPatch`
+#### Position data
-9. `FoodPatchWheel` - wheel data (angle, intensity) from a particular `ExperimentFoodPatch`
++ `qc.CameraQC` - A quality control procedure applied to each `ExperimentCamera` (e.g. missing frame, etc.)
-10. `WheelState` - wheel states (threshold, d1, delta) associated with a given `ExperimentFoodPatch`
++ `tracking.SLEAPTracking` - Position tracking of object(s) from a particular `VideoSource` per chunk. Key tables include:
+ - `PoseIdentity` - Identifies the Subject (i.e. Identity) and stores the name of the body part used as "anchor_part".
+ - `Part` - Contains the inferred x,y positions over time for all body parts, as derived from the SLEAP model.
-11. `WeightMeasurement` - scale measurements associated with a given `ExperimentScale`
+#### Standard analyses
++ `Visit` - A `Visit` is defined as a period of time during which a particular animal remains at a specific place.
-#### Position data
++ `Block` - A `Block` refers to a specific period of time, typically lasting around 3 hours, during which the reward rate for each patch is predefined to facilitate certain animal behaviors.
-12. `qc.CameraQC` - quality control procedure applied to each `ExperimentCamera` (e.g. missing frame, etc.)
++ `BlockAnalysis` - A higher-level aggregation of events and metrics occurring within a defined block of time during an experiment.This analysis computes patch-related and subject-related metrics separately, without combining or cross-correlating data between them. It provides an overview of behavior and environmental interactions at a broader level, integrating data from multiple subjects and patches.
-13. `tracking.CameraTracking` - position tracking for object(s), from each `ExperimentCamera`
++ `BlockSubjectAnalysis` - A detailed analysis focusing on individual subjects within a block, explicitly combining a subject's interactions with specific patches. This involves examining how a subject interacts with a particular patch, including metrics such as total interaction time and overall time spent at the patch. Key components include:
+ - `Patch`: Tracks the interactions of each subject with specific patches (areas of interests)
+ - `Preference`: Measures a subject's preference for specific patches using various analyses, including cumulative preferences based on time spent and distance traveled in relation to each patch.
-#### Standard analyses
+#### Data stream
-14. `Visit` - a `Visit` is defined as a ***period of time***
-that a particular ***animal*** spends time at a particular ***place***
++ `SpinnakerVideoSource` - Tracks the placement and operation of a Spinnaker video source at a specific location during a defined period in an experiment. This class includes metadata such as the installation time of the device, enabling the tracking of video data associated with the device.
-15. `VisitSubjectPosition` - position data (x, y, z, area) of the subject for any particular visit.
-Position data per visit are stored in smaller time slices (10-minute long) allowing for
-more efficient searches, queries and fetches from the database.
++ `RfidReader` - Represents the placement and operation of an RFID reader at a specific location for a defined period in an experiment. It records the installation time of the device and facilitates the collection of RFID event data, such as RFID tag detections. The data stream includes sample counts, timestamps, and RFID tag events.
-16. `VisitSummary` - a table for computation and storing some summary statistics on a
-per-session level - i.e. total pellet delivered, total distance the animal travelled, total
-distance the wheel travelled (or per food-patch), etc.
++ `WeightScale` - Monitors the placement and operation of a weight scale within an experiment. It records the installation time of the weight scale and other related metadata, enabling the collection of weight measurements. The data streams include sample counts, timestamps, and weight readings.
-17. `VisitTimeDistribution` - a table for computation and storing where the animal is at,
-for each timepoint, e.g. in the nest, in corridor, in arena, in each of the food patches.
-This can be used to produce the ethogram plot.
++ `UndergroundFeeder` - Tracks the operation of an underground feeder device in a specific experiment. It stores installation time and other metadata, such as configuration and calibration settings. The data stream includes events like beam breaks, pellet deliveries, and depletion states.
## Operating the pipeline - how the auto ingestion/processing work?
Some meta information about the experiment is entered - e.g. experiment name, participating
animals, cameras, food patches setup, etc.
+
+ These information are either entered by hand, or parsed and inserted from configuration
yaml files.
+ For experiments these info can be inserted by running
@@ -105,7 +103,6 @@ These routines are prepared in this [auto-processing script](populate/process.py
Essentially, turning on the auto-processing routine amounts to running the
following 2 commands (in different processing threads)
-
aeon_ingest high
aeon_ingest mid
diff --git a/aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg b/aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg
index 94087629..2dc0f3b7 100644
--- a/aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg
+++ b/aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg
@@ -1,222 +1,254 @@
-
\ No newline at end of file
diff --git a/aeon/dj_pipeline/docs/datajoint_overview_acquisition_related_diagram.svg b/aeon/dj_pipeline/docs/datajoint_overview_acquisition_related_diagram.svg
new file mode 100644
index 00000000..f0420b7c
--- /dev/null
+++ b/aeon/dj_pipeline/docs/datajoint_overview_acquisition_related_diagram.svg
@@ -0,0 +1,243 @@
+
\ No newline at end of file
diff --git a/aeon/dj_pipeline/docs/datajoint_overview_data_stream_diagram.svg b/aeon/dj_pipeline/docs/datajoint_overview_data_stream_diagram.svg
new file mode 100644
index 00000000..c41f2603
--- /dev/null
+++ b/aeon/dj_pipeline/docs/datajoint_overview_data_stream_diagram.svg
@@ -0,0 +1,407 @@
+
\ No newline at end of file
diff --git a/aeon/dj_pipeline/docs/datajoint_overview_diagram.svg b/aeon/dj_pipeline/docs/datajoint_overview_diagram.svg
deleted file mode 100644
index 02590b4a..00000000
--- a/aeon/dj_pipeline/docs/datajoint_overview_diagram.svg
+++ /dev/null
@@ -1,381 +0,0 @@
-
diff --git a/aeon/dj_pipeline/docs/datajoint_overview_pyrat_related_diagram.svg b/aeon/dj_pipeline/docs/datajoint_overview_pyrat_related_diagram.svg
new file mode 100644
index 00000000..59e4fe43
--- /dev/null
+++ b/aeon/dj_pipeline/docs/datajoint_overview_pyrat_related_diagram.svg
@@ -0,0 +1,84 @@
+
\ No newline at end of file
diff --git a/aeon/dj_pipeline/docs/notebooks/analysis_diagram.svg b/aeon/dj_pipeline/docs/notebooks/analysis_diagram.svg
deleted file mode 100644
index 94087629..00000000
--- a/aeon/dj_pipeline/docs/notebooks/analysis_diagram.svg
+++ /dev/null
@@ -1,222 +0,0 @@
-
diff --git a/aeon/dj_pipeline/docs/notebooks/diagram.ipynb b/aeon/dj_pipeline/docs/notebooks/diagram.ipynb
index fc6734d2..c16763fb 100644
--- a/aeon/dj_pipeline/docs/notebooks/diagram.ipynb
+++ b/aeon/dj_pipeline/docs/notebooks/diagram.ipynb
@@ -2,25 +2,17 @@
"cells": [
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"id": "b18d8c4a",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "/nfs/nhome/live/thinh/code/ProjectAeon/aeon\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"cd ../../../.."
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": null,
"id": "b9db4795",
"metadata": {},
"outputs": [],
@@ -30,42 +22,38 @@
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": null,
"id": "571c2760",
"metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "[2022-07-28 19:53:12,486][INFO]: Connecting thinh@aeon-db2:3306\n",
- "[2022-07-28 19:53:12,497][INFO]: Connected thinh@aeon-db2:3306\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
- "_db_prefix = \"aeon_\"\n",
+ "_db_prefix = 'aeon_'\n",
"\n",
- "lab = dj.create_virtual_module(\"lab\", _db_prefix + \"lab\")\n",
+ "streams = dj.create_virtual_module(\"streams\", _db_prefix + \"streams\")\n",
+ "tracking = dj.create_virtual_module(\"tracking\", _db_prefix + \"tracking\")\n",
+ "analysis = dj.create_virtual_module(\"analysis\", _db_prefix + \"analysis\")\n",
+ "block_analysis = dj.create_virtual_module(\"block_analysis\", _db_prefix + \"block_analysis\")\n",
"subject = dj.create_virtual_module(\"subject\", _db_prefix + \"subject\")\n",
+ "lab = dj.create_virtual_module(\"lab\", _db_prefix + \"lab\")\n",
"acquisition = dj.create_virtual_module(\"acquisition\", _db_prefix + \"acquisition\")\n",
- "qc = dj.create_virtual_module(\"qc\", _db_prefix + \"qc\")\n",
- "tracking = dj.create_virtual_module(\"tracking\", _db_prefix + \"tracking\")\n",
- "analysis = dj.create_virtual_module(\"analysis\", _db_prefix + \"analysis\")"
+ "qc = dj.create_virtual_module(\"qc\", _db_prefix + \"qc\")\n"
]
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": null,
"id": "bcf4bd2e",
"metadata": {},
"outputs": [],
"source": [
- "lab.schema.spawn_missing_classes()\n",
+ "streams.schema.spawn_missing_classes()\n",
+ "tracking.schema.spawn_missing_classes()\n",
+ "analysis.schema.spawn_missing_classes()\n",
+ "block_analysis.schema.spawn_missing_classes()\n",
"subject.schema.spawn_missing_classes()\n",
+ "lab.schema.spawn_missing_classes()\n",
"acquisition.schema.spawn_missing_classes()\n",
- "tracking.schema.spawn_missing_classes()\n",
- "analysis.schema.spawn_missing_classes()"
+ "qc.schema.spawn_missing_classes()"
]
},
{
@@ -78,433 +66,104 @@
},
{
"cell_type": "code",
- "execution_count": 12,
- "id": "5c447e42",
+ "execution_count": null,
+ "id": "6e9442ac",
"metadata": {},
- "outputs": [
- {
- "data": {
- "image/svg+xml": [
- ""
- ],
- "text/plain": [
- ""
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
+ "outputs": [],
"source": [
- "diagram = (\n",
- " dj.Diagram(acquisition.Epoch)\n",
- " + lab.Arena\n",
- " + lab.FoodPatch\n",
- " + lab.Camera\n",
- " + lab.WeightScale\n",
- " + acquisition.Experiment\n",
- " + acquisition.ExperimentCamera\n",
- " + acquisition.FoodPatchWheel\n",
- " + acquisition.FoodPatchEvent\n",
- " + acquisition.WheelState\n",
- " + acquisition.EventType\n",
- " + acquisition.WeightMeasurement\n",
- " + qc.CameraQC\n",
- " + tracking.CameraTracking.Object\n",
- ")\n",
+ "# acquisition-related tasks\n",
+ "acquisition_diagram = (\n",
+ " dj.Diagram(lab.Location) +\n",
+ " lab.Arena +\n",
+ " acquisition.ExperimentType +\n",
+ " subject.Subject +\n",
+ " acquisition.Experiment.Subject +\n",
+ " acquisition.Experiment +\n",
+ " acquisition.Epoch +\n",
+ " acquisition.EpochConfig +\n",
+ " acquisition.Chunk +\n",
+ " acquisition.Environment +\n",
+ " acquisition.Environment.SubjectWeight +\n",
+ " acquisition.Environment.EnvironmentState +\n",
+ " acquisition.Environment.SubjectState +\n",
+ " acquisition.Environment.MessageLog +\n",
+ " acquisition.Environment.SubjectVisits +\n",
+ " acquisition.Environment.BlockState +\n",
+ " acquisition.Environment.LightEvents\n",
+ " ) - dj.Diagram(acquisition.Experiment)\n",
"\n",
- "diagram"
+ "acquisition_diagram"
]
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": null,
+ "id": "0600b4b7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "acquisition_diagram.save(\"aeon/dj_pipeline/docs/datajoint_overview_acquisition_related_diagram.svg\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "aea37324",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# data streams\n",
+ "data_stream_diagram =(\n",
+ " dj.Diagram(streams) \n",
+ " - streams.SpinnakerVideoSource.RemovalTime\n",
+ " - streams.SpinnakerVideoSource.Attribute\n",
+ " - streams.UndergroundFeeder.Attribute\n",
+ " - streams.UndergroundFeeder.RemovalTime\n",
+ " - streams.RfidReader.Attribute\n",
+ " - streams.RfidReader.RemovalTime\n",
+ " - streams.WeightScale.Attribute\n",
+ " - streams.WeightScale.RemovalTime\n",
+ " + acquisition.Experiment\n",
+ " + acquisition.Epoch\n",
+ " ) \n",
+ "data_stream_diagram "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
"id": "fc5e04ff",
"metadata": {},
"outputs": [],
"source": [
- "diagram.save(\"aeon/dj_pipeline/docs/datajoint_overview_diagram.svg\")"
+ "data_stream_diagram.save(\"aeon/dj_pipeline/docs/datajoint_overview_data_stream_diagram.svg\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8d3f7a66",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Pyrat-related tables\n",
+ "\n",
+ "pyrat_diagram = (\n",
+ " dj.Diagram(subject.SubjectDetail) + \n",
+ " subject.PyratCommentWeightProcedure\n",
+ ") - 1\n",
+ "\n",
+ "pyrat_diagram"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8ff44b6f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "pyrat_diagram.save(\"aeon/dj_pipeline/docs/datajoint_overview_pyrat_related_diagram.svg\")"
]
},
{
@@ -517,361 +176,46 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": null,
"id": "ead30859",
"metadata": {},
- "outputs": [
- {
- "data": {
- "image/svg+xml": [
- ""
- ],
- "text/plain": [
- ""
- ]
- },
- "execution_count": 14,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
+ "outputs": [],
"source": [
+ "# analysis tables\n",
"analysis_diagram = (\n",
- " dj.Diagram(analysis.Visit)\n",
- " + analysis.Place\n",
- " + analysis.VisitEnd\n",
- " + analysis.OverlapVisit.Visit\n",
- " + acquisition.Experiment\n",
- " + tracking.CameraTracking\n",
- " + analysis.VisitSubjectPosition.TimeSlice\n",
- " + analysis.VisitTimeDistribution\n",
- " + analysis.VisitSummary\n",
- ")\n",
+ " dj.Diagram(analysis.Visit) +\n",
+ " analysis.OverlapVisit.Visit +\n",
+ " block_analysis.Block +\n",
+ " block_analysis.BlockAnalysis +\n",
+ " block_analysis.BlockSubjectAnalysis.Preference +\n",
+ " block_analysis.BlockSubjectAnalysis.Patch +\n",
+ " block_analysis.BlockForaging +\n",
+ " block_analysis.BlockPatchPlots +\n",
+ " block_analysis.BlockSubjectPositionPlots \n",
+ ") - 1\n",
"\n",
"analysis_diagram"
]
},
{
"cell_type": "code",
- "execution_count": 15,
+ "execution_count": null,
"id": "56a3b461",
"metadata": {},
"outputs": [],
"source": [
- "diagram.save(\"aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg\")"
+ "analysis_diagram.save(\"aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg\")"
]
},
{
"cell_type": "code",
- "execution_count": 19,
+ "execution_count": null,
"id": "fd2745c0",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "datajoint_analysis_diagram.svg datajoint_overview_diagram.svg \u001b[0m\u001b[01;34mnotebooks\u001b[0m/\r\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"ls aeon/dj_pipeline/docs/"
]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "fce5849b",
- "metadata": {},
- "outputs": [],
- "source": []
}
],
"metadata": {
diff --git a/aeon/dj_pipeline/docs/notebooks/diagram.svg b/aeon/dj_pipeline/docs/notebooks/diagram.svg
deleted file mode 100644
index d7872725..00000000
--- a/aeon/dj_pipeline/docs/notebooks/diagram.svg
+++ /dev/null
@@ -1,381 +0,0 @@
-
-The diagram below shows the analysis portion of the pipeline (work in progress).
+The diagram below represents the **data stream flow within the pipeline**, highlighting the subset of tables critical to understanding the process.
-
+
+The diagram below illustrates the **Pyrat synchronization process within the pipeline**, highlighting the key tables involved in syncing data across different components.
-From the diagram above, we can see that the pipeline is organized in layers of
-tables, going top down, from `lookup`-tier (in gray) and `manual`-tier (in green) tables
-to `imported`-tier (in purple) and `computed`-tier (in red) tables.
+
+
+The diagram below shows the **analysis portion of the pipeline**.
+
+
+
+
+The pipeline is structured into hierarchical layers of tables, which are depicted in the diagrams above. These layers include:
+
++ `lookup`-tier tables (gray): Define static reference information
++ `manual`-tier tables (green): Contain user-inputted data
++ `imported`-tier tables (purple): Store data ingested from external sources
++ `computed`-tier tables (red): Represent results of automated computations
+
+Data flows through the pipeline in a top-down manner, driven by a combination of ingestion and computation routines. This layered organization facilitates efficient data processing and modular analysis.
-Such is also the way the data flows through the pipeline, by a combination of ingestion and
-computation routines.
## Core tables
#### Experiment and data acquisition
-1. `Experiment` - the `aquisition.Experiment` table stores meta information about the experiments
++ `Experiment` - The `aquisition.Experiment` table stores meta information about the experiments
done in Project Aeon, with secondary information such as the lab/room the experiment is carried out,
which animals participating, the directory storing the raw data, etc.
-2. `Epoch` - A recording period reflecting on/off of the hardware acquisition system.
++ `Epoch` - A recording period reflecting on/off of the hardware acquisition system.
The `aquisition.Epoch` table records all acquisition epochs and their associated configuration for
any particular experiment (in the above `aquisition.Experiment` table).
-3.`Chunk` - the raw data are acquired by Bonsai and stored as
++ `Chunk` - The raw data are acquired by Bonsai and stored as
a collection of files every one hour - we call this one-hour a time chunk.
The `aquisition.Chunk` table records all time chunks and their associated raw data files for
any particular experiment (in the above `aquisition.Experiment` table). A chunk must belong to one epoch.
-#### Devices
-
-5. `ExperimentCamera` - the cameras and associated specifications used for this experiment -
-e.g. camera serial number, frame rate, location, time of installation and removal, etc.
-
-6. `ExperimentFoodPatch` - the food-patches and associated specifications used for this experiment -
-e.g. patch serial number, sampling rate of the wheel, location, time of installation and removal, etc.
-
-7. `ExperimentWeightScale` - the scales for measuring animal weights, usually placed at the nest, one per nest
-
-#### Data streams
-
-8. `FoodPatchEvent` - all events (e.g. pellet triggered, pellet delivered, etc.)
-from a particular `ExperimentFoodPatch`
+#### Position data
-9. `FoodPatchWheel` - wheel data (angle, intensity) from a particular `ExperimentFoodPatch`
++ `qc.CameraQC` - A quality control procedure applied to each `ExperimentCamera` (e.g. missing frame, etc.)
-10. `WheelState` - wheel states (threshold, d1, delta) associated with a given `ExperimentFoodPatch`
++ `tracking.SLEAPTracking` - Position tracking of object(s) from a particular `VideoSource` per chunk. Key tables include:
+ - `PoseIdentity` - Identifies the Subject (i.e. Identity) and stores the name of the body part used as "anchor_part".
+ - `Part` - Contains the inferred x,y positions over time for all body parts, as derived from the SLEAP model.
-11. `WeightMeasurement` - scale measurements associated with a given `ExperimentScale`
+#### Standard analyses
++ `Visit` - A `Visit` is defined as a period of time during which a particular animal remains at a specific place.
-#### Position data
++ `Block` - A `Block` refers to a specific period of time, typically lasting around 3 hours, during which the reward rate for each patch is predefined to facilitate certain animal behaviors.
-12. `qc.CameraQC` - quality control procedure applied to each `ExperimentCamera` (e.g. missing frame, etc.)
++ `BlockAnalysis` - A higher-level aggregation of events and metrics occurring within a defined block of time during an experiment.This analysis computes patch-related and subject-related metrics separately, without combining or cross-correlating data between them. It provides an overview of behavior and environmental interactions at a broader level, integrating data from multiple subjects and patches.
-13. `tracking.CameraTracking` - position tracking for object(s), from each `ExperimentCamera`
++ `BlockSubjectAnalysis` - A detailed analysis focusing on individual subjects within a block, explicitly combining a subject's interactions with specific patches. This involves examining how a subject interacts with a particular patch, including metrics such as total interaction time and overall time spent at the patch. Key components include:
+ - `Patch`: Tracks the interactions of each subject with specific patches (areas of interests)
+ - `Preference`: Measures a subject's preference for specific patches using various analyses, including cumulative preferences based on time spent and distance traveled in relation to each patch.
-#### Standard analyses
+#### Data stream
-14. `Visit` - a `Visit` is defined as a ***period of time***
-that a particular ***animal*** spends time at a particular ***place***
++ `SpinnakerVideoSource` - Tracks the placement and operation of a Spinnaker video source at a specific location during a defined period in an experiment. This class includes metadata such as the installation time of the device, enabling the tracking of video data associated with the device.
-15. `VisitSubjectPosition` - position data (x, y, z, area) of the subject for any particular visit.
-Position data per visit are stored in smaller time slices (10-minute long) allowing for
-more efficient searches, queries and fetches from the database.
++ `RfidReader` - Represents the placement and operation of an RFID reader at a specific location for a defined period in an experiment. It records the installation time of the device and facilitates the collection of RFID event data, such as RFID tag detections. The data stream includes sample counts, timestamps, and RFID tag events.
-16. `VisitSummary` - a table for computation and storing some summary statistics on a
-per-session level - i.e. total pellet delivered, total distance the animal travelled, total
-distance the wheel travelled (or per food-patch), etc.
++ `WeightScale` - Monitors the placement and operation of a weight scale within an experiment. It records the installation time of the weight scale and other related metadata, enabling the collection of weight measurements. The data streams include sample counts, timestamps, and weight readings.
-17. `VisitTimeDistribution` - a table for computation and storing where the animal is at,
-for each timepoint, e.g. in the nest, in corridor, in arena, in each of the food patches.
-This can be used to produce the ethogram plot.
++ `UndergroundFeeder` - Tracks the operation of an underground feeder device in a specific experiment. It stores installation time and other metadata, such as configuration and calibration settings. The data stream includes events like beam breaks, pellet deliveries, and depletion states.
## Operating the pipeline - how the auto ingestion/processing work?
Some meta information about the experiment is entered - e.g. experiment name, participating
animals, cameras, food patches setup, etc.
+
+ These information are either entered by hand, or parsed and inserted from configuration
yaml files.
+ For experiments these info can be inserted by running
@@ -105,7 +103,6 @@ These routines are prepared in this [auto-processing script](populate/process.py
Essentially, turning on the auto-processing routine amounts to running the
following 2 commands (in different processing threads)
-
aeon_ingest high
aeon_ingest mid
diff --git a/aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg b/aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg
index 94087629..2dc0f3b7 100644
--- a/aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg
+++ b/aeon/dj_pipeline/docs/datajoint_analysis_diagram.svg
@@ -1,222 +1,254 @@
-
\ No newline at end of file
diff --git a/aeon/dj_pipeline/docs/datajoint_overview_acquisition_related_diagram.svg b/aeon/dj_pipeline/docs/datajoint_overview_acquisition_related_diagram.svg
new file mode 100644
index 00000000..f0420b7c
--- /dev/null
+++ b/aeon/dj_pipeline/docs/datajoint_overview_acquisition_related_diagram.svg
@@ -0,0 +1,243 @@
+
\ No newline at end of file
diff --git a/aeon/dj_pipeline/docs/datajoint_overview_data_stream_diagram.svg b/aeon/dj_pipeline/docs/datajoint_overview_data_stream_diagram.svg
new file mode 100644
index 00000000..c41f2603
--- /dev/null
+++ b/aeon/dj_pipeline/docs/datajoint_overview_data_stream_diagram.svg
@@ -0,0 +1,407 @@
+
\ No newline at end of file
diff --git a/aeon/dj_pipeline/docs/datajoint_overview_diagram.svg b/aeon/dj_pipeline/docs/datajoint_overview_diagram.svg
deleted file mode 100644
index 02590b4a..00000000
--- a/aeon/dj_pipeline/docs/datajoint_overview_diagram.svg
+++ /dev/null
@@ -1,381 +0,0 @@
-
diff --git a/aeon/dj_pipeline/docs/datajoint_overview_pyrat_related_diagram.svg b/aeon/dj_pipeline/docs/datajoint_overview_pyrat_related_diagram.svg
new file mode 100644
index 00000000..59e4fe43
--- /dev/null
+++ b/aeon/dj_pipeline/docs/datajoint_overview_pyrat_related_diagram.svg
@@ -0,0 +1,84 @@
+
\ No newline at end of file
diff --git a/aeon/dj_pipeline/docs/notebooks/analysis_diagram.svg b/aeon/dj_pipeline/docs/notebooks/analysis_diagram.svg
deleted file mode 100644
index 94087629..00000000
--- a/aeon/dj_pipeline/docs/notebooks/analysis_diagram.svg
+++ /dev/null
@@ -1,222 +0,0 @@
-
diff --git a/aeon/dj_pipeline/docs/notebooks/diagram.ipynb b/aeon/dj_pipeline/docs/notebooks/diagram.ipynb
index fc6734d2..c16763fb 100644
--- a/aeon/dj_pipeline/docs/notebooks/diagram.ipynb
+++ b/aeon/dj_pipeline/docs/notebooks/diagram.ipynb
@@ -2,25 +2,17 @@
"cells": [
{
"cell_type": "code",
- "execution_count": 1,
+ "execution_count": null,
"id": "b18d8c4a",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "/nfs/nhome/live/thinh/code/ProjectAeon/aeon\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"cd ../../../.."
]
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": null,
"id": "b9db4795",
"metadata": {},
"outputs": [],
@@ -30,42 +22,38 @@
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": null,
"id": "571c2760",
"metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "[2022-07-28 19:53:12,486][INFO]: Connecting thinh@aeon-db2:3306\n",
- "[2022-07-28 19:53:12,497][INFO]: Connected thinh@aeon-db2:3306\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
- "_db_prefix = \"aeon_\"\n",
+ "_db_prefix = 'aeon_'\n",
"\n",
- "lab = dj.create_virtual_module(\"lab\", _db_prefix + \"lab\")\n",
+ "streams = dj.create_virtual_module(\"streams\", _db_prefix + \"streams\")\n",
+ "tracking = dj.create_virtual_module(\"tracking\", _db_prefix + \"tracking\")\n",
+ "analysis = dj.create_virtual_module(\"analysis\", _db_prefix + \"analysis\")\n",
+ "block_analysis = dj.create_virtual_module(\"block_analysis\", _db_prefix + \"block_analysis\")\n",
"subject = dj.create_virtual_module(\"subject\", _db_prefix + \"subject\")\n",
+ "lab = dj.create_virtual_module(\"lab\", _db_prefix + \"lab\")\n",
"acquisition = dj.create_virtual_module(\"acquisition\", _db_prefix + \"acquisition\")\n",
- "qc = dj.create_virtual_module(\"qc\", _db_prefix + \"qc\")\n",
- "tracking = dj.create_virtual_module(\"tracking\", _db_prefix + \"tracking\")\n",
- "analysis = dj.create_virtual_module(\"analysis\", _db_prefix + \"analysis\")"
+ "qc = dj.create_virtual_module(\"qc\", _db_prefix + \"qc\")\n"
]
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": null,
"id": "bcf4bd2e",
"metadata": {},
"outputs": [],
"source": [
- "lab.schema.spawn_missing_classes()\n",
+ "streams.schema.spawn_missing_classes()\n",
+ "tracking.schema.spawn_missing_classes()\n",
+ "analysis.schema.spawn_missing_classes()\n",
+ "block_analysis.schema.spawn_missing_classes()\n",
"subject.schema.spawn_missing_classes()\n",
+ "lab.schema.spawn_missing_classes()\n",
"acquisition.schema.spawn_missing_classes()\n",
- "tracking.schema.spawn_missing_classes()\n",
- "analysis.schema.spawn_missing_classes()"
+ "qc.schema.spawn_missing_classes()"
]
},
{
@@ -78,433 +66,104 @@
},
{
"cell_type": "code",
- "execution_count": 12,
- "id": "5c447e42",
+ "execution_count": null,
+ "id": "6e9442ac",
"metadata": {},
- "outputs": [
- {
- "data": {
- "image/svg+xml": [
- ""
- ],
- "text/plain": [
- "