Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up ROI drawing for users #103

Draft
wants to merge 35 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
87368c3
added pooch as dependency
niksirbi Jul 6, 2023
6e21546
removed old sample project
niksirbi Jul 6, 2023
4b5c07a
added scripts for generating sample projects automatically
niksirbi Jul 6, 2023
50a6da5
fixed typo in file name
niksirbi Jul 6, 2023
6e67eb3
modified scripts for automatic sample project generation
niksirbi Jul 11, 2023
cf2886d
Added datasets module to handle the fetching of sample datasets
niksirbi Jul 13, 2023
c1ee54a
added info about sample datasets in CONTRIBUTING.md
niksirbi Jul 13, 2023
d5e81f8
added pytest fixtures for fetching the sample project
niksirbi Jul 13, 2023
6f5d25d
fixed paths to sample project in tests
niksirbi Jul 13, 2023
ae40627
modify assertions in tess for metadatda dfs
niksirbi Jul 14, 2023
fa79654
added expected hashes for sample datasets
niksirbi Jul 14, 2023
4d8656f
added tqdm dependency
niksirbi Jul 14, 2023
93d6222
renamed sample dataset to sample projects
niksirbi Jul 14, 2023
64b3026
delete placeholder unit test
niksirbi Jul 14, 2023
fdb7c1c
moved sample_project fixture to conftest.py
niksirbi Jul 14, 2023
c56d63d
added caching of test data during gh actions to speed up CI
niksirbi Aug 2, 2023
81861cc
deleted duplicate word in docstring
niksirbi Nov 7, 2023
95c551c
modified instructions for adding new projects on GIN
niksirbi Nov 7, 2023
bbf86b6
show rectangle and circle buttons in modebar
niksirbi Nov 7, 2023
266d573
convert rectangles and circles to path
niksirbi Nov 7, 2023
686f086
group all utils operating on shapes together
niksirbi Nov 7, 2023
20a08a4
Make rectangle the default drag mode
niksirbi Nov 7, 2023
67a1fd2
Use built-name shape key "name" instead of custom "roi_name"
niksirbi Nov 7, 2023
cc00ee3
ROI table now shows ROI area instead of path
niksirbi Nov 7, 2023
28aa173
Add text labels to ROI shapes on graph
niksirbi Nov 8, 2023
3ae5be5
Don't show "on frame" in the ROI table
niksirbi Nov 8, 2023
6d5d0cf
allocate more space to the frame graph
niksirbi Nov 8, 2023
9ac22d0
moved ROI table buttons from footer to header
niksirbi Nov 8, 2023
dbba2e6
implemented functionality for copying ROIs from another video
niksirbi Nov 8, 2023
b7f474c
added tooltip to copy ROIs button
niksirbi Nov 10, 2023
4c97bbf
remember to change the "drawn_on_frame" property when copying ROIs
niksirbi Nov 10, 2023
fb2c26e
make circle polygons smoother
niksirbi Nov 10, 2023
74e8bac
fix ROI status alert by correctly checking the equality of ROI lists
niksirbi Nov 10, 2023
d80f9f5
added check-manifest pre-commit hook (#99)
niksirbi Jul 7, 2023
34404d9
Data hosting (#100)
niksirbi Nov 10, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/test_and_deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,14 @@ jobs:
- os: windows-latest
python-version: "3.10"
steps:
# Cache the test data to avoid re-downloading
- name: Cache Test Data
uses: actions/cache@v3
with:
path: ${{ github.workspace }}/.WAZP/*
key: cached-test-data
enableCrossOsArchive: true

# A hack because chrome isn't in the PATH on Windows
- name: Fix Chrome application path on Windows
if: matrix.os == 'windows-latest'
Expand Down
6 changes: 6 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,9 @@ repos:
- types-PyYaML
- pandas-stubs
- types-Pillow
- repo: https://github.com/mgedmin/check-manifest
rev: "0.49"
hooks:
- id: check-manifest
args: [--no-build-isolation]
additional_dependencies: [setuptools-scm]
83 changes: 80 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,12 @@ For Windows, be sure to download the ``chromedriver_win32.zip`` file, extract th

It's a good idea to test locally before pushing. Pytest will run all tests and also report test coverage.

#### Test data
For some tests, you will need to use real experimental data.
We store some sample projects in an external data repository.
See [sample projects](#sample-projects) for more information.


### Continuous integration
All pushes and pull requests will be built by [GitHub actions](https://docs.github.com/en/actions). This will usually include linting, testing and deployment.

Expand All @@ -139,7 +145,7 @@ We use [semantic versioning](https://semver.org/), which includes `MAJOR`.`MINOR
* MINOR = new feature
* MAJOR = breaking change

We use [`setuptools_scm`](https://github.com/pypa/setuptools_scm) to automatically version WAZP. It has been pre-configured in the `pyproject.toml` file. [`setuptools_scm` will automatically infer the version using git](https://github.com/pypa/setuptools_scm#default-versioning-scheme). To manually set a new semantic version, create a tag and make sure the tag is pushed to GitHub. Make sure you commit any changes you wish to be included in this version. E.g. to bump the version to `1.0.0`:
We use [`setuptools_scm`](https://github.com/pypa/setuptools_scm) to automatically version WAZP. It has been pre-configured in the `pyproject.toml` file. `setuptools_scm` will automatically infer the version using git. To manually set a new semantic version, create a tag and make sure the tag is pushed to GitHub. Make sure you commit any changes you wish to be included in this version. E.g. to bump the version to `1.0.0`:

```sh
git add .
Expand Down Expand Up @@ -175,8 +181,6 @@ If you create a new documentation source file (e.g. `my_new_file.md` or `my_new_
my_new_file
```



### Building the documentation locally
We recommend that you build and view the documentation website locally, before you push it.
To do so, first install the requirements for building the documentation:
Expand All @@ -197,5 +201,78 @@ rm -rf docs/build
sphinx-build docs/source docs/build
```

## Sample projects

We maintain some sample WAZP projects to be used for testing, examples and tutorials on an [external data repository](https://gin.g-node.org/SainsburyWellcomeCentre/WAZP).
Our hosting platform of choice is called [GIN](https://gin.g-node.org/) and is maintained by the [German Neuroinformatics Node](https://www.g-node.org/).
GIN has a GitHub-like interface and git-like [CLI](https://gin.g-node.org/G-Node/Info/wiki/GIN+CLI+Setup#quickstart) functionalities.

### Project organisation

The projects are stored in folders named after the species - e.g. `jewel-wasp` (*Ampulex compressa*).
Each species folder may contain various WAZP sample projects as zipped archives. For example, the `jewel-wasp` folder contains the following projects:
- `short-clips_raw.zip` - a project containing short ~10 second clips extracted from raw .avi files.
- `short-clips_compressed.zip` - same as above, but compressed using the H.264 codec and saved as .mp4 files.
- `entire-video_raw.zip` - a project containing the raw .avi file of an entire video, ~32 minutes long.
- `entire-video_compressed.zip` - same as above, but compressed using the H.264 codec and saved as .mp4 file.

Each WAZP sample project has the following structure:
```
{project-name}.zip
└── videos
├── {video1-name}.{ext}
├── {video1-name}.metadata.yaml
├── {video2-name}.{ext}
├── {video2-name}.metadata.yaml
└── ...
└── pose_estimation_results
├── {video1-name}{model-name}.h5
├── {video2-name}{model-name}.h5
└── ...
└── WAZP_config.yaml
└── metadata_fields.yaml
```
To learn more about how the sample projects were generated, see `scripts/generate_sample_projects` in the [WAZP GitHub repository](https://github.com/SainsburyWellcomeCentre/WAZP).

### Fetching projects
To fetch the data from GIN, we use the [pooch](https://www.fatiando.org/pooch/latest/index.html) Python package, which can download data from pre-specified URLs and store them locally for all subsequent uses. It also provides some nice utilities, like verification of sha256 hashes and decompression of archives.

The relevant funcitonality is implemented in the `wazp.datasets.py` module. The most important parts of this module are:

1. The `sample_projects` registry, which contains a list of the zipped projects and their known hashes.
2. The `find_sample_projects()` function, which returns the names of available projects per species, in the form of a dictionary.
3. The `get_sample_project()` function, which downloads a project (if not already cached locally), unzips it, and returns the path to the unzipped folder.

Example usage:
```python
>>> from wazp.datasets import find_sample_projects, get_sample_project

>>> projects_per_species = find_sample_projects()
>>> print(projects_per_species)
{'jewel-wasp': ['short-clips_raw', 'short-clips_compressed', 'entire-video_raw', 'entire-video_compressed']}

>>> project_path = get_sample_project('jewel-wasp', 'short-clips_raw')
>>> print(project_path)
/home/user/.WAZP/sample_data/jewel-wasp/short-clips_raw
```

### Local storage
By default, the projects are stored in the `~/.WAZP/sample_data` folder. This can be changed by setting the `LOCAL_DATA_DIR` variable in the `wazp.datasets.py` module.

### Adding new projects
Only core WAZP developers may add new projects to the external data repository.
To add a new poject, you will need to:

1. Create a [GIN](https://gin.g-node.org/) account
2. Ask to be added as a collaborator on the [WAZP data repository](https://gin.g-node.org/SainsburyWellcomeCentre/WAZP) (if not already)
3. Download the [GIN CLI](https://gin.g-node.org/G-Node/Info/wiki/GIN+CLI+Setup#quickstart) and set it up with your GIN credentials, by running `gin login` in a terminal.
4. Clone the WAZP data repository to your local machine, by running `gin get SainsburyWellcomeCentre/WAZP` in a terminal.
5. Add your new projects, followed by `gin commit -m <message> <filename>`. Make sure to follow the [project organisation](#project-organisation) as described above. Don't forget to modify the README file accordingly.
6. Upload the committed changes to the GIN repository, by running `gin upload`. Latest changes to the repository can be pulled via `gin download`. `gin sync` will synchronise the latest changes bidirectionally.
7. Determine the sha256 checksum hash of each new project archive, by running `sha256sum {project-name.zip}` in a terminal. Alternatively, you can use `pooch` to do this for you: `python -c "import pooch; pooch.file_hash('/path/to/file.zip')"`. If you wish to generate a text file containing the hashes of all the files in a given folder, you can use `python -c "import pooch; pooch.make_registry('/path/to/folder', 'hash_registry.txt')`.
8. Update the `wazp.datasets.py` module on the [WAZP GitHub repository](https://github.com/SainsburyWellcomeCentre/WAZP) by adding the new projects to the `sample_projects` registry. Make sure to include the correct sha256 hash, as determined in the previous step. Follow all the usual [guidelines for contributing code](#contributing-code). Additionally, you may want to update the scripts in `scripts/generate_sample_projects`, depending on how you generated the new projects. Make sure to test whether the new projects can be fetched successfully (see [fetching projects](#fetching-projects) above) before submitting your pull request.

You can also perform steps 3-6 via the GIN web interface, if you prefer to avoid using the CLI.

## Template
This package layout and configuration (including pre-commit hooks and GitHub actions) have been copied from the [python-cookiecutter](https://github.com/SainsburyWellcomeCentre/python-cookiecutter) template.
3 changes: 1 addition & 2 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,8 @@ include *.md
recursive-include wazp/*.py
recursive-include wazp/pages *.py

recursive-exclude sample_project *.avi
recursive-exclude sample_project *.h5
recursive-exclude docs *
recursive-exclude scripts *
recursive-exclude * __pycache__
recursive-exclude * *.py[co]

Expand Down
2 changes: 1 addition & 1 deletion docs/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@ myst-parser
nbsphinx
pydata-sphinx-theme
setuptools-scm
sphinx
sphinx>=7.1
sphinx-autodoc-typehints
7 changes: 6 additions & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,17 @@
"**/includes/**",
]

# Don't check the anchors for the following URLs during linkcheck
linkcheck_anchors_ignore_for_url = [
"https://gin.g-node.org/G-Node/Info/wiki/",
]

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
html_theme = "pydata_sphinx_theme"
html_title = "wazp"

# Cutomize the theme
# Customize the theme
html_theme_options = {
"icon_links": [
{
Expand Down
4 changes: 3 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ dependencies = [
"PyYAML",
"shapely",
"openpyxl",
"defusedxml"
"defusedxml",
"pooch",
"tqdm",
]

classifiers = [
Expand Down
Loading
Loading