Restructure website sections

.github/workflows/docs_build_and_deploy.yml

@@ -2,14 +2,13 @@ name: Docs
 
 # Generate the documentation on all merges to main, all pull requests, or by
 # manual workflow dispatch. The build job can be used as a CI check that the
-# docs still build successfully. The deploy job only runs when a tag is
-# pushed and actually moves the generated html to the gh-pages branch
-# (which triggers a GitHub pages deployment).
+# docs still build successfully. The deploy job only runs (1) when a tag is
+# pushed or (2) on manual workflow dispatch. This will move the generated html
+# to the gh-pages branch (which triggers a GitHub pages deployment).
 on:
   push:
     branches:
       - main
-      - new-theme # temporarily add branch to test new theme deployment
     tags:
       - '*'
   pull_request:
@@ -61,10 +60,9 @@ jobs:
         shell: bash
         run: python3 -m pip install -r ./requirements.txt
 
-      # temporarily disabled until we fix broken links
-      # - name: Check links
-      #  shell: bash
-      #  run: make linkcheck
+      - name: Check external links
+        shell: bash
+        run: make linkcheck
 
       # needs to have sphinx.ext.githubpages in conf.py extensions list
       - name: Building documentation
@@ -82,7 +80,7 @@ jobs:
     needs: build_sphinx_docs
     permissions:
       contents: write
-    if: github.event_name == 'push' && github.ref_type == 'tag'
+    if: (github.event_name == 'push' && github.ref_type == 'tag') || github.event_name == 'workflow_dispatch'
     runs-on: ubuntu-latest
     steps:
       - name: Checkout repository

.gitignore

@@ -1,4 +1,4 @@
 # Sphinx documentation
 docs/
-src/api/
-src/api.rst
+src/reference/api/
+src/reference/api.rst

Makefile

@@ -21,8 +21,8 @@ api.rst:
 # Remove all generated files
 clean:
 	rm -rf ./docs
-	rm -f ./src/api.rst
-	rm -rf ./src/api
+	rm -f ./src/reference/api.rst
+	rm -rf ./src/reference/api
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).

make_api_doctree.py

@@ -1,6 +1,5 @@
 import os
 
-
 # modules to ignore in the api doc
 ignore_modules = [
     "aeon.dj_pipeline",
@@ -36,7 +35,7 @@ def make_api_doctree():
         api_head = f.read()
 
     # write file for api doc with header + doctree
-    with open("./src/api.rst", "w") as f:
+    with open("./src/reference/api.rst", "w") as f:
         f.write("..\n  This file is auto-generated.\n\n")
         f.write(api_head)
         f.write(doctree)

readme.md

@@ -2,7 +2,7 @@
 
 This repo contains the source for the currently WIP version of Project Aeon's online docs. 
 
-The docs are built via Sphinx, and hosted via GitHub Pages at: sainsburywellcomecentre.github.io/aeon_docs/. `src/` is the Sphinx source directory, and the site is built and deployed from the `gh-pages` branch. This is handled by a GitHub actions workflow (`.github/workflows/docs_build_and_deploy.yml`). The build job is triggered on each PR, ensuring that the documentation build is not broken by new changes. The deployment job is only triggered whenever a tag is pushed to the main branch.
+The docs are built via [Sphinx](https://www.sphinx-doc.org/en/master/), and hosted via GitHub Pages at [sainsburywellcomecentre.github.io/aeon_docs/](https://sainsburywellcomecentre.github.io/aeon_docs/). `src/` is the Sphinx source directory, and the site is built and deployed from the `gh-pages` branch. This is handled by a GitHub actions workflow (`.github/workflows/docs_build_and_deploy.yml`). The build job is triggered on each PR, ensuring that the documentation build is not broken by new changes. The deployment job is only triggered whenever a tag is pushed to the main branch.
 
 
 ## Building the documentation locally
@@ -28,13 +28,48 @@ Finally, build the documentation:
 ```bash
 make html
 ```
-You can view the local build by opening ``docs/html/index.html`` in a browser.
+You can view the local build by opening `docs/html/index.html` in a browser.
 
 To apply new changes to the documentation, remove all automatically generated files and folders, and rebuild:
 ```bash
 make clean html
 ```
 
+To check that external URLs are correctly resolved, run:
+```bash
+make linkcheck
+```
+
+If the linkcheck step incorrectly marks URLs with valid anchors as broken, 
+you can skip checking the anchors in specific URLs by adding them to 
+`linkcheck_anchors_ignore_for_url` in `src/conf.py`, e.g.: 
+```python
+# linkcheck will skip verifying that anchors exist when checking
+# these URLs
+linkcheck_anchors_ignore_for_url = [
+    "https://example.com",
+]
+```
+
+To skip linkcheck for specific URLs, add them to
+`linkcheck_ignore` in `src/conf.py`, e.g.:
+```python
+# linkcheck will skip checking these URLs entirely
+linkcheck_ignore = [
+    "https://github.com/org/private_repository",
+]
+```
+
+To suppress warnings for expected redirects, add them to 
+`linkcheck_allowed_redirects` in `src/conf.py`, e.g.:
+```python
+# linkcheck will treat redirections from these source URI:canonical URI
+# mappings as "working".
+linkcheck_allowed_redirects = {
+    r"https://zenodo\.org/doi/.*": r"https://zenodo\.org/records/.*",
+}
+```
+
 ## Project Aeon Organization Overview
 
 ProjectAeon is a collaborative effort to perform behavioral neuroscience experiments where the behavior and neural activity of freely moving animals engaging in a complex task are continuously recorded. This project is contributed to by researchers and support staff at UCL's SWC, Neurogears, and Datajoint.
@@ -67,7 +102,7 @@ To be granted these credentials, please send a single email to all contact parti
 Project Aeon's main library for interfacing with acquired data. Contains Python modules for raw data file io, data querying, data processing, data qc, database ingestion, and building computational data pipelines. This is the main user repository.
 
 > [!NOTE]
-> All experiment data is acquired and/or triggered and/or synced by [Harp devices](https://www.cf-hw.org/harp). Code in the 'aeon_acquisition' and 'aeon_mecha' repos makes use of the [Harp protocol](https://github.com/harp-tech/protocol) during data acquisition, raw data file writing, and raw data file reading. In the 'harp-tech/protocol' Github repo, you can find documentation on [Harp device operation and common registers](https://github.com/harp-tech/protocol/blob/master/Device%201.0%201.4%2020200901.pdf), the [Harp binary protocol](https://github.com/harp-tech/protocol/blob/master/Binary%20Protocol%201.0%201.1%2020180223.pdf), and [Harp clock synchronization](https://github.com/harp-tech/protocol/blob/master/Synchronization%20Clock%201.0%201.0%2020200712.pdf).
+> All experiment data is acquired and/or triggered and/or synced by [Harp devices](https://www.cf-hw.org/harp). Code in the 'aeon_acquisition' and 'aeon_mecha' repos makes use of the [Harp protocol](https://harp-tech.org/articles/about.html) during data acquisition, raw data file writing, and raw data file reading. See also the documentation on [Harp device operation and common registers](https://harp-tech.org/protocol/Device.html), the [Harp binary protocol](https://harp-tech.org/protocol/BinaryProtocol-8bit.html), and [Harp clock synchronization](https://harp-tech.org/protocol/SynchronizationClock.html).
 
 ### [aeon_experiments](https://github.com/SainsburyWellcomeCentre/aeon_experiments)
 

src/_templates/api_head.rst

@@ -1,6 +1,6 @@
 .. _api:
 
-Developer API
+API Reference
 =============
 
 .. toctree::

src/dev_practices/code_of_conduct.md → src/about/code_of_conduct.md

@@ -43,4 +43,4 @@ Project maintainers who do not follow or enforce the Code of Conduct in good fai
 
 ## Attribution
 
-This Code of Conduct is adapted from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4.
+This Code of Conduct is adapted from the [Contributor Covenant, version 1.4](https://www.contributor-covenant.org/version/1/4/).

src/about/index.md

@@ -0,0 +1,38 @@
+(target-about)=
+# About Project Aeon
+
+Development of Project Aeon is supported by... 
+
+The goals of Project Aeon are to
+* ...
+* ...
+
+For an in-depth look at the unique challenges and opportunities presented by the datasets collected in Project Aeon, please refer to our [Design Considerations](target-design-considerations).
+If you have any questions or need further information, feel free to reach out to the [project maintainers](target-project-maintainers).
+
+(target-project-maintainers)=
+## Project Maintainers
+
+* Jai Bhagat ([email protected])
+* Gonçalo Lopes ([email protected])
+* Dario Campagner ([email protected])
+
+(target-project-contributors)=
+## Contributors
+
+List researchers and support staff at UCL's SWC, Neurogears, and Datajoint...
+
+## Code of Conduct
+
+Project Aeon is dedicated to providing a welcoming and positive experience for all its participants. We encourage respectful and considerate behavior, and we expect our community members to adhere to these principles. Please read our [Code of Conduct](target-code-of-conduct) for more details on our community guidelines and expectations.
+
+## Licence
+
+Insert link to licence here.
+
+:::{toctree}
+:maxdepth: 1
+:hidden:
+
+design_considerations
+code_of_conduct

src/conf.py

@@ -28,8 +28,8 @@
 # -- Project information -----------------------------------------------------
 
 project = "aeon_docs"
-copyright = f"2022–{date.today().year}, Jai Bhagat, Goncalo Lopes, Chang Huan Lo"
 author = "Jai Bhagat, Goncalo Lopes, Chang Huan Lo"
+copyright = f"2022–{date.today().year}, {author}"
 organisation = "Sainsbury Wellcome Centre"
 
 
@@ -48,7 +48,6 @@ def get_current_release_tag():
 
 # The full version, including alpha/beta/rc tags
 release = get_current_release_tag()
-# release = "0.1.0"
 
 # GitHub repo URL
 github_url = f"https://github.com/{organisation.replace(' ', '')}/{project}"
@@ -85,7 +84,12 @@ def get_current_release_tag():
 templates_path = ["_templates"]
 
 autosummary_generate = True
-autodoc_default_flags = ["members", "inherited-members"]
+autodoc_default_options = {
+    "members": True,
+    "inherited-members": True,
+    "show-inheritance": True,
+    "undoc-members": True,
+}
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -125,3 +129,26 @@ def get_current_release_tag():
 html_css_files = [
     "https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.1.1/css/all.min.css"
 ]
+
+# linkcheck will skip checking these URLs entirely
+linkcheck_ignore = [
+    "https://github.com/SainsburyWellcomeCentre/aeon_experiments",  # This is currently a private repository
+]
+
+# linkcheck will treat redirections from these source URI:canonical URI
+# mappings as "working".
+linkcheck_allowed_redirects = {
+    r"https://doi\.org/10\.5281/zenodo\..*": r"https://zenodo\.org/records/.*",
+    r"https://zenodo\.org/doi/.*": r"https://zenodo\.org/records/.*",
+}
+
+myst_url_schemes = {
+    "http": None,
+    "https": None,
+    "ftp": None,
+    "mailto": None,
+    "aeon-mecha-github": "https://github.com/SainsburyWellcomeCentre/aeon_mecha/{{path}}",
+    "semver": "https://semver.org/",
+    "harp-tech": "https://harp-tech.org/{{path}}#{{fragment}}",
+    "python-pep": "https://peps.python.org/pep-{{path}}",
+}

src/dev_practices/index.md → src/contributor/index.md

@@ -1,12 +1,12 @@
-# Community
+# Contributor Guide
 
 ## Software Development Life Cycle (SDLC)
 
 Our SDLC roughly follows the [iterative model](https://www.tutorialspoint.com/sdlc/sdlc_iterative_model.htm).
 
 ## Versioning
 
-We version all the following, according to [SemVer](http://semver.org/) numbering: 
+We version all the following, according to [SemVer](semver:) numbering: 
 
 - Experiments, by name, including full hardware specs (arena, I/O devices, 
   acquisition computer, etc.)
@@ -27,10 +27,10 @@ We use Github Actions to run CI. We run unit tests on Github Virtual Machines on
 
 ## Contributing
 
-Each repository roughly follows the [github flow](https://guides.github.com/introduction/flow/) (which is adapted from the more general 
+Each repository roughly follows the [github flow](https://docs.github.com/en/get-started/quickstart/github-flow) (which is adapted from the more general 
 [gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)). 
 
-In brief, each of our repos has 'main' and 'prod' branches. Feature and bug fix branches are branched off of 'main', with Pull Requests sent back into 'main'. 'main' contains the full commit history of the project, up to the latest stable commit. Upon merges into 'main', a squash merge is performed into 'prod', such that 'prod' contains an abbreviated commit history of the project, with commits pertaining only to Pull Request merges. 'prod' thus serves as a "production" branch to allow for easier readability of project history and easier reversion of uncaught bug commits. At certain agreed upon timepoints we create "stable releases" (available in the "releases" section of the repository) which serve as a snapshot of the code at the time, version numbered according to [SemVer](http://semver.org/).
+In brief, each of our repos has 'main' and 'prod' branches. Feature and bug fix branches are branched off of 'main', with Pull Requests sent back into 'main'. 'main' contains the full commit history of the project, up to the latest stable commit. Upon merges into 'main', a squash merge is performed into 'prod', such that 'prod' contains an abbreviated commit history of the project, with commits pertaining only to Pull Request merges. 'prod' thus serves as a "production" branch to allow for easier readability of project history and easier reversion of uncaught bug commits. At certain agreed upon timepoints we create "stable releases" (available in the "releases" section of the repository) which serve as a snapshot of the code at the time, version numbered according to [SemVer](semver:).
 
 When contributing to any repository, the change to be made should first be discussed in a Github Discussion or a Github Issue. Thereafter, contributors should create a new branch (branched off of 'main') that contains the changes/additions they wish to make, and then create a pull request for merging this branch into 'main'.
 
@@ -40,21 +40,17 @@ All pull requests will be reviewed by the [project maintainers](target-project-m
 
 2) As necessary, ensure `changelog`, `readme`, config and doc files are updated.
 
-3) When a branch is ready to be merged back into 'main', always make sure to first pull 'main' locally, then rebase the feature branch onto 'main' (cleaning up any merge conflicts as necessary), before merging the PR. The squash merge into 'prod' can be handled via CI. E.g., see [here](https://github.com/SainsburyWellcomeCentre/aeon_mecha/tree/main/.github/workflows/squash_merge_to_prod.yml)
+3) When a branch is ready to be merged back into 'main', always make sure to first pull 'main' locally, then rebase the feature branch onto 'main' (cleaning up any merge conflicts as necessary), before merging the PR. The squash merge into 'prod' can be handled via CI. E.g., see [here](aeon-mecha-github:blob/main/.github/workflows/squash_merge_to_prod.yml)
 
 ## Style and Documentation Guidelines
 
 Please see our [style and documentation guidelines](target-style-doc-guidelines).
 
 We also believe in the [readme manifesto](http://thinkinghard.com/blog/TheREADMEManifesto.html), which says that `readme` files should provide at least a general description that covers _all_ of a project's files, and that one `readme` per subdirectory is generally good practice.
 
-## Code of Conduct
-
-Please see our [code of conduct](target-code-of-conduct).
 
 :::{toctree}
 :maxdepth: 1
 :hidden:
 
 style_and_doc_guidelines
-code_of_conduct

src/dev_practices/style_and_doc_guidelines.md → src/contributor/style_and_doc_guidelines.md

@@ -3,7 +3,7 @@
 
 The purpose of this document is to establish Project Aeon code style and documentation guidelines.
 
-We generally adhere to [pep8](https://www.python.org/dev/peps/pep-0008/), [pep257](https://www.python.org/dev/peps/pep-0257/), and [google's style guide](https://google.github.io/styleguide/pyguide.html) for code and docstring style guidelines. We run black, ruff, and pyright to format our Python code, the config settings for which can be found in a repository's `pyproject.toml`.
+We generally adhere to [pep8](python-pep:0008), [pep257](python-pep:0257), and [google's style guide](https://google.github.io/styleguide/pyguide.html) for code and docstring style guidelines. We run black, ruff, and pyright to format our Python code, the config settings for which can be found in a repository's `pyproject.toml`.
 
 ## General Guidelines
 

src/repositories/index.md → src/getting_started/index.md

@@ -1,8 +1,20 @@
 (target-repositories)=
-# Repositories
+# Getting Started
+
+## Introduction to Project Aeon
+
+## Repositories
 
 :::{important}
-You must be an SWC Github 'aeon' project member to view some of these repositories.
+You must be an SWC Github 'aeon' project member to view some of these repositories. The required sets of credentials are as follows: 
+- Microsoft Teams: contact Jai Bhagat, Goncalo Lopes, or Dario Campagner
+- SWC Github organization: contact SWC Helpdesk ([email protected])
+- SWC Github 'aeon' project: contact Jai Bhagat or Goncalo Lopes
+- SWC HPC: contact SWC Helpdesk
+- 'aeon' HPC Linux group: contact SWC Helpdesk
+- Datajoint database username: contact Thinh Nguyen ([email protected])
+
+To be granted these credentials, please send a single email to all contact parties requesting this access.
 :::
 
 ::::{grid} 1 2 2 2
@@ -23,7 +35,7 @@ Aeon experiment workflows written in the Bonsai visual programming language
 :::{grid-item-card} {fas}`computer;sd-text-primary` aeon_acquisition
 :link: https://github.com/SainsburyWellcomeCentre/aeon_acquisition
 :link-type: url
-Source code for the 'aeon_acquisition' Bonsai package used in Aeon experiment workflows
+Source code for the `aeon_acquisition` Bonsai package used in Aeon experiment workflows
 :::
 
 :::{grid-item-card} {fas}`chart-line;sd-text-primary` aeon_analysis
@@ -46,5 +58,5 @@ Source code for pellet delivery via feeders used in Aeon experiments
 ::::
 
 :::{note}
-All experiment data is acquired and/or triggered and/or synced by [Harp devices](https://www.cf-hw.org/harp). Code in the 'aeon_acquisition' and 'aeon_mecha' repos makes use of the [Harp protocol](https://github.com/harp-tech/protocol) during data acquisition, raw data file writing, and raw data file reading. In the 'harp-tech/protocol' Github repo, you can find documentation on [Harp device operation and common registers](https://github.com/harp-tech/protocol/blob/master/Device%201.0%201.4%2020200901.pdf), the [Harp binary protocol](https://github.com/harp-tech/protocol/blob/master/Binary%20Protocol%201.0%201.1%2020180223.pdf), and [Harp clock synchronization](https://github.com/harp-tech/protocol/blob/master/Synchronization%20Clock%201.0%201.0%2020200712.pdf).
+All experiment data is acquired and/or triggered and/or synced by [Harp devices](https://www.cf-hw.org/harp). Code in the 'aeon_acquisition' and 'aeon_mecha' repos makes use of the [Harp protocol](harp-tech:articles/about) during data acquisition, raw data file writing, and raw data file reading. See also the documentation on [Harp device operation and common registers](harp-tech:protocol/Device), the [Harp binary protocol](harp-tech:protocol/BinaryProtocol-8bit), and [Harp clock synchronization](harp-tech:protocol/SynchronizationClock).
 :::