Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pair developing HDF5 #181 #250

Closed
wants to merge 21 commits into from
Closed

Pair developing HDF5 #181 #250

wants to merge 21 commits into from

Conversation

samcunliffe
Copy link
Member

Resolves #181. Relates to #70.

@samcunliffe samcunliffe added the enhancement New feature or request label Mar 9, 2023
@samcunliffe samcunliffe force-pushed the 181-hdf5-io-pairdev branch 2 times, most recently from 1864c00 to 113f9ca Compare March 9, 2023 14:01
if: ${{ contains(matrix.os, 'ubuntu') }}
run: |
sudo apt-get update
sudo apt-get install libhdf5-dev
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to work, but we may want to statically link when compiling. (Actually, I thought we were doing this...?)

Comment on lines 116 to 111
// get the dataset and dataspace
H5::DataSet dataset = file_->openDataSet(dataset_name);
H5::DataSpace dataspace = dataset.getSpace();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to handle the case where this is a H5::Group, I think.

willGraham01 and others added 19 commits May 19, 2023 10:02
Co-authored-by: Mosè Giordano <[email protected]>
Co-authored-by: Will Graham <[email protected]>
Basic class structure and some failing tests for TDD.
With unfortunate duplication of memory into buffers. To be fixed in the
future with flattened classes.
* Trying to decipher structs

* Further towards reading a struct

* Bypass the stack-smash bug.

MATLAB saves character arrays as uint16s, and HDF5 is unable to interpret them as chars. As such, we need to do manual conversion in the test.

* Rename binary and update contents
…ility (#280)

* Reorganise hdf5 unit tests

- Move uint16 to char/str functions to the unit test utils file
- Create hdf5_and_tdms_objects subdirectory of unit/ to hold unit tests on interaction between hdf5 and tdms classes
- Move the Matrix<double> test from test_hdf5_io into the new subdirectory
- Data files needed for unit tests are defined in a unit_test_utils namespace to avoid redefinition across multiple files

* Create file to test interface and hdf5 interactions

- Add docstrings to interface.h since these are missing and I've just had to work out what they do
- Create a matlab script that can reproduce the class_data.mat file which the hdf5 unit tests will try to create tdms objects from
- Create the barebones test_hdf5_interface file

* HDF5Reader can read from .mat file and produce an InterfaceComponent

* File restructure: accounting for how many tests we are going to have with HDF5

* Prune includes

* Add .mat file for HDF5-TDMS-object unit tests to run.

- Adds scripts to reproduce this data, so in theory a new user can run a short MATLAB script to reproduce this
- Had a play with trying to get setup-matlab to run these scripts before the unit tests, but alas, no.
* Skeleton for new method

* FrequencyVectors use std::vectors instead of our custom object as members. [DOESN'T COMPILE YET]

- FrequencyVectors is now just a struct that uses std::vector datatypes
- SimualationManager is broken, along with most parts of the codebase that use FrequencyVectors
- Pending update to allow code to COMPILE

* Allow TDMS to actually be compiled again. [TESTS STILL FAIL, READER METHOD NOT READY YET]

- Move .mat data generation scripts for unit tests into benchmarking folder
- Adjust paths to unit test data accordingly
- InputMatrices stores the filename so that we can minimise disruption as we transition away from MATLAB

* IT WORKS

* Create the abstract H5Dimensions class to ease reading objects.

- Update HDF5Base::shape_of to return instances of this class
- Add unit tests for class
- Update HDF5Reader method to avoid recycled code

* Remove MATLAB pointers to InterfaceComponents in initialisation.

- simulation_manager can now instantate these members using HDF5Reader rather than still relying on the old InputMatrices object

* Add docstrings to files touched

* Produce unit test binaries on CI rather than track in repo

* - Detaches MATLAB from Cuboid class.
- Turns Cuboid class into a struct because MATLAB is now gone.
- Adds failure-case checks to HDF5Reader::read() functions.

Squashed commit of the following:

commit ce8d15e311e9dbb93bd7d7fbcd6c9cffe69657e3
Author: willGraham01 <[email protected]>
Date:   Fri May 5 15:19:51 2023 +0100

    Rename shapes.h to cuboid.h because that's all it contains

commit fe83f568c3da7491d450693574975df90a3f15e1
Author: willGraham01 <[email protected]>
Date:   Fri May 5 15:13:48 2023 +0100

    Fix faulty logic thrown up by test

commit b37a89f031519c6a356b50f91a12ad8867b78333
Author: willGraham01 <[email protected]>
Date:   Fri May 5 15:01:58 2023 +0100

    Allow tdms to actually compile

commit d7c982b9165e3ae4ccf3f42e0fb8c5cdde55b644
Author: willGraham01 <[email protected]>
Date:   Fri May 5 14:31:43 2023 +0100

    Write unit test and read method for Cuboid. Add failure test for FrequencyVectors

    - Add new .m script to produce an input file with bad object data
    - Update unit_test_utils with this new filepath
    - Write HDF5Reader(Cuboid *cube) method and unit test
    - Add failure-test check for FrequencyVectors using bad data file

commit cf61970d2052ab02ae23822f0edd28e89044734f
Author: willGraham01 <[email protected]>
Date:   Fri May 5 14:12:11 2023 +0100

    Change Cuboid to be a struct because a class is overkill

* Use one master script for setting up .mat files for unit tests.

- setup_unit_tests.m now calls all the other data-generating scripts in the folder to save updating the ci.yaml each time.
* Rework DispersiveMultiLayer class into a struct. TDMS builds.

- xyz_vector struct of 3 vector<double>s introduced
- DispersiveMultiLayer is now a struct with xyz_vector members
- Preserve method for checking if the medium is dispersive
- Temporarily disable unit tests for DispersiveMultiLayer class
- HDF5Reader::read() can now assemble a DispersiveMultiLayer
- HDF5Reader has a method for reading directly into a vector<T>
- H5Dimension can be passed a H5::DataSet as well as a H5::DataSpace

* Update paths now we need HDF5 data as well as matlab data

* TDMS builds and tests HDF5Reader::read_data_from_group (passes, no)

* Fix seg-fault in read_dataset_from_group

* Add DispersiveMultiLayer test

* Update DispersiveMultiLayer tests to not be entangled with the MATLAB jargon

* Update the CI so things actually work

* Move content of benchmark_scripts readme into doc/developers

* Update CI to produce unit test `.mat` and `.hdf5` data (#289)

* Move content of benchmark_scripts readme into doc/developers

* Update CI attempt 1

* Actually use the right syntax

* Force python version to prevent MacOS using python2.7

* Rename files to better match the data they produce

* Aplease windows syntax

* Remove pip cache due to known windows bug: actions/setup-python#436

* Apply suggestions from code review
@willGraham01 willGraham01 force-pushed the 181-hdf5-io-pairdev branch from 05ef9fc to 30899d5 Compare May 19, 2023 09:07
@codecov
Copy link

codecov bot commented May 19, 2023

Codecov Report

Patch coverage: 70% and project coverage change: +24 🎉

Comparison is base (ab6cf65) 25% compared to head (e414f1c) 48%.

❗ Current head e414f1c differs from pull request most recent head 4e12af0. Consider uploading reports for the commit 4e12af0 to get more accurate results

Additional details and impacted files
@@          Coverage Diff          @@
##           main   #250     +/-   ##
=====================================
+ Coverage    25%    48%    +24%     
=====================================
  Files        65     72      +7     
  Lines      3627   7931   +4304     
=====================================
+ Hits        893   3817   +2924     
- Misses     2734   4114   +1380     
Impacted Files Coverage Δ
tdms/include/arrays.h 26% <0%> (-6%) ⬇️
tdms/include/cuboid.h 0% <0%> (ø)
tdms/include/hdf5_io/hdf5_base.h 0% <0%> (ø)
tdms/include/input_matrices.h 0% <ø> (ø)
tdms/include/interface.h 0% <0%> (ø)
tdms/include/output_matrices/output_matrices.h 0% <ø> (ø)
...s/include/simulation_manager/objects_from_infile.h 0% <ø> (ø)
tdms/include/hdf5_io/hdf5_reader.h 27% <27%> (ø)
tdms/src/input_matrices.cpp 82% <50%> (+82%) ⬆️
tdms/src/hdf5_io/hdf5_base.cpp 77% <77%> (ø)
... and 6 more

... and 51 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@willGraham01 willGraham01 marked this pull request as ready for review May 19, 2023 09:27
@willGraham01 willGraham01 marked this pull request as draft May 19, 2023 09:27
@willGraham01
Copy link
Collaborator

willGraham01 commented May 19, 2023

Cherry pick and put this into main in sensible chunks:

Made it ready for review, then decided to not do that since the merge queue will eat it up.

@willGraham01 willGraham01 changed the base branch from main to 181-hdf5-first-part-extracted-from-pairdev-working-branch May 19, 2023 14:03
@willGraham01 willGraham01 changed the base branch from 181-hdf5-first-part-extracted-from-pairdev-working-branch to main May 19, 2023 14:03
@samcunliffe samcunliffe deleted the 181-hdf5-io-pairdev branch June 5, 2023 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use HD5 input and library
2 participants