diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c510bd257..b8f85a0f4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -189,4 +189,4 @@ jobs: shell: bash run: | export OMP_NUM_THREADS=2 - pytest ${GITHUB_WORKSPACE}/tdms/tests/system/ -s -x + pytest ${GITHUB_WORKSPACE}/tdms/tests/system/ --ignore=${GITHUB_WORKSPACE}/tdms/tests/system/test_regen.py -s -x diff --git a/.gitignore b/.gitignore index 974107a95..1c904f34e 100644 --- a/.gitignore +++ b/.gitignore @@ -12,7 +12,7 @@ html/ # (py)tests and test data **.pyc **/.pytest_cache/ -tdms/tests/system/data +tdms/tests/system/data/*.zip **.mat # text editor files diff --git a/doc/developers.md b/doc/developers.md index f856b54b7..6c7392254 100644 --- a/doc/developers.md +++ b/doc/developers.md @@ -222,19 +222,78 @@ It's good practice, and reassuring for your pull-request reviewers, if new C++ f The full system tests are written in Python 3, and call the `tdms` executable for known inputs and compare to expected outputs. We use [pytest](https://docs.pytest.org) and our example data is provided as zip files on [zenodo](https://zenodo.org/). -There are a few [python packages you will need](https://github.com/UCL/TDMS/blob/main/tdms/tests/requirements.txt) before running the tests so run: +There are a few [python packages you will need](https://github.com/UCL/TDMS/blob/main/tdms/tests/requirements.txt) before you are able to run the tests, which can be installed by executing: ```{.sh} python -m pip install -r tdms/tests/requirements.txt ``` if you don't already have them. +You'll then need to [compile](#compiling) `tdms` with `-DBUILD_TESTING=ON`. +Once compiled, the system tests can be run by invoking `pytest` and pointing it to the `tdms/tests/system` directory. +For example, from the build directory: +```{.sh} +$ pwd +/path/to/repository/TDMS/tdms/build +$ pytest ../tests/system/ +``` +The [`test_system.py`](https://github.com/UCL/TDMS/blob/main/tdms/tests/system/test_system.py) script runs each system test in sequence. -When you run the tests for the first time, the example data will be downloaded to `tdms/tests/system/data` (which is [ignored by git](https://github.com/UCL/TDMS/blob/main/.gitignore)). -Subsequent runs of the test will not re-download unless you manually delete the zip file. +**[`test_regen.py`](https://github.com/UCL/TDMS/blob/main/tdms/tests/system/test_regen.py) and [`tdms_testing_class.py`](https://github.com/UCL/TDMS/blob/main/tdms/tests/system/tdms_testing_class.py.py) will replace `test_system.py` and `read_config.py` when the [input overhaul](https://github.com/UCL/TDMS/issues/70) is complete.** -A good example of running the `tdms` executable for a given input and expected output is [test_arc01.py](https://github.com/UCL/TDMS/blob/main/tdms/tests/system/test_arc01.py) +When you run the tests for the first time, test data is downloaded to `tdms/tests/system/data` (and will be [ignored by git](https://github.com/UCL/TDMS/blob/main/.gitignore)). +These reference input files contain arrays for: the incident electric field, the computational grid, etc. which are needed by the simulation, and have been generated by a trusted version of the relevant MATLAB scripts. +Subsequent runs of the tests will not re-download unless you manually delete the zip file(s). -You need to [compile](#compiling) `tdms`, then the system tests can be run, e.g. from the build directory: +The system tests for `tdms` are configured with yaml files in the `data/input_generation/` directory. +They are named `config_XX.yaml` where `XX` matches the ID of the system test, which themselves are named `arc_XX` by historical convention. +This should also match the `test_id` field in the configuration file itself. +The _reference outputs_ or _reference data_ are a collection of `.mat` files, produced from the _reference inputs_ by a trusted version of the `tdms` executable. +We test for regression using these reference files. -```{.sh} -pytest ../tests/system/ -``` +A given system test typically has two calls to the `tdms` executable; one for when there is no scattering object present, and one for when there is some obstacle. +More than two runs in a test might be due to the use of band-limited interpolation over cubic interpolation. +Each call to the executable has a reference input and reference output. +In the scripts, a given execution is called by `tests.utils.run_tdms` which wraps a [subprocess.Popen](https://docs.python.org/3/library/subprocess.html#subprocess.Popen). + +#### Workflow of a System Test + +The workflow of a particular system test `arc_XX` is: +- Locally generate the reference inputs using `data/input_generation/WHATISTHESCRIPTNAME.py`. + - `arc_XX` fails if its reference input cannot be successfully generated. + This indicates a failure in the scripts and/or functions in the `data/input_generation/{bscan,matlab}` directories. +- Fetch the reference outputs from [Zenodo](https://zenodo.org/record/7440616/files). +- For each run, named `run_id` in `arc_XX`: + - Execute the call to `tdms` corresponding to `run_id`. + - Compare the output of each run to the corresponding reference data. + - `run_id` fails if the output produced differs significantly from the reference data. + - Outputs produced by `run_id` are cleaned up. +- `arc_XX` fails if any one of its runs fail. Failed runs are reported by name. +- Reference inputs are cleaned up. +- `arc_XX` passes if this step is reached successfully. + +Due to [licensing issues regarding running `MATLAB` on GitHub runners](https://github.com/matlab-actions/setup-matlab/issues/13), we cannot use `matlabengine` to regenerate the reference input data during CI. (Although we are currently thinking of removing the `MATLAB` dependency which will then enable us to resolve this issue). The work-in-progress `test_regen.py` workflow can still be run locally through `pytest`, however in addition to `requirements.txt` you will also need to [install `matlabengine`](https://uk.mathworks.com/help/matlab/matlab_external/install-the-matlab-engine-for-python.html). See the [MathWorks page](https://uk.mathworks.com/help/matlab/matlab_external/install-the-matlab-engine-for-python.html) link for detailed requirements. You will need (at least) `Python >= 3.8` and a licensed version of `MATLAB`. + +### Generating Input Data for the System Tests + +The system tests rely on `.mat` input files that are generated through a series of MATLAB function calls and scripts. This directory contains the functionality to automatically (re)generate this input data, which serves two purposes: +- The `.mat` _input_ files do not need to be uploaded and fetched from Zenodo each time the system tests are run. They can be generated locally instead. + - Note that the reference output files corresponding to these inputs still need to be downloaded from Zenodo. +- We track changes to the way we handle inputs to `tdms`, and the system tests. Ensuring we test against unexpected behaviour due to input changes. + +#### (Re)generation of the Data + +(Re)generating the input data for a particular test case, `arc_XX`, is a three-step process: +1. Determine variables, filenames, and the particular setup of `arc_XX`. This information is stored in the corresponding `config_XX.yaml` file. For example, is an illumination file required? What are the spatial obstacles? What is the solver method? +1. Call the `run_bscan.m` function (and sub-functions in `./matlab`) using the information in `config_XX.yaml` to produce the `.mat` input files. Each test case requires an input file (`input_file_XX.m`) which defines test-specific variables (domain size, number of period cells, material properties, etc) which are too complex to specify in a `.yaml` file. +1. Clean up the auxillary `.mat` files that are generated by this process. In particular, any `gridfiles.mat`, illumination files, or other `.mat` files that are temporarily created when generating the input `.mat` file. + +#### Contents of the `data/input_generation` Directory (and subdirectories) + +The `run_bscan` function is inside the `bscan/` directory. + +The `matlab/` directory contains functions that `run_bscan` will need to call on during the creation of the input data. This in particular includes the `iteratefdtd_matrix` function, which does the majority of the work in setting up gridfiles, illumination files, and the `.mat` inputs themselves. + +The `generate_test_input.py` file contains `.py` files that the system tests can invoke to regenerate the input data. Since the system test framework uses `pytest`, but the data generation requires `MATLAB` (for now), we use `Python` to read in and process the information that each test requires, and then call `run_bscan` with the appropriate commands from within Python. + +The `regenerate_all.py` file will work through all of the `config_tc.yaml` files in the directory and regenerate the input `.mat` data corresponding to each. + +The remaining `config_XX.yaml` and `input_file_XX.m` files are as mentioned in [the previous section](#regeneration-of-the-data). These contain the information about each test that Python and `run_bscan` will need to regenerate the input files. diff --git a/tdms/tests/matlab/test_fdtdduration.m b/tdms/tests/matlab/test_fdtdduration.m index 368976cf2..5e91fac86 100644 --- a/tdms/tests/matlab/test_fdtdduration.m +++ b/tdms/tests/matlab/test_fdtdduration.m @@ -1,7 +1,7 @@ % Define the tests as all the locally defined functions function tests = test_iteratefdtd_matrix_function - addpath('../../matlab/', 'data/'); + addpath('../system/data/input_generation/matlab', 'data/'); tests = functiontests(localfunctions); end diff --git a/tdms/tests/system/data/__init__.py b/tdms/tests/system/data/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tdms/matlab/.clang-format b/tdms/tests/system/data/input_generation/.clang-format similarity index 100% rename from tdms/matlab/.clang-format rename to tdms/tests/system/data/input_generation/.clang-format diff --git a/tdms/tests/system/data/input_generation/__init__.py b/tdms/tests/system/data/input_generation/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tdms/tests/system/data/input_generation/bscan/run_bscan.m b/tdms/tests/system/data/input_generation/bscan/run_bscan.m new file mode 100644 index 000000000..d27c203fe --- /dev/null +++ b/tdms/tests/system/data/input_generation/bscan/run_bscan.m @@ -0,0 +1,46 @@ +function [] = run_bscan(test_directory, input_filename) +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%This function generates the files used as input to the executeable +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +%% Create directory into which to place the input files, if it doesn't exist already +dir_to_place_input_mats = test_directory;%strcat(test_directory,'/in') +if ~exist(dir_to_place_input_mats, 'dir') + mkdir(dir_to_place_input_mats); +end + +%% Generate the input file + +%start by defining the coordinates of the computational grid +[x,y,z,lambda] = fdtd_bounds(input_filename); + +%15 micron radius cylinder +rad = 15e-6; +%refractive index of cylinder +refind = 1.42; + +%insert a cylinder at the origin +y = 0; +[X,Y,Z] = ndgrid(x,y,z); + +%generate scattering matrix +I = zeros(size(X)); +%set all Yee cells within the cylinder to have index of 1 +I( (X.^2 + Z.^2) < rad^2 ) = 1; +I( (end-3):end,1,:) = 0; +I( :, 1, (end-3):end) = 0; +inds = find(I(:)); +[ii,jj,kk] = ind2sub(size(I), inds); +composition_matrix = [ii jj kk ones(size(ii))]; +material_matrix = [1 refind^2 1 0 0 0 0 0 0 0 0]; + +save('gridfile_cyl', 'composition_matrix', 'material_matrix'); +%setup free space matrix and save +composition_matrix = []; +save('gridfile_fs', 'composition_matrix', 'material_matrix'); + +%generate tdms executable input files +iteratefdtd_matrix(input_filename,'filesetup',strcat(dir_to_place_input_mats,'/pstd_cyl_input'),'gridfile_cyl.mat',''); +iteratefdtd_matrix(input_filename,'filesetup',strcat(dir_to_place_input_mats,'/pstd_fs_input'),'gridfile_fs.mat',''); + +end diff --git a/tdms/tests/system/data/input_generation/config_01.yaml b/tdms/tests/system/data/input_generation/config_01.yaml new file mode 100644 index 000000000..7762ff7f8 --- /dev/null +++ b/tdms/tests/system/data/input_generation/config_01.yaml @@ -0,0 +1,20 @@ +test_id: '01' +tests: + fs_bli: + input_file: pstd_fs_input.mat + reference: pstd_fs_bli_reference.mat + fs_cubic: + input_file: pstd_fs_input.mat + reference: pstd_fs_cubic_reference.mat + cubic_interpolation: True + cyl_bli: + input_file: pstd_cyl_input.mat + reference: pstd_cyl_bli_reference.mat + cyl_cubic: + input_file: pstd_cyl_input.mat + reference: pstd_cyl_cubic_reference.mat + cubic_interpolation: True +input_generation: + input_file: input_file_01.m + spatial_obstacles: ["fs", "cyl"] + illumination_input_file: diff --git a/tdms/tests/system/data/input_generation/generate_test_input.py b/tdms/tests/system/data/input_generation/generate_test_input.py new file mode 100644 index 000000000..5044cbf48 --- /dev/null +++ b/tdms/tests/system/data/input_generation/generate_test_input.py @@ -0,0 +1,102 @@ +import os +from glob import glob +from pathlib import Path + +import matlab.engine as matlab +import yaml +from matlab.engine import MatlabEngine + +LOCATION_OF_THIS_FILE = os.path.dirname(os.path.abspath(__file__)) +# Additional options for running matlab on the command-line +MATLAB_OPTS_LIST = ["-nodisplay", "-nodesktop", "-nosplash", "-r"] +MATLAB_STARTUP_OPTS = " ".join(MATLAB_OPTS_LIST) +# Paths to matlab functions not in LOCATION_OF_THIS_FILE +MATLAB_EXTRA_PATHS = [ + os.path.abspath(LOCATION_OF_THIS_FILE + "/bscan"), + os.path.abspath(LOCATION_OF_THIS_FILE + "/matlab"), +] + + +def run_bscan( + test_directory: Path | str, input_filename: Path | str, engine: MatlabEngine +) -> None: + """Wrapper for running the run_bscan MATLAB function in the MATLAB engine provided. + + MatlabEngine cannot parse Path objects so file and directory paths must be cast to string when calling. + + The bscan/ and matlab/ directories are assumed to already be in the + includepath of the engine instance, so that the run_bscan and supporting + MATLAB files can be called. + """ + # function [] = run_bscan(test_directory, input_filename) + engine.run_bscan(str(test_directory), str(input_filename), nargout=0) + return + + +def start_MatlabEngine_with_extra_paths( + working_directory: str | Path | None = None, +) -> MatlabEngine: + """Starts a new MatlabEngine and adds the bscan/ and matlab/ folders to its path, which are required to be in scope when regenerating the input data. + + :param working_directory: The working directory to start the MatlabEngine in. Should be an absolute path. Defaults to the working directory of the currently executing script if not passed. + :returns: MatlabEngine instance with the additional bscan/ and matlab/ files on the MATLABPATH. + """ + engine = matlab.start_matlab(MATLAB_STARTUP_OPTS) + # Change to requested working directory if provided + if working_directory: + engine.cd(str(working_directory)) + # Append tdms scripts and functions to MATLABPATH + for path in MATLAB_EXTRA_PATHS: + engine.addpath(path) + return engine + + +def generate_test_input( + config_filepath: Path | str, engine: MatlabEngine | None = None +) -> None: + """(re)Generates the input data (.mat files) contained in the config file, using the MATLAB session provided. + + This function is equivalent to running the run_{pstd,fdtd}_bscan.m scripts on the (test corresponding to the) config file in question. + + :param config_filepath: The path to the config file containing information about this system test + :param engine: The MATLAB session to run the run_bscan function within. A session will be created and quit() if one is not provided. + """ + with open(config_filepath, "r") as file: + config_data = yaml.safe_load(file) + + # ID of the test we are generating input data for + test_id = config_data["test_id"] + # Absolute path to the directory into which the input data should be placed + test_dir = Path(LOCATION_OF_THIS_FILE, "arc_" + test_id) + # Ensure that the directory to place the output into exists, or create it otherwise + if not test_dir.exists(): + print(f"The Path {test_dir} does not exist - creating now") + os.mkdir(test_dir) + elif not test_dir.is_dir(): + raise RuntimeError(f"{test_dir} is not a directory!") + # else: the directory already exists, we don't need to do anything + + # Extract necessary input data generation information + generation_info = config_data["input_generation"] + # Fetch the location of the input file that generates the binary .mat input + input_file = Path(LOCATION_OF_THIS_FILE, generation_info["input_file"]) + if not input_file.exists(): + raise RuntimeError(f"{input_file} does not exist") + # Fetch the spatial obstacles + obstacles = generation_info["spatial_obstacles"] + + # Determine if we need to create our own MATLAB session + # Explicit instance check since MatlabEngine may not have implicit casts/ interpretations + engine_provided = isinstance(engine, MatlabEngine) + if not engine_provided: + # Start a new Matlab engine operating in the test directory + engine = start_MatlabEngine_with_extra_paths(working_directory=test_dir) + + run_bscan(test_dir, input_file, engine) + + # Quit our temporary MATLAB session, if we started one + if not engine_provided: + engine.quit() + # Cleanup auxillary .mat files that are placed into this directory + for aux_mat in sorted(glob(LOCATION_OF_THIS_FILE + "/*.mat")): + os.remove(aux_mat) diff --git a/tdms/tests/system/data/input_generation/input_file_01.m b/tdms/tests/system/data/input_generation/input_file_01.m new file mode 100644 index 000000000..b30fa5751 --- /dev/null +++ b/tdms/tests/system/data/input_generation/input_file_01.m @@ -0,0 +1,143 @@ +%specify the interpolation method (1 or not present - cubic, 2 - bandlimited) +intmethod = 1; + +%these are not involved in the formal input file spec +lambda = 1300e-9; + +%size of Yee cell in metres +delta.x = lambda/4; +delta.y = lambda/4; +delta.z = lambda/4; + + +%define the grid size, a square of side 1.5 wavelengths +I = 256; +J = 0; +K = 256; +%K = 5500; + +%order of the PML conductivity profile curve +n = 4; + +%maximum reflection at PML +R0 = 1e-7; + +%number of PML cells in each direction +Dxl = 10; +Dxu = 10; +Dyl = 0; +Dyu = 0; +Dzl = 10; +Dzu = 10; + +%courant time step +dt = 2/sqrt(2)/pi*delta.x/(3e8/1.35)*.95; + +%define the number of time steps +Nt = 500; +%Nt=12000; + +%water +epsr = [1.35^2]; +mur = [1]; +kappa_max = [1]; +multilayer = []; + +%frequency in Hz +f_an = asin( 2*pi/1300e-9*2.997924580105029e+08*dt/2)/(pi*dt); + +%This is where we define the planes where the incident waveforms +%are introduced. The variable interface has 6 members, I0, I1, J0, +%J1, K0, K1 which are 1x2 vectors. The first entry is the position +%of the plane, in local coordinates and the second entry os whether +%or not to apply the interface condition at that plane +interface.I0 = [5 0]; +interface.I1 = [I-5 0]; +interface.J0 = [5 0]; +interface.J1 = [J-5 0]; +interface.K0 = [10 1]; +interface.K1 = [K-5 0]; + +%not used as run mode is complete +outputs_array ={}; + +%these are the function names used to generate the field +efname = 'efield_gauss'; +hfname = 'hfield_focused_equiv'; + +%this is the z value at which the field is launched, in metres +z_launch = 0; + + +%this defines the point about which the illumination is centred in +%the so called 'interior' coordinate system +illorigin = [floor(I/2) floor(J/2) floor(K/2)]; + +%the wavelength width (in m). This corresponds to the FWHM of the +wavelengthwidth = 120e-9; + +%this defines the run mode of the simulation, can be 'analyse' or +%'complete'. 'analyse' means that sub results can be saved using +%the statements in outputs_array. When complete is specified, only +%the final results will be saved using the outputs_array statements. +%runmode = 'analyse'; +runmode = 'complete'; + + +%this is the kind of source mode, can be 'steadystate' or +sourcemode = 'pulsed'; + +%this determines whether or not to extract phasors in the volume of +%the grid +exphasorsvolume = 1; + +%this determines whether or not to extract phasors around a +%specified surface +exphasorssurface = 0; + +%this specifies a surface to extract the phasors at. These +%quantities are in interior coordinate system; +%has the form [I0 I1 J0 J1 K0 K1] which defines the extremes of a +%cuboid wihch defines the surface to extract phasors at +%These should be set so that the interpolation scheme can work +phasorsurface = [5 I-5 1 1 20 K-5]; + +%could be '3' 'TE' or 'TM' +dimension = '3'; + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +Nlambda = 512; +lambda0 = 1300e-9; +dlambda = 170e-9; +b = 4*sqrt(log(2))*lambda0^2/(2*pi*3e8*dlambda); +omega0 = 2*pi*3e8/lambda0; +%omega1 = 2*pi*3e8/(lambda0-dlambda/2); + +omega_min = omega0 - sqrt(4/b^2*log(10^3)); +omega_max = omega0 + sqrt(4/b^2*log(10^3)); + +lambda_min = 3e8*2*pi/omega_max; +lambda_max = 3e8*2*pi/omega_min; + + +omega_vec = linspace(omega_min,omega_max,Nlambda); +k_vec = omega_vec/2.997924580105029e+08; +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +f_ex_vec = asin( k_vec*2.997924580105029e+08*dt/2)/(pi*dt); + +%ignore all below here +exdetintegral=0; +k_det_obs=10; +%k_obs = k_det_obs; +NA_det=(7e-3/2)/36e-3; +%NA = NA_det; +beta_det=25/36; +detmodevec=1:3; +detsensefun='gaussian_d_telesto_matlab'; +air_interface = []; + +det_trans_x = (-30:2:30)*1e-6; +det_trans_y = (-30:2:30)*1e-6; + +illspecfun = ''; diff --git a/tdms/tests/system/data/input_generation/matlab/.clang-format b/tdms/tests/system/data/input_generation/matlab/.clang-format new file mode 100644 index 000000000..47a38a93f --- /dev/null +++ b/tdms/tests/system/data/input_generation/matlab/.clang-format @@ -0,0 +1,2 @@ +DisableFormat: true +SortIncludes: Never diff --git a/tdms/matlab/efield_gauss.m b/tdms/tests/system/data/input_generation/matlab/efield_gauss.m similarity index 100% rename from tdms/matlab/efield_gauss.m rename to tdms/tests/system/data/input_generation/matlab/efield_gauss.m diff --git a/tdms/matlab/fdtd_bounds.m b/tdms/tests/system/data/input_generation/matlab/fdtd_bounds.m similarity index 100% rename from tdms/matlab/fdtd_bounds.m rename to tdms/tests/system/data/input_generation/matlab/fdtd_bounds.m diff --git a/tdms/matlab/fdtdduration.m b/tdms/tests/system/data/input_generation/matlab/fdtdduration.m similarity index 100% rename from tdms/matlab/fdtdduration.m rename to tdms/tests/system/data/input_generation/matlab/fdtdduration.m diff --git a/tdms/matlab/fdtdts.m b/tdms/tests/system/data/input_generation/matlab/fdtdts.m similarity index 100% rename from tdms/matlab/fdtdts.m rename to tdms/tests/system/data/input_generation/matlab/fdtdts.m diff --git a/tdms/matlab/file_parser/is_white_space.m b/tdms/tests/system/data/input_generation/matlab/file_parser/is_white_space.m similarity index 100% rename from tdms/matlab/file_parser/is_white_space.m rename to tdms/tests/system/data/input_generation/matlab/file_parser/is_white_space.m diff --git a/tdms/matlab/file_parser/read_material_data.m b/tdms/tests/system/data/input_generation/matlab/file_parser/read_material_data.m similarity index 100% rename from tdms/matlab/file_parser/read_material_data.m rename to tdms/tests/system/data/input_generation/matlab/file_parser/read_material_data.m diff --git a/tdms/matlab/focstratfield_general_pol_2d.m b/tdms/tests/system/data/input_generation/matlab/focstratfield_general_pol_2d.m similarity index 100% rename from tdms/matlab/focstratfield_general_pol_2d.m rename to tdms/tests/system/data/input_generation/matlab/focstratfield_general_pol_2d.m diff --git a/tdms/matlab/gauss_legendre.m b/tdms/tests/system/data/input_generation/matlab/gauss_legendre.m similarity index 100% rename from tdms/matlab/gauss_legendre.m rename to tdms/tests/system/data/input_generation/matlab/gauss_legendre.m diff --git a/tdms/matlab/gauss_pol.m b/tdms/tests/system/data/input_generation/matlab/gauss_pol.m similarity index 100% rename from tdms/matlab/gauss_pol.m rename to tdms/tests/system/data/input_generation/matlab/gauss_pol.m diff --git a/tdms/matlab/hfield_focused_equiv.m b/tdms/tests/system/data/input_generation/matlab/hfield_focused_equiv.m similarity index 100% rename from tdms/matlab/hfield_focused_equiv.m rename to tdms/tests/system/data/input_generation/matlab/hfield_focused_equiv.m diff --git a/tdms/matlab/import_constants.m b/tdms/tests/system/data/input_generation/matlab/import_constants.m similarity index 100% rename from tdms/matlab/import_constants.m rename to tdms/tests/system/data/input_generation/matlab/import_constants.m diff --git a/tdms/matlab/initialisesplitgrid.m b/tdms/tests/system/data/input_generation/matlab/initialisesplitgrid.m similarity index 100% rename from tdms/matlab/initialisesplitgrid.m rename to tdms/tests/system/data/input_generation/matlab/initialisesplitgrid.m diff --git a/tdms/matlab/initialiseupdateterms.m b/tdms/tests/system/data/input_generation/matlab/initialiseupdateterms.m similarity index 100% rename from tdms/matlab/initialiseupdateterms.m rename to tdms/tests/system/data/input_generation/matlab/initialiseupdateterms.m diff --git a/tdms/matlab/iteratefdtd_matrix.m b/tdms/tests/system/data/input_generation/matlab/iteratefdtd_matrix.m similarity index 100% rename from tdms/matlab/iteratefdtd_matrix.m rename to tdms/tests/system/data/input_generation/matlab/iteratefdtd_matrix.m diff --git a/tdms/matlab/minsteps_fdtd.m b/tdms/tests/system/data/input_generation/matlab/minsteps_fdtd.m similarity index 100% rename from tdms/matlab/minsteps_fdtd.m rename to tdms/tests/system/data/input_generation/matlab/minsteps_fdtd.m diff --git a/tdms/matlab/minsteps_pstd.m b/tdms/tests/system/data/input_generation/matlab/minsteps_pstd.m similarity index 100% rename from tdms/matlab/minsteps_pstd.m rename to tdms/tests/system/data/input_generation/matlab/minsteps_pstd.m diff --git a/tdms/matlab/multi_layer.m b/tdms/tests/system/data/input_generation/matlab/multi_layer.m similarity index 100% rename from tdms/matlab/multi_layer.m rename to tdms/tests/system/data/input_generation/matlab/multi_layer.m diff --git a/tdms/matlab/updateupdateterms.m b/tdms/tests/system/data/input_generation/matlab/updateupdateterms.m similarity index 100% rename from tdms/matlab/updateupdateterms.m rename to tdms/tests/system/data/input_generation/matlab/updateupdateterms.m diff --git a/tdms/matlab/yeeposition.m b/tdms/tests/system/data/input_generation/matlab/yeeposition.m similarity index 100% rename from tdms/matlab/yeeposition.m rename to tdms/tests/system/data/input_generation/matlab/yeeposition.m diff --git a/tdms/tests/system/regenerate_all.py b/tdms/tests/system/regenerate_all.py new file mode 100644 index 000000000..099f80fbc --- /dev/null +++ b/tdms/tests/system/regenerate_all.py @@ -0,0 +1,36 @@ +import os +import sys +from glob import glob +from pathlib import Path + +from data.input_generation.generate_test_input import ( + generate_test_input, + start_MatlabEngine_with_extra_paths, +) + +LOCATION_OF_THIS_FILE = os.path.dirname(os.path.abspath(__file__)) +TESTS_TO_REGEN = glob(LOCATION_OF_THIS_FILE + "/data/input_generation/*.yaml") +N_TESTS_TO_REGEN = len(TESTS_TO_REGEN) + + +def regenerate_all() -> None: + """Regenerate the input data for all tests corresponding to the config files present in this directory.""" + print(f"Found {N_TESTS_TO_REGEN} config files") + # Use a single MATLAB session to avoid startup and shutdown + engine = start_MatlabEngine_with_extra_paths() + + for i, test_case in enumerate(TESTS_TO_REGEN): + # Fetch config file + config_file_loc = Path(test_case) + print(f"Regenerating ({i+1}/{N_TESTS_TO_REGEN}) {config_file_loc}") + # Regenerate input data + generate_test_input(config_file_loc, engine) + + # Quit the engine we started + engine.quit() + return + + +if __name__ == "__main__": + regenerate_all() + sys.exit(0) diff --git a/tdms/tests/system/run_system_test.py b/tdms/tests/system/run_system_test.py new file mode 100644 index 000000000..d3ae0ccb6 --- /dev/null +++ b/tdms/tests/system/run_system_test.py @@ -0,0 +1,149 @@ +import os +from dataclasses import dataclass, field +from pathlib import Path +from zipfile import ZipFile + +import yaml +from pytest_check import check +from utils import HDF5File, Result, run_tdms + +LOCATION_OF_THIS_FILE = Path(os.path.abspath(os.path.dirname(__file__))) +ZIP_DIR = LOCATION_OF_THIS_FILE / "data" + + +@dataclass +class TDMSRun: + """One run, or execution, of the TDMS executable. That is, one call to + + tdms [OPTIONS] [input_file] [gridfile] [output_file] + + that is required as part of a single system test. + The execute() command executes the above call to TDMS + """ + + # ID of this run for diagnostics/debug + id: str + + # The .mat input to the tdms executable + input_file: Path + # Output .mat file to write to + output_file: Path + # Gridfile .mat input, if provided + gridfile: Path | None + # Additional command-line flags + flags: list[str] + # Command in list form to be fed into utils.run_tdms + command: list[str] = field(init=False) + + # The reference data file (relative to the top-level of the .zip archive for this test) + ref_data: Path + + def __post_init__(self): + """Setup the command that will be passed to utils.run_tdms.""" + self.command = self.flags + [str(self.input_file)] + # Gridfile comes next if it was provided + if self.gridfile: + self.command.append(str(self.gridfile)) + # Output file is final input on command line + self.command.append(str(self.output_file)) + + def __str__(self) -> str: + return f"{' '.join(['tdms'] + self.command)}" + + def execute(self) -> Result: + """Call tdms with the commands specified in this instance""" + print("Now running:\n\t", self) + result = run_tdms(*self.command) + return result + + +def run_system_test(config_filepath: Path | str) -> dict[str, bool]: + """Perform the system test detailed in the config file passed. + + :param config_filepath: Path to a .yaml file containing the information about the system test to run. + :returns: Dictionary indexed by the run_ids. Values are boolean; True if that run passed, and False if the run failed. + """ + with open(config_filepath, "r") as file: + test_information = yaml.safe_load(file) + + # Track the test's id (arc_{test_id}) and the information about the runs it contains + test_id = test_information["test_id"] + run_information = test_information["tests"] + + # Infer the location of the input data and .zip archive + input_data_folder = ( + LOCATION_OF_THIS_FILE / "data" / "input_generation" / f"arc_{test_id}" + ) + ref_data_zip_foler = ZipFile(ZIP_DIR / f"arc_{test_id}.zip", "r") + + # Setup the runs that are part of this test, using the test_dict + tdms_runs: list[TDMSRun] = [] + # These are reference files that have been pulled out of .zip archives and will need cleaning up afterwards + # set() ensures that duplicate filenames will not be added if already present + extracted_ref_files = set() + for run_name, run_info in run_information.items(): + # Fetch the input file + input_file = input_data_folder / run_info["input_file"] + + # Create a suitable name for the output of the run, and recycle the input_data_folder for the location of the output + output_file = input_data_folder / f"output_{test_id}_{run_name}.mat" + + # Fetch the filename of the reference data that this run compares it's output to + ref_output = run_info["reference"] + extracted_ref_files.add(ref_output) + + # Fetch the gridfile, if it exists + if "gridfile" in run_info.keys(): + gridfile = run_info["gridfile"] + else: + gridfile = None + + # Determine if there are any flags that need to be passed to the tdms run + run_flags = [] + if "cubic_interpolation" in run_info.keys(): + if run_info["cubic_interpolation"]: + run_flags += ["-c"] + if "fdtd_solver" in run_info.keys(): + if run_info["fdtd_solver"]: + run_flags += ["--finite-difference"] + + # Create the run and store it with the reference data file that it needs + tdms_runs.append( + TDMSRun(run_name, input_file, output_file, gridfile, run_flags, ref_output) + ) + + # Extract all unique reference data files from the .zip archive + dir_to_extract_refs_to = str(input_data_folder / "reference_data") + for ref_data in extracted_ref_files: + # extract() cannot take Path arguments + ref_data_zip_foler.extract(ref_data, dir_to_extract_refs_to) + + # Perform each run and log the success, assume runs fail by defualt + run_success = dict() + + # Perform each run, and compare the generated output to the reference output + for run in tdms_runs: + # Run the TDMS call + run.execute() + + # Compare the output to the reference data + run_output = HDF5File(run.output_file) + ref_output = HDF5File(f"{dir_to_extract_refs_to}/{run.ref_data}") + + # Use check, so that all runs are performed before we report test failure + # This also ensures that we cleanup the reference data REGARDLESS of test success + with check: + test_passed, comparison_information = run_output.matches( + ref_output, return_message=True + ) + assert test_passed, f"In {test_id} -> {run.id}: {comparison_information}" + # Append whether or not the test passed + run_success[run.id] = test_passed + + # Cleanup the .mat files we had to copy across + for mat_file in extracted_ref_files: + os.remove(f"{dir_to_extract_refs_to}/{mat_file}") + # then remove the (what should be empty) directory + os.rmdir(dir_to_extract_refs_to) + + return run_success diff --git a/tdms/tests/system/test_regen.py b/tdms/tests/system/test_regen.py new file mode 100644 index 000000000..ac9284517 --- /dev/null +++ b/tdms/tests/system/test_regen.py @@ -0,0 +1,107 @@ +import os +from glob import glob +from pathlib import Path +from warnings import warn + +import pytest +import yaml +from data.input_generation.generate_test_input import ( + generate_test_input as regenerate_test, +) +from run_system_test import run_system_test +from utils import download_data + +# Location of this file, which is where the tests are running from +LOCATION_OF_THIS_FILE = Path(os.path.abspath(os.path.dirname(__file__))) +# This will determine whether or not we want to retain the regenerated input .mat files (if say, we are planning a new Zenodo upload). Recommended FALSE on CLI, TRUE locally if you're doing the update +PRESERVE_FLAG = True + +# Path to the directory containing files we need to read from +path_to_input_generation = Path(LOCATION_OF_THIS_FILE, "data", "input_generation") + +# Dataset is stored at https://zenodo.org/record/7440616/ +# ccaegra@ucl.ac.uk (William Graham, @willGraham01) has access. +ZENODO_URL = "https://zenodo.org/record/7440616/files" +# directory in which to store the downloaded zip files +ZIP_DESTINATION = Path(os.path.dirname(os.path.abspath(__file__)), "data") + +# Find all config_XX.yaml files and hence infer all test cases that need to be run +TEST_IDS = glob(str(path_to_input_generation / "config_*.yaml")) +for i, id in enumerate(TEST_IDS): + # remove everything bar the ID from what we found + id = id.removeprefix(str(path_to_input_generation / "config_")) + id = id.removesuffix(".yaml") + TEST_IDS[i] = id +# The number of tests will be useful to know later +N_TESTS = len(TEST_IDS) + + +def workflow(test_id: str, preserve_inputs: bool = PRESERVE_FLAG) -> None: + """Performs all tdms runs contained in the system test arc_{test_id}. Assumes that reference data is present in the appropriate .zip folder within the /data directory. + + The workflow steps are as follows: + 1. Regenerate the input data for arc_{test_id} + 2. Perform each run of TDMS as specified by arc_{test_id}, comparing the output to the previously obtained references + 3. Perform clean-up on generated outputs and inputs, so they are not saved to the cache and accidentally re-used + """ + # Fetch config file location + config_file_path = path_to_input_generation / f"config_{test_id}.yaml" + # Fetch test run information, warn if there is an ID inconsistency + with open(config_file_path, "r") as opened_config_file: + # Read data into dictionary, and take the "tests" value + INFO = yaml.safe_load(opened_config_file) + yaml_test_id = INFO["test_id"] + if yaml_test_id != test_id: + warn( + f"{str(config_file_path)} implicit ID does not match config file description (found {test_id})" + ) + + # Regenerate the input data for arc_{test_id}, error (hence fail) if not successful + regenerate_test(config_file_path) + + # Perform each run of TDMS as specified by the config file + run_success = run_system_test(config_file_path) + + # TEAR-DOWN: remove the regenerated inputs and outputs from the data/input_generation/arc_{test_id} folder. + # To be safe, we can just remove all .mat files from this directory the the subdirectories, since these should be the only system-test TDMS artefacts + if not PRESERVE_FLAG: + input_data_dump = str(path_to_input_generation / f"arc_{test_id}/*.mat") + mat_artefacts = glob(input_data_dump, recursive=True) + for mat_file in mat_artefacts: + os.remove(mat_file) + + # Although we should have check-ed whether each run was a pass/fail, we can also assert that all runs need to pass here to report failures + failed_run_names = [] + for run_id, passed in run_success.items(): + if not passed: + failed_run_names.append(run_id) + assert all( + run_success + ), f"arc_{test_id} : Some runs were unsuccessful ({len(run_success)-sum(run_success)}/{len(run_success)}) :\n {failed_run_names}" + return + + +# Run each system test that is currently saved in the repository, as identified through the config files +# "MATLAB doesn't run on GH runners. If you want to check that regenerating the input data still allows the tests to pass, run this locally and remove the skip mark." +@pytest.mark.parametrize("test_id", TEST_IDS) +def test_system(test_id) -> None: + """Runs the system test arc_{test_id}, including fetching missing reference data from Zenodo. + + Wraps the workflow() method, which actually does the bulk of the testing. + """ + print(f"\nRunning {test_id}", end=" | ", flush=True) + + # the reference OUTPUT data should be at this location + ZIP_PATH = ZIP_DESTINATION / f"arc_{test_id}.zip" + # download data if not present + if not ZIP_PATH.exists(): + url = ZENODO_URL + f"/arc_{test_id}.zip" + print(f"Downloading from {url}") + download_data(url, to=ZIP_PATH) + else: + print(f"Using cache in {ZIP_PATH}") + + # Run the test workflow, for this test + workflow(test_id) + # End of system test + return