[WIP] i.sentinel3.import: New addon to import Sentinel-3 data

[ninsbl]

The Sentinel-3 data has a really peculiar format that is not supported by GDAL.
This module implements import based on numpy and netCDF4. Although it should be working in general, thorough review of the import procedure (and the geometric acceptability) would be very much appreciated.
(Optional/additional) import as verctor points would be easy to implement if that would be an advantage...

[veroandreo]

I have only checked the manual so far

[veroandreo]

What does QGIS uses then to open the nc files within the S3 zip files? I tested with a S3SY2VG1 product. A double click just opens the ndvi layer in QGIS and r.import works just fine for that layer as well. Maybe, this statement is a bit too strong or only valid for S3 LST products...?

[ninsbl]

Do you mean the semi automated classification plugin? That one has a similar (but not the same) workaround for the S3-format...

[veroandreo]

No, plain QGIS opens the .nc mentioned file just fine, I do not use the semi automated classification plugin

[ninsbl]

OK, thanks for clarifying.

Yes, QGIS does open the .nc but - at least the QGIS 3.16 version I am running does not resolve the geometrical information correctly. The image is flipped and distorted....

The sentinel-3 format is pretty peculiar and one needs to combine the geometry and data nc files in order to get a reasonble geographical representation of the data....

[veroandreo]

NDVI products look just fine

LST is flipped

This is what I meant originally with my comment, some of the S3 products are read just fine (NDVI from the S3A_SY_2_VG1 above). Hence, it's not completely true that none of the S3 formats is supported by GDAL as stated.
I use GDAL 3.2.2 and QGIS 3.20

[ninsbl]

Thanks for clarifying and bearing with me. My focus has been on SLSTR. So, the different products seem to come in different formats / packaging also across processing levels...

So a "final" i.sentinel3.import module should support the formats for the different instruments

• Sentinel-3 OLCI
• Sentinel-3 SLSTR
• Sentinel-3 Synergy
• Sentinel-3 Altimetry

for level 1 and 2.

Not sure if I find the time to investigate and implement all of them (can be as simple as r.import/r.in.gdal/r.external).

In any case this should be made clear in the documentation, as you correctly point out...

[veroandreo]

Why "data"? Are the quotes necessary there? Also, maybe put some of the resulting printed info as example and not only the ... ;)

[veroandreo]

I pulled your branch and tested with:

python i.sentinel.download.py -l settings=/home/veroandreo/sentinel producttype=S3SL2LST sort=ingestiondate
python i.sentinel.download.py settings=/home/veroandreo/sentinel uuid=5d24b2c1-f7e1-45b4-ba4e-cd87b247e90e output=/home/veroandreo/
python i.sentinel3.import.py -p input=/home/veroandreo/ product=LST basename=S3_LST

and I thought it would import everything, but I get:

ERROR: 
       /home/veroandreo/S3A_SY_2_VG1____20210729T000000_20210729T235959_20210804T144835_SOUTH_AMERICA_____LN2_O_NT_002.zip
       does not contain a a container LST_in.nc with band LST_uncertainty,
       LST

However, the LST_in is there... what am I doing wrong??

Also, seems the user should set the region resolution before hand, right? I get some pretty course pixels in the bands that do get imported... Maybe a full example in the manual would be useful.

[ninsbl]

Thanks, @veroandreo for reviewing.

[ninsbl]

However, the LST_in is there... what am I doing wrong??

Nothing, a bug in the code. Thanks for pointing out.

Also, seems the user should set the region resolution before hand, right? I get some pretty course pixels in the bands that do get imported... Maybe a full example in the manual would be useful.

Yes, that is mentioned in the manual. However, the default could also be to use the "native" resolution and extent from the scene (Markus mentioend the irregularities in the raster alignment there so this would be a rough approximation) and an r-flag could limit import to the computational region...

[ninsbl]

and I thought it would import everything, but I get:

ERROR: 
       /home/veroandreo/S3A_SY_2_VG1____20210729T000000_20210729T235959_20210804T144835_SOUTH_AMERICA_____LN2_O_NT_002.zip
       does not contain a a container LST_in.nc with band LST_uncertainty,
       LST

However, the LST_in is there... what am I doing wrong??

There seems to be Synergy product in the input directory that does not get filtered out:
S3A_SY_2_VG1____20210729T000000_20210729T235959_20210804T144835_SOUTH_AMERICA_____LN2_O_NT_002
That does not have a LST_in.nc ...
I adjusted the default pattern, so for now only LST products are imported. However, I see that I need to make the module less product specific...

Also, seems the user should set the region resolution before hand, right? I get some pretty course pixels in the bands that do get imported... Maybe a full example in the manual would be useful.

Right, that is mentioned in the manual. But resolution and extent could be grabbed from the data and a temporary region set for import accordingly....

[echoix]

If we want to make it closer to include in the repo, we might need to target the new default branch, grass8.

[neteler]

If we want to make it closer to include in the repo, we might need to target the new default branch, grass8.

Using "Edit" next to the PR title would let us change the base to grass8.
It then warns:

Are you sure you want to change the base?

Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated. 

Would that be ok?

[ninsbl]

FYI: I am currently re-working the PR / module...

[echoix]

Nice! Mark the PR ready to review when you want us to take a new look on it ;)