Fixing field constraints

[leonardolaiolo]

No description provided.

[leonardolaiolo]

Hi @clairedavies
I'm doing a bit of maintenance on the water sampling database code - We realised some time ago that some fields marked as 'required' were not being validated as expected. You can find more background information on the issue here: #77

I made some changes to address these issues, but I encountered some errors along the way. I was able to fix some of the obvious ones, but I'd like to double-check with you to ensure that you're satisfied with the constraints implemented originally:

bgc_lfish_samples -> SAMPVOL_M3 is set are required true - some values are missing and will cause error in the next harvest
bgc_picoplankton -> PROCHLOROCOCCUS_FLAG, SYNECOCHOCCUS_FLAG and PICOEUKARYOTES_FLAG are set as required true - some are missing and will cause error in the next harvest
bgc_picoplankton_meta -> PROCHLOROCOCCUS_FLAG, SYNECOCHOCCUS_FLAG and PICOEUKARYOTES_FLAG are set as required true - some are missing and will cause error in the next harvest
bgc_trip -> ZOOPSAMPLEDEPTH_M and PHYTOSAMPLEDEPTH_M are set as required true - many values are missing and will cause error in the next harvest
cpr_samp -> SAMPLETYPE is set as required true - many values are missing and will cause error in the next harvest.

To resolve these issues, we need to decide which of these variables are always required and which are not. Then, we can update the code or the data at your end accordingly.

[clairedavies]

bgc_lfish_samples -> SAMPVOL_M3 is set are required true - some values are missing and will cause error in the next harvest
The samples are missing and there is no associated data with them. I have removed them so this can remain as required,

bgc_picoplankton -> PROCHLOROCOCCUS_FLAG, SYNECOCHOCCUS_FLAG and PICOEUKARYOTES_FLAG are set as required true - some are missing and will cause error in the next harvest
There are samples from SARDI that only have birus and bacteria data, therefore the columns mentioned above will be null. These should not be required fields.

bgc_picoplankton_meta -> PROCHLOROCOCCUS_FLAG, SYNECOCHOCCUS_FLAG and PICOEUKARYOTES_FLAG are set as required true - some are missing and will cause error in the next harvest
There are samples from SARDI that only have birus and bacteria data, therefore the columns mentioned above will be null. These should not be required fields.

bgc_trip -> ZOOPSAMPLEDEPTH_M and PHYTOSAMPLEDEPTH_M are set as required true - many values are missing and will cause error in the next harvest
There are many samples which don't have zoop or phyto samples, i.e. coastal stations, so these should not be required fields.

cpr_samp -> SAMPLETYPE is set as required true - many values are missing and will cause error in the next harvest.
This is not a required field, as these are the unprocessed samples that only have PCI values.

[leonardolaiolo]

@clairedavies thanks for your feedback - I updated those and now some other errors came up, sorry it was a long redundant list and I missed some..
bgc_lfish_samples -> SAMPVOL_M3 set as required true - Still come as an error, maybe we need to wait the CSIRO Geoserver to refresh?
bgc_picoplankton -> SAMPLEDATELOCAL set as required true?
bgc_picoplankton_meta -> SAMPLEDATELOCAL set as required true?
bgc_trip -> SAMPLEDATEUTC set as required true?

[clairedavies]

bgc_lfish_samples -> SAMPVOL_M3 set as required true - Still come as an error, maybe we need to wait the CSIRO Geoserver to refresh?
Refreshed geoserver so try again

bgc_picoplankton -> SAMPLEDATELOCAL set as required true?
bgc_picoplankton_meta -> SAMPLEDATELOCAL set as required true?
bgc_trip -> SAMPLEDATEUTC set as required true?
These 3 should all be set required as true, I have filled in all the blanks from the coastal data and refreshed geoserver, so should work - fingers crossed.

[mhidas]

👍

[mhidas]

Thanks for working through this and fixing up all the schemas and data!