Below are simple checks which can be implemented to ensure that the file content is as expected. It is assumed that all fields in all expected files should remain the same if there is not any change or issue in data production (dispatching etc).
An example how to create a reference field list from given files, GRIBs in this case, and compare it to an actual field list follows.
The same approach can be used for any type of files but an appropriate tool for field list creation must exist or be coded.
An example of creation of the reference or actual field list using python script get_field_list.py (ecCodes python api is prerequisite).
#!/bin/ksh set -ex # $reflist is a link to the reference field list # $DTS_ALLOW_NEW_REFERENCE is "true" if a new reference is required/expected # get actual field list for comparison to the reference python $DTS_BIN/get_field_list.py -c lw.grib2 > list.tmp awk '{print $1}' list.tmp | sort > list # check if anything changed diff --changed-group-format='%%<' --unchanged-group-format='' list $reflist > diff.added.tmp || true diff --changed-group-format='%%>' --unchanged-group-format='' list $reflist > diff.removed.tmp || true cat diff.added.tmp | sort > diff.added cat diff.removed.tmp | sort > diff.removed if [[ -s diff.added || -s diff.removed ]] ; then # some differences found.. if [[ "${DTS_ALLOW_NEW_REFERENCE}" = "true" ]] ; then cp -f list $reflist echo "A new partial reference field list created! ($reflist)" else echo "Differences comparing to the actual reference field list found!" exit -1 fi else smslabel info "The actual reference is valid ($reflist)" fi |
lw_sabm_000000001800_xxxx_fc_sl_level0000_step0_10u lw_sabm_000000001800_xxxx_fc_sl_level0000_step0_10v lw_sabm_000000001800_xxxx_fc_sl_level0000_step0_pp1d lw_sabm_000000001800_xxxx_fc_sl_level0000_step0_swh lw_sabm_000000001800_xxxx_fc_sl_level0000_step10_10u lw_sabm_000000001800_xxxx_fc_sl_level0000_step10_10v lw_sabm_000000001800_xxxx_fc_sl_level0000_step10_pp1d lw_sabm_000000001800_xxxx_fc_sl_level0000_step10_swh lw_sabm_000000001800_xxxx_fc_sl_level0000_step11_10u ... ... lw_sabm_000000001800_xxxx_fc_sl_level0000_step96_pp1d lw_sabm_000000001800_xxxx_fc_sl_level0000_step96_swh lw_sabm_000000001800_xxxx_fc_sl_level0000_step9_10u lw_sabm_000000001800_xxxx_fc_sl_level0000_step9_10v lw_sabm_000000001800_xxxx_fc_sl_level0000_step9_pp1d lw_sabm_000000001800_xxxx_fc_sl_level0000_step9_swh |