This page describes the Brazilian Wigos test case. The goal was to encode the input data into a BUFR file including the Wigos information.
The input data was kindly provided by Jose Mauro Rezende from INMET. This data consisted in several
records in csv format.
| A806 | 2018 | 9 | 18 | 0 | 12.8 | 20 | -48.6 | -27.6 | 4.87 | 19 | 19 | 18.9 | 91 | 91 | 91 | 17.5 | 17.5 | 17.5 | 1011.1 | 1011.1 | 1011 | 0.6 | 20 | 2.9 | -0.255 | 0 | / | //// | ///// | ///// | = |
| A765 | 2018 | 9 | 18 | 0 | 12.6 | 22 | -46.14 | -23.84 | 5 | 20.2 | 20.5 | 20.2 | 93 | 94 | 91 | 19.1 | 19.2 | 18.9 | 1011.7 | 1011.8 | 1011.6 | 0.7 | 324 | 1.3 | -1.057 | 0 | / | //// | ///// | ///// | = |
| A907 | 2018 | 9 | 18 | 0 | 12.5 | 28 | -54.6 | -16.5 | 289.8 | 26.2 | 26.2 | 25.6 | 77 | 78 | 74 | 21.8 | 21.8 | 21.1 | 979.1 | 979.1 | 977.4 | 1.4 | 296 | 2.4 | -3.378 | 0 | / | //// | ///// | ///// | = |
| A305 | 2018 | 9 | 18 | 0 | 12.3 | 28 | -38.5 | -3.1 | 29.55 | 26.2 | 26.3 | 26.1 | 69 | 72 | 69 | 20.1 | 20.7 | 20.1 | 1009.6 | 1009.6 | 1008.8 | 2.8 | 105 | 7.7 | -3.54 | 0 | / | //// | ///// | ///// | = |
| A450 | 2018 | 9 | 18 | 0 | 12.5 | 28 | -38.3 | -10.08 | 261 | 24.9 | 25.7 | 24.9 | 67 | 67 | 62 | 18.3 | 18.3 | 17.9 | 985 | 985 | 984.1 | 4.1 | 111 | 10.7 | -3.143 | 0 | / | //// | ///// | ///// | = |
| A025 | 2018 | 9 | 18 | 0 | 12.3 | 21 | -50.96 | -17.78 | 780.1 | 19.2 | 19.5 | 19.2 | 87 | 88 | 85 | 17 | 17.3 | 16.9 | 927.5 | 927.5 | 926.7 | 2.6 | 70 | 3.5 | -3.498 | 0 | / | //// | ///// | ///// | = |
Jose Mauro also provided the following excel file containing the metadata giving the meaning of each column of the previous file.
| DATE | HOUR | TENS | TEMP | AIR TEMP. | RELATIVE UMID | DEW POINT | PRESSURE | WINDS | CLOUD COVER | |||||||||||||||||||
| STATION | YEAR | MON | DAY | OBS. | BAT | CPU | INST. | MAX | MIN | INST | MAX | MIN | INST | MAX | MIN | INST | MAX | MIN | Speed | Dir | Gust | RAD | PREC | TOT | CODE | BASE | VISIB | |
| UTC | V | ºC | ºC | ºC | ºC | % | % | % | ºC | ºC | ºC | hPa | hPa | hPa | m/s | º | m/s | kJ/m2 | mm | |||||||||
| A001 | 2000 | 7 | 1 | 0 | 12.5 | 22 | 19.6 | 20.7 | 19.5 | 45 | 45 | 40 | 7.2 | 7.5 | 6.6 | 889.1 | 889.1 | 888.8 | 1.6 | 113 | 4.0 | -4 | 0.0 | / | //// | ///// | ///// | = |
| A001 | 2000 | 7 | 1 | 1 | 12.5 | 21 | 20.1 | 20.2 | 19.7 | 39 | 47 | 39 | 5.8 | 8.1 | 5.8 | 889.4 | 889.4 | 889.1 | 2.2 | 71 | 4.5 | -4 | 0.0 | / | //// | ///// | ///// | = |
| A001 | 2000 | 7 | 1 | 2 | 12.5 | 21 | 19.6 | 20.2 | 19.6 | 36 | 39 | 36 | 4.3 | 5.8 | 4.3 | 889.3 | 889.4 | 889.3 | 2.3 | 75 | 4.0 | -4 | 0.0 | / | //// | ///// | ///// | = |
| A001 | 2000 | 7 | 1 | 3 | 12.5 | 21 | 18.3 | 19.7 | 18.3 | 43 | 43 | 36 | 5.5 | 5.5 | 4.2 | 889.0 | 889.3 | 889.0 | 1.5 | 78 | 4.1 | -4 | 0.0 | / | //// | ///// | ///// | = |
| CODIGOS | SIM | I101 | I608 | I612 | I105 | I617 | I618 | I103 | I613 | I614 | I106 | I615 | I616 | I111 | I113 | I608 | I133 | I175 | I118 | I110 | ||||||||
| ESTAÇÃO = Mnnn | M=Organização | xnn = sequencial | Primeiro digito x=Distrito, nn=sequencial de instalação em cada Distrito | |||||||||||||||||||||||||
| YEAR MONTH DAY HOUR | (HOUR UTC) | |||||||||||||||||||||||||||
| BATERY VOLTAGE | ||||||||||||||||||||||||||||
| CPU TEMPERATURE | ||||||||||||||||||||||||||||
| AIR TEMPERATURE | INSTANTANEOUS, MAXIMA and MINIMA | |||||||||||||||||||||||||||
| RELATIVE HUMIDITY | INSTANTANEOUS, MAXIMA and MINIMA | |||||||||||||||||||||||||||
| DEW POINT | INSTANTANEOUS, MAXIMA and MINIMA | |||||||||||||||||||||||||||
| PRESSURE | INSTANTANEOUS, MAXIMA and MINIMA | |||||||||||||||||||||||||||
| WINDS | DIREÇÃO, VELOCIDADE E RAJADA | |||||||||||||||||||||||||||
| SOLAR RADIATION | ||||||||||||||||||||||||||||
| PRECIPITATION | ||||||||||||||||||||||||||||
| CLOUD COVER | TOTAL, CODE and CLOUD BASE (Manually inserted on the station via keyboard) | |||||||||||||||||||||||||||
| VISIBILITY | ||||||||||||||||||||||||||||
To achieve this goal, the SYNOP template 307091 was used from the current messages like ISAI01-SBBR-041400-RRA.bfr and added the sequence 301150 that contains the
Wigos information.
The script code follows
test_wigos4.py
#!/usr/bin/env python
'''
# Copyright 2005-2018 ECMWF.
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction
This is a test program to encode Wigos Synop
requires
1) ecCodes version 2.8 or above (available at https://confluence.ecmwf.int/display/ECC/Releases)
2) python2.7
To run the program
./test_wigos4.py -a Ascii_csv_file -b output_Bufr
Uses BUFR version 4 template 301150 307091
Author : Roberto Ribas Garcia ECMWF 27 Sep 2018
'''
from eccodes import (codes_set, codes_set_array, codes_bufr_new_from_samples,codes_write,codes_get_api_version,
codes_release, CodesInternalError,CODES_MISSING_DOUBLE,CODES_MISSING_LONG)
import pandas as pd
from datetime import datetime
import argparse
def read_cmdline():
'''
reads the command line to get the input ascii filename and the output bufr file
usage
prog -a Ascii_input_file -b Bufr_output_file
'''
p=argparse.ArgumentParser()
p.add_argument("-a","--ascii",help=" input Ascii filename")
p.add_argument("-b","--bufr", help="output Bufr filename")
args=p.parse_args()
return args
def read_ascii(inputFilename):
'''
function to read the Ascii data into a pandas dataframe,
args:
inputFilename : full path of the Ascii file for example /tmp/data/rema_20180918.txt
uses white spaces as column delimiters
index_col=False avoids using the first column (Station) as index
names is the list of names from the excel it can be changed but this affects the dataframe
'''
df=pd.read_csv(inputFilename,header=None,index_col=False,delim_whitespace=True,names=["station","year","month","day",
"ObsHour","TensBat","TempCpu","lon","lat","hp","airTinst","airTmax",
"airTmin","relHinst","relHmax","relHmin","dewPInst","dewPmax","dewPmin" ,
"PresInst","PresMax","PresMin","WindSpeed","WindDir","WindGust",
"Rad","Precip","CloudCoverTot","CloudCODE","CloudBase","Visib"])
print df.head()
return df
def message_encoding(FullInputFileName,fout):
'''
Message encoding function
FullInputFilename : full path of the Ascii file for example /tmp/data/rema_20180918.txt
fout : file Object to write the output bufr file( obtained by a call to open )
Requires ecCodes and the BUFR4_local template on
ECCODES_PATH/share/eccodes/samples
'''
TEMPLATE='BUFR4_local'
# reads the Ascii file into a pandas Dataframe
dfFull=read_ascii(FullInputFileName)
# loops over the rows of the dataFrame dfFull
for _,row in dfFull.iterrows():
bid=codes_bufr_new_from_samples(TEMPLATE)
try:
bufr_encode_new(bid,row)
codes_write(bid,fout)
except CodesInternalError as ec:
print ec
codes_release(bid)
def bufr_encode_new(ibufr,row):
'''
encodes the new SYNO 307091 adding the 1125, 1126, 1127, 1128 wigos keys before.
'''
ivalues = (
1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1 ,)
codes_set_array(ibufr, 'inputShortDelayedDescriptorReplicationFactor', ivalues)
codes_set(ibufr, 'edition', 4)
codes_set(ibufr, 'masterTableNumber', 0)
codes_set(ibufr, 'bufrHeaderCentre', 98)
codes_set(ibufr, 'bufrHeaderSubCentre', 0)
codes_set(ibufr, 'updateSequenceNumber', 0)
codes_set(ibufr, 'dataCategory', 0)
codes_set(ibufr, 'internationalDataSubCategory', 255)
codes_set(ibufr, 'dataSubCategory', 170)
codes_set(ibufr, 'masterTablesVersionNumber', 29)
codes_set(ibufr, 'localTablesVersionNumber', 0)
# set the YMD
codes_set(ibufr, 'typicalYear', row["year"])
codes_set(ibufr, 'typicalMonth', row["month"])
codes_set(ibufr, 'typicalDay', row["day"])
codes_set(ibufr, 'typicalHour', row["ObsHour"])
codes_set(ibufr, 'typicalMinute', 0)
codes_set(ibufr, 'typicalSecond', 0)
# Encodes the Section 2 of the BUFR used internally at ECMWF (start here)
codes_set(ibufr, 'rdbType', 1)
codes_set(ibufr, 'oldSubtype', 176)
codes_set(ibufr, 'localYear', row["year"])
codes_set(ibufr, 'localMonth', row["month"])
codes_set(ibufr, 'localDay', row["day"])
codes_set(ibufr, 'localHour', row["ObsHour"])
codes_set(ibufr, 'localMinute', 0)
codes_set(ibufr, 'localSecond', 0)
procTime=datetime.now()
codes_set(ibufr, 'rdbtimeDay', procTime.day)
codes_set(ibufr, 'rdbtimeHour', procTime.hour)
codes_set(ibufr, 'rdbtimeMinute', procTime.minute)
codes_set(ibufr, 'rdbtimeSecond', procTime.second)
codes_set(ibufr, 'rectimeDay', procTime.day )
codes_set(ibufr, 'rectimeHour', procTime.hour)
codes_set(ibufr, 'rectimeMinute', procTime.minute)
codes_set(ibufr, 'rectimeSecond', procTime.second)
codes_set(ibufr, 'correction1', 0)
codes_set(ibufr, 'correction1Part', 0)
codes_set(ibufr, 'correction2', 0)
codes_set(ibufr, 'correction2Part', 0)
codes_set(ibufr, 'correction3', 0)
codes_set(ibufr, 'correction3Part', 0)
codes_set(ibufr, 'correction4', 0)
codes_set(ibufr, 'correction4Part', 0)
codes_set(ibufr, 'qualityControl', 70)
codes_set(ibufr, 'newSubtype', 0)
codes_set(ibufr, 'numberOfSubsets', 1)
lat=row["lat"]
lon=row["lon"]
codes_set(ibufr, 'localLatitude', lat)
codes_set(ibufr, 'localLongitude', lon)
#### End of encoding local section 2
codes_set(ibufr, 'observedData', 1)
codes_set(ibufr, 'compressedData', 0)
ivalues=(301150,307091)
codes_set_array(ibufr, 'unexpandedDescriptors', ivalues)
codes_set(ibufr, 'wigosIdentifierSeries',0 )
codes_set(ibufr, 'wigosIssuerOfIdentifier', 76)
codes_set(ibufr, 'wigosIssueNumber', 0)
codes_set(ibufr, 'wigosLocalIdentifierCharacter','0760999999999')
codes_set(ibufr, 'stateIdentifier', CODES_MISSING_LONG)
codes_set(ibufr, 'nationalStationNumber', CODES_MISSING_LONG)
codes_set(ibufr, 'blockNumber', CODES_MISSING_LONG)
codes_set(ibufr, 'stationNumber', CODES_MISSING_LONG)
codes_set(ibufr, 'stationOrSiteName',row["station"])
codes_set(ibufr, 'stationType', CODES_MISSING_LONG)
codes_set(ibufr, 'year', row["year"])
codes_set(ibufr, 'month', row["month"])
codes_set(ibufr, 'day', row["day"])
codes_set(ibufr, 'hour', row["ObsHour"])
codes_set(ibufr, 'minute', 0)
codes_set(ibufr, 'latitude', lat)
codes_set(ibufr, 'longitude', lon)
height=row["hp"]
codes_set(ibufr, 'heightOfStationGroundAboveMeanSeaLevel', height)
codes_set(ibufr, 'heightOfBarometerAboveMeanSeaLevel', 1.5)
codes_set(ibufr, 'surfaceQualifierForTemperatureData', CODES_MISSING_LONG)
codes_set(ibufr, 'mainPresentWeatherDetectingSystem', CODES_MISSING_LONG)
codes_set(ibufr, 'supplementaryPresentWeatherSensor', CODES_MISSING_LONG)
codes_set(ibufr, 'visibilityMeasurementSystem', CODES_MISSING_LONG)
codes_set(ibufr, 'cloudDetectionSystem', CODES_MISSING_LONG)
codes_set(ibufr, 'lightningDetectionSensorType', CODES_MISSING_LONG)
codes_set(ibufr, 'skyConditionAlgorithmType', CODES_MISSING_LONG)
codes_set(ibufr, 'capabilityToDetectPrecipitationPhenomena', CODES_MISSING_LONG)
codes_set(ibufr, 'capabilityToDetectOtherWeatherPhenomena', CODES_MISSING_LONG)
codes_set(ibufr, 'capabilityToDetectObscuration', CODES_MISSING_LONG)
codes_set(ibufr, 'capabilityToDiscriminateLightningStrikes', CODES_MISSING_LONG)
codes_set(ibufr, '#1#nonCoordinatePressure', CODES_MISSING_DOUBLE)
pressure=row["PresInst"]*100
codes_set(ibufr, 'pressureReducedToMeanSeaLevel', pressure)
codes_set(ibufr, '3HourPressureChange', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'characteristicOfPressureTendency', CODES_MISSING_LONG)
codes_set(ibufr, 'pressure', pressure)
codes_set(ibufr, 'nonCoordinateGeopotentialHeight', CODES_MISSING_LONG)
codes_set(ibufr, '#1#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#1#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
temperature=row["airTinst"]+273.15
codes_set(ibufr, '#1#airTemperature', temperature)
dewPoint=row["dewPInst"]+273.15
codes_set(ibufr, 'dewpointTemperature', dewPoint)
codes_set(ibufr, '#1#relativeHumidity', row["relHinst"])
codes_set(ibufr, '#1#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#1#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#4#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#4#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#5#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#5#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#6#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#1#attributeOfFollowingValue', CODES_MISSING_LONG)
if row["Visib"]=="/////":
visib=CODES_MISSING_DOUBLE
else:
visib=row["Visib"]
codes_set(ibufr, 'horizontalVisibility', visib)
codes_set(ibufr, '#3#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'iceDepositThickness', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'rateOfIceAccretionEstimated', CODES_MISSING_LONG)
codes_set(ibufr, 'methodOfWaterTemperatureAndOrOrSalinityMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, 'oceanographicWaterTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'wavesDirection', CODES_MISSING_LONG)
codes_set(ibufr, 'periodOfWaves', CODES_MISSING_LONG)
codes_set(ibufr, 'heightOfWaves', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'methodOfStateOfGroundMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, 'stateOfGround', CODES_MISSING_LONG)
codes_set(ibufr, 'methodOfSnowDepthMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, 'totalSnowDepth', CODES_MISSING_DOUBLE)
if row["CloudCoverTot"]=="/":
CloudCover=CODES_MISSING_LONG
else:
CloudCover=row["CloudCoverTot"]
codes_set(ibufr, 'cloudCoverTotal', CloudCover)
codes_set(ibufr, '#1#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
codes_set(ibufr, '#1#cloudAmount', CODES_MISSING_LONG)
codes_set(ibufr, '#1#cloudType', CODES_MISSING_LONG)
codes_set(ibufr, '#2#attributeOfFollowingValue', CODES_MISSING_LONG)
codes_set(ibufr, '#1#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
codes_set(ibufr, '#2#cloudAmount', CODES_MISSING_LONG)
codes_set(ibufr, '#2#cloudType', CODES_MISSING_LONG)
codes_set(ibufr, '#3#attributeOfFollowingValue', CODES_MISSING_LONG)
codes_set(ibufr, '#2#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
codes_set(ibufr, '#3#cloudAmount', CODES_MISSING_LONG)
codes_set(ibufr, '#3#cloudType', CODES_MISSING_LONG)
codes_set(ibufr, '#4#attributeOfFollowingValue', CODES_MISSING_LONG)
codes_set(ibufr, '#3#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#4#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
codes_set(ibufr, '#4#cloudAmount', CODES_MISSING_LONG)
codes_set(ibufr, '#4#cloudType', CODES_MISSING_LONG)
codes_set(ibufr, '#5#attributeOfFollowingValue', CODES_MISSING_LONG)
codes_set(ibufr, '#4#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'presentWeather', CODES_MISSING_LONG)
codes_set(ibufr, '#1#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'pastWeather1', CODES_MISSING_LONG)
codes_set(ibufr, 'pastWeather2', CODES_MISSING_LONG)
codes_set(ibufr, '#1#timeSignificance', CODES_MISSING_LONG)
codes_set(ibufr, '#2#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'precipitationIntensityHighAccuracy', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'sizeOfPrecipitatingElement', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#timeSignificance', CODES_MISSING_LONG)
codes_set(ibufr, '#3#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'precipitationType', CODES_MISSING_LONG)
codes_set(ibufr, 'characterOfPrecipitation', CODES_MISSING_LONG)
codes_set(ibufr, 'durationOfPrecipitation', CODES_MISSING_LONG)
codes_set(ibufr, 'otherWeatherPhenomena', CODES_MISSING_LONG)
codes_set(ibufr, 'intensityOfPhenomena', CODES_MISSING_LONG)
codes_set(ibufr, 'obscuration', CODES_MISSING_LONG)
codes_set(ibufr, 'characterOfObscuration', CODES_MISSING_LONG)
codes_set(ibufr, '#4#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#4#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#timeSignificance', CODES_MISSING_LONG)
codes_set(ibufr, '#4#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#1#windDirection', row["WindDir"])
codes_set(ibufr, '#1#windSpeed', row["WindSpeed"])
codes_set(ibufr, '#4#timeSignificance', CODES_MISSING_LONG)
codes_set(ibufr, '#5#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#1#maximumWindGustDirection', CODES_MISSING_LONG)
codes_set(ibufr, '#1#maximumWindGustSpeed', row["WindGust"])
codes_set(ibufr, '#6#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#2#maximumWindGustDirection', CODES_MISSING_LONG)
codes_set(ibufr, '#2#maximumWindGustSpeed', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#7#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'extremeCounterclockwiseWindDirectionOfAVariableWind', CODES_MISSING_LONG)
codes_set(ibufr, 'extremeClockwiseWindDirectionOfAVariableWind', CODES_MISSING_LONG)
codes_set(ibufr, '#5#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#5#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#8#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'maximumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#1#minimumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#6#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#9#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#2#minimumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#6#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#7#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'methodOfPrecipitationMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, 'methodOfLiquidContentMeasurementOfPrecipitation', CODES_MISSING_LONG)
codes_set(ibufr, '#10#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'totalPrecipitationOrTotalWaterEquivalent', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#8#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'methodOfEvaporationMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, '#11#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'evaporation', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#12#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'totalSunshine', CODES_MISSING_LONG)
codes_set(ibufr, '#13#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'longWaveRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'shortWaveRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'netRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'globalSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'diffuseSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'directSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#14#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'numberOfFlashesThunderstorm', CODES_MISSING_LONG)
codes_set(ibufr, '#15#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#1#firstOrderStatistics', CODES_MISSING_LONG)
codes_set(ibufr, '#2#nonCoordinatePressure', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#windDirection', CODES_MISSING_LONG)
codes_set(ibufr, '#2#windSpeed', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#airTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#relativeHumidity', CODES_MISSING_LONG)
codes_set(ibufr, '#2#firstOrderStatistics', CODES_MISSING_LONG)
codes_set(ibufr, 'qualityInformationAwsData', CODES_MISSING_LONG)
codes_set(ibufr, 'internalMeasurementStatusInformationAws', CODES_MISSING_LONG)
# Encode the keys back in the data section
codes_set(ibufr, 'pack', 1)
def main():
'''
main program reads the command line and encodes the messages into the output filename
to run the program
program_name.py -a Ascii_input_file -b Bufr_output_file
'''
print " codes version {0}".format(codes_get_api_version())
cmdLine=read_cmdline()
inputFilename=cmdLine.ascii
outFilename=cmdLine.bufr
fout=open(outFilename,"w")
message_encoding(inputFilename,fout)
fout.close()
print " output file {0}".format(outFilename)
if __name__ == '__main__':
main()
The program reads the input csv file into a pandas Dataframe and uses it to encode the BUFR data with the function encode_bufr_new. This function receives a handle to the bufr message and the dataframe . A template of this function can
be obtained by using
bufr_dump -E python synop.bufr > synop.py
The synop.bufr file must contain only one message.
The file synop.py contains all the ecCodes instructions needed to produce the synop.bufr file and can be used as a template to create the bufr_encode_new function from the script. In particular, the 301150 sequence can be added at the
beginning of the unexpandedDescriptors list to include the Wigos information accordingly.
The function message_encoding receives the input Filename, reads the data into the dataframe and loops over each of the records of the dataframe creating individual BUFR mesages that are copied into the file object fout.
This function relies on having the BUFR4_local.tmp file that is part of the ecCodes installation in the samples directory ( usually available by the following command codes_info)
codes_info
ecCodes Version 2.9.0
Default definition files path is used: /usr/local/apps/eccodes/2.9.0/GNU/6.3.0/share/eccodes/definitions
Definition files path can be changed setting ECCODES_DEFINITION_PATH environment variable
Default SAMPLES path is used: /usr/local/apps/eccodes/2.9.0/GNU/6.3.0/share/eccodes/samples
SAMPLES path can be changed setting ECCODES_SAMPLES_PATH environment variable
The BUFR file contains section 2 keys that are used locally at ECMWF.
The keys maching the information from the CSV file were populated. For the repeated keys only the first occurrence was populated. The rest of the keys was filled with missing values.
To run the program
./test_wigos4.py -a Ascii_input_file -b Bufr_output_file
This produces the output file Bufr_output_file containing the
output bufr
"key" : "unexpandedDescriptors",
"value" :
[
301150, 307091
]
},
[
[
{
"key" : "subsetNumber",
"value" : 1
},
{
"key" : "wigosIdentifierSeries",
"value" : 0,
"units" : "Numeric"
},
[
{
"key" : "wigosIssuerOfIdentifier",
"value" : 76,
"units" : "Numeric"
This process can be adapted for other BUFR data as well. For instance, from Brazil we received two different BUFR files
SYNOP-wsi.bfr
climat-wsi-test.bfr
To find the proper encoding function to replace bufr_encode_new in the script, we can use the ecCodes command line tool bufr_dump
bufr_dump -E python SYNOP-wsi.bfr > SYNOP-wsi.bfr.py
bufr_dump -E python climat-wsi-test.bfr >climat-wsi-test.bfr.py
This command generates a python script that contains the code to update the bufr_encode_new function.
synop_bfr.py
def bufr_encode():
ibufr = codes_bufr_new_from_samples('BUFR4')
ivalues = (
0, 0 ,)
codes_set_array(ibufr, 'inputDelayedDescriptorReplicationFactor', ivalues)
codes_set(ibufr, 'edition', 4)
codes_set(ibufr, 'masterTableNumber', 0)
codes_set(ibufr, 'bufrHeaderCentre', 43)
codes_set(ibufr, 'bufrHeaderSubCentre', 0)
codes_set(ibufr, 'updateSequenceNumber', 0)
codes_set(ibufr, 'dataCategory', 0)
codes_set(ibufr, 'internationalDataSubCategory', 2)
codes_set(ibufr, 'dataSubCategory', 2)
codes_set(ibufr, 'masterTablesVersionNumber', 28)
codes_set(ibufr, 'localTablesVersionNumber', 0)
codes_set(ibufr, 'typicalYear', 2018)
codes_set(ibufr, 'typicalMonth', 10)
codes_set(ibufr, 'typicalDay', 16)
codes_set(ibufr, 'typicalHour', 0)
codes_set(ibufr, 'typicalMinute', 0)
codes_set(ibufr, 'typicalSecond', 0)
codes_set(ibufr, 'numberOfSubsets', 1)
codes_set(ibufr, 'observedData', 1)
codes_set(ibufr, 'compressedData', 0)
ivalues = (
301150, 307080,)
# Create the structure of the data section
codes_set_array(ibufr, 'unexpandedDescriptors', ivalues)
As the bufr_encode_new function receives a handle ( ibufr) from the message_encoding function so the first line
ibufr = codes_bufr_new_from_samples('BUFR4')
is not needed as the message handle is provided by the message_encoding function.
message_encode
def message_encoding(FullInputFileName,fout):
'''
Message encoding function
FullInputFilename : full path of the Ascii file for example /tmp/data/rema_20180918.txt
fout : file Object to write the output bufr file( obtained by a call to open )
Requires ecCodes and the BUFR4_local template on
ECCODES_PATH/share/eccodes/samples
'''
TEMPLATE='BUFR4_local'
# reads the Ascii file into a pandas Dataframe
dfFull=read_ascii(FullInputFileName)
# loops over the rows of the dataFrame dfFull
for _,row in dfFull.iterrows():
bid=codes_bufr_new_from_samples(TEMPLATE)
try:
bufr_encode_new(bid,row)
codes_write(bid,fout)
except CodesInternalError as ec:
print ec
codes_release(bid)
The last four lines of SYNOP-wsi.bfr.py are not needed either as they are replaced by the try/except block in the message_encoding function
write_block
outfile = open('outfile.bufr', 'w')
codes_write(ibufr, outfile)
print ("Created output BUFR file 'outfile.bufr'")
codes_release(ibufr)
The tool bufr_dump can generate code in Fortran90, Python, C and ecCodes filter language ( that can be executed and tested with bufr_filter command line tool). More information, examples etc available at
UPDATES
Updated the code to encode BUFR and fixed dataframes. Includes new command line options to provide the Wigos Identifier tables and log file, Jose Mauro Resende provided the Wigos Identifier csv file and contributed to the code.
test_wigos5.py
#!/usr/bin/env python
'''
# Copyright 2005-2018 ECMWF.
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction
This is a test program to encode Wigos Synop
requires
1) ecCodes version 2.8 or above (available at https://confluence.ecmwf.int/display/ECC/Releases)
2) python2.7
To run the program
./test_wigos4.py -a Ascii_csv_file -b output_Bufr
Uses BUFR version 4 template 301150 307091
Author : Roberto Ribas Garcia ECMWF 27 Sep 2018
Modifications
Author : Roberto Ribas Garcia ECMWF 7 Nov 2018
added new command line option to read the station codes
added simplified quality control
changed logic to read station codes and update bufr accordingly
added logging information
'''
import argparse
#import ipdb
import pandas as pd
import numpy as np
from datetime import datetime
from eccodes import (codes_set, codes_set_array, codes_bufr_new_from_samples,codes_write,codes_get_api_version,
codes_release, CodesInternalError,CODES_MISSING_DOUBLE,CODES_MISSING_LONG)
import logging
def read_cmdline():
'''
reads the command line to get the input ascii filename and the output bufr file
usage
prog -a Ascii_input_file -b Bufr_output_file -w wigosIDs_csvfile -l logFileName
'''
p = argparse.ArgumentParser()
p.add_argument('-a', '--ascii', help = 'input Ascii filename')
p.add_argument('-b', '--bufr', help = 'output Bufr filename')
p.add_argument("-w", "--wigoscodes",help="csv with the station codes")
p.add_argument("-l","--logging", help="log file")
args = p.parse_args()
return args
def read_ascii(inputFilename):
'''
function to read the Ascii data into a pandas dataframe,
args:
inputFilename : full path of the Ascii file for example /tmp/data/rema_20180918.txt
uses white spaces as column delimiters
index_col = False avoids using the first column (Station) as index
names is the list of names from the excel it can be changed but this affects the dataframe
returns the pandas dataframe dfM (for Measurement)
'''
dfM = pd.read_csv(inputFilename,
header = None,
index_col = False,
delim_whitespace = True,
names = ['station', 'year', 'month', 'day', 'ObsHour', 'TensBat', 'TempCpu', 'airTinst',
'airTmax', 'airTmin', 'relHinst', 'relHmax', 'relHmin', 'dewPInst', 'dewPmax',
'dewPmin', 'PresInst', 'PresMax', 'PresMin', 'WindSpeed', 'WindDir', 'WindGust',
'Rad', 'Precip', 'CloudCoverTot', 'CloudCODE', 'CloudBase', 'Visib'])
cols=dfM.columns.drop("station")
dfM[cols]=dfM[cols].apply(pd.to_numeric,errors="coerce")
return dfM
# column 1 = station column 2 = year column 3 = month column 4 = day column 5 = ObsHour column 6 = TensBat column 7 = TempCpu
# column 8 = airTinst column 9 = airTmax column 10 = airTmin column 11 = relHinst column 12 = relHmax column 13 = relHmin
# column 14 = dewPInst column 15 = dewPmax column 16 = dewPmin column 17 = PresInst column 18 = PresMax column 19 = PresMin
# column 20 = WindSpeed column 21 = WindDir column 22 = WindGust column 23 = Rad column 24 = Precip column 25 = CloudCoverTot
# column 26 = CloudCODE column 27 = CloudBase column 28 = Visib
#
#
# column 1 = station column 2 = Name column 3 = Latitude column 4 = Longitude column 5 = Elevation column 6 = UF
# column 7 = Region column 8 = Date column 9 = ID_WIGOS column 10 = status
def message_encoding(FullInputFileName, fout,dfEst):
'''
Message encoding function
FullInputFilename : full path of the Ascii file for example /tmp/data/rema_20180918.txt
fout : file Object to write the output bufr file( obtained by a call to open )
Requires ecCodes and the BUFR4_local template on
ECCODES_PATH/share/eccodes/samples
'''
TEMPLATE = 'BUFR4_local'
# reads the Ascii file into a pandas Dataframe of measurements dfM
dfM = read_ascii(FullInputFileName)
# Header
fout.write('ISAI99 SBBR ' + str(dfM['day'][0]) + str(dfM['ObsHour'][0]) + '00\r\r\n')
# loops over the rows of the dataFrame dfFull
nValidMessages=0
nWrongMessages=0
for imsg, mrow in dfM.iterrows():
#here it checks that the measurement station is in the WigosCodes file otherwise,
#does not encode the message
stationToEncode=mrow["station"]
dfStationInfo=dfEst.query("CODIGO=='{0}'".format(stationToEncode))
if dfStationInfo.empty:
logging.warn("message for station {0} could not be encoded ".format(mrow["station"]))
nWrongMessages+=1
else:
logging.info( "encoding message {0}".format(imsg+1))
bid = codes_bufr_new_from_samples(TEMPLATE)
try:
bufr_encode_new(bid, mrow,dfStationInfo)
codes_write(bid, fout)
except CodesInternalError as ec:
print ec
codes_release(bid)
nValidMessages+=1
logging.info(" number of valid messages {0} number of invalid messages {1}".format(nValidMessages,nWrongMessages))
def read_estacoes_data(StationCodeFile):
'''
reads the Stations Code File ( with information such as name,lat, lon,alt, WigosId etc)
the name is read through the command line argument -c
'''
dfEst = pd.read_csv(StationCodeFile,
header = 0,
index_col = False)
return dfEst
def bufr_encode_new(ibufr, row,dfStationInfo):
'''
encodes the new SYNO 307091 adding the 1125, 1126, 1127, 1128 wigos keys before.
'''
ivalues = (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,)
codes_set_array(ibufr, 'inputShortDelayedDescriptorReplicationFactor', ivalues)
codes_set(ibufr, 'edition', 4)
codes_set(ibufr, 'masterTableNumber', 0)
codes_set(ibufr, 'bufrHeaderCentre', 43)
codes_set(ibufr, 'bufrHeaderSubCentre', 0)
codes_set(ibufr, 'updateSequenceNumber', 0)
codes_set(ibufr, 'dataCategory', 0)
codes_set(ibufr, 'internationalDataSubCategory', 255)
codes_set(ibufr, 'dataSubCategory', 170)
codes_set(ibufr, 'masterTablesVersionNumber', 29)
codes_set(ibufr, 'localTablesVersionNumber', 0)
# set the YMD
codes_set(ibufr, 'typicalYear', row['year'])
codes_set(ibufr, 'typicalMonth', row['month'])
codes_set(ibufr, 'typicalDay', row['day'])
codes_set(ibufr, 'typicalHour', row['ObsHour'])
codes_set(ibufr, 'typicalMinute', 0)
codes_set(ibufr, 'typicalSecond', 0)
# Encodes the Section 2 of the BUFR used internally at ECMWF (start here)
# codes_set(ibufr, 'rdbType', 1)
# codes_set(ibufr, 'oldSubtype', 176)
# codes_set(ibufr, 'localYear', row['year'])
# codes_set(ibufr, 'localMonth', row['month'])
# codes_set(ibufr, 'localDay', row['day'])
# codes_set(ibufr, 'localHour', row['ObsHour'])
# codes_set(ibufr, 'localMinute', 0)
# codes_set(ibufr, 'localSecond', 0)
# procTime = datetime.now()
# codes_set(ibufr, 'rdbtimeDay', procTime.day)
# codes_set(ibufr, 'rdbtimeHour', procTime.hour)
# codes_set(ibufr, 'rdbtimeMinute', procTime.minute)
# codes_set(ibufr, 'rdbtimeSecond', procTime.second)
# codes_set(ibufr, 'rectimeDay', procTime.day )
# codes_set(ibufr, 'rectimeHour', procTime.hour)
# codes_set(ibufr, 'rectimeMinute', procTime.minute)
# codes_set(ibufr, 'rectimeSecond', procTime.second)
# codes_set(ibufr, 'correction1', 0)
# codes_set(ibufr, 'correction1Part', 0)
# codes_set(ibufr, 'correction2', 0)
# codes_set(ibufr, 'correction2Part', 0)
# codes_set(ibufr, 'correction3', 0)
# codes_set(ibufr, 'correction3Part', 0)
# codes_set(ibufr, 'correction4', 0)
# codes_set(ibufr, 'correction4Part', 0)
# codes_set(ibufr, 'qualityControl', 70)
# codes_set(ibufr, 'newSubtype', 0)
# codes_set(ibufr, 'numberOfSubsets', 1)
# codes_set(ibufr, 'localLatitude', data['lat'])
# codes_set(ibufr, 'localLongitude', data['lon'])
#### End of encoding local section 2
codes_set(ibufr, 'observedData', 1)
codes_set(ibufr, 'compressedData', 0)
ivalues = (301150, 307091)
codes_set_array(ibufr, 'unexpandedDescriptors', ivalues)
codes_set(ibufr, 'wigosIdentifierSeries',0 )
codes_set(ibufr, 'wigosIssuerOfIdentifier', 76)
codes_set(ibufr, 'wigosIssueNumber', 0)
WigosCode=dfStationInfo["WIGOS"].values[0]
wigosIdChar=WigosCode.split("-")[-1]
codes_set(ibufr, 'wigosLocalIdentifierCharacter',wigosIdChar )
codes_set(ibufr, 'stateIdentifier', CODES_MISSING_LONG)
codes_set(ibufr, 'nationalStationNumber', CODES_MISSING_LONG)
block=dfStationInfo["BLOCK"].values[0]
codes_set(ibufr, 'blockNumber', block)
stationNumber=dfStationInfo["STATION"].values[0]
codes_set(ibufr, 'stationNumber', stationNumber)
stationName=dfStationInfo["NOME"].values[0]
codes_set(ibufr, 'stationOrSiteName', stationName)
codes_set(ibufr, 'stationType', 0)
codes_set(ibufr, 'year', row['year'])
codes_set(ibufr, 'month', row['month'])
codes_set(ibufr, 'day', row['day'])
codes_set(ibufr, 'hour', row['ObsHour'])
codes_set(ibufr, 'minute', 0)
stationLat=dfStationInfo["LATITUDE"].values[0]
stationLon=dfStationInfo["LONGITUDE"].values[0]
stationAltitude=dfStationInfo["ALTITUDE"].values[0]
codes_set(ibufr, 'latitude', stationLat)
codes_set(ibufr, 'longitude', stationLon)
codes_set(ibufr, 'heightOfStationGroundAboveMeanSeaLevel', stationAltitude)
altitude = float(stationAltitude) + 1.5
codes_set(ibufr, 'heightOfBarometerAboveMeanSeaLevel', altitude)
codes_set(ibufr, 'surfaceQualifierForTemperatureData', CODES_MISSING_LONG)
codes_set(ibufr, 'mainPresentWeatherDetectingSystem', CODES_MISSING_LONG)
codes_set(ibufr, 'supplementaryPresentWeatherSensor', CODES_MISSING_LONG)
codes_set(ibufr, 'visibilityMeasurementSystem', CODES_MISSING_LONG)
codes_set(ibufr, 'cloudDetectionSystem', CODES_MISSING_LONG)
codes_set(ibufr, 'lightningDetectionSensorType', CODES_MISSING_LONG)
codes_set(ibufr, 'skyConditionAlgorithmType', CODES_MISSING_LONG)
codes_set(ibufr, 'capabilityToDetectPrecipitationPhenomena', CODES_MISSING_LONG)
codes_set(ibufr, 'capabilityToDetectOtherWeatherPhenomena', CODES_MISSING_LONG)
codes_set(ibufr, 'capabilityToDetectObscuration', CODES_MISSING_LONG)
codes_set(ibufr, 'capabilityToDiscriminateLightningStrikes', CODES_MISSING_LONG)
codes_set(ibufr, '#1#nonCoordinatePressure', CODES_MISSING_DOUBLE)
pressure=row["PresInst"]
pressure = float(pressure) * 100
codes_set(ibufr, 'pressureReducedToMeanSeaLevel', pressure)
codes_set(ibufr, '3HourPressureChange', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'characteristicOfPressureTendency', CODES_MISSING_LONG)
codes_set(ibufr, 'pressure', pressure)
codes_set(ibufr, 'nonCoordinateGeopotentialHeight', CODES_MISSING_LONG)
codes_set(ibufr, '#1#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#1#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
airTInst=row["airTinst"]
if airTInst!=CODES_MISSING_DOUBLE:
temperature = airTInst + 273.15
else:
temperature=airTInst
codes_set(ibufr, '#1#airTemperature', temperature)
dewPInst=row["dewPInst"]
dewPInst=dewPInst+273.15
codes_set(ibufr, 'dewpointTemperature', dewPInst)
relHinst=row["relHinst"]
codes_set(ibufr, '#1#relativeHumidity', relHinst)
codes_set(ibufr, '#1#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#1#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#4#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#4#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#5#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#5#soilTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#6#depthBelowLandSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#1#attributeOfFollowingValue', CODES_MISSING_LONG)
visib=row["Visib"]
codes_set(ibufr, 'horizontalVisibility', visib)
codes_set(ibufr, '#3#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'iceDepositThickness', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'rateOfIceAccretionEstimated', CODES_MISSING_LONG)
codes_set(ibufr, 'methodOfWaterTemperatureAndOrOrSalinityMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, 'oceanographicWaterTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'wavesDirection', CODES_MISSING_LONG)
codes_set(ibufr, 'periodOfWaves', CODES_MISSING_LONG)
codes_set(ibufr, 'heightOfWaves', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'methodOfStateOfGroundMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, 'stateOfGround', CODES_MISSING_LONG)
codes_set(ibufr, 'methodOfSnowDepthMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, 'totalSnowDepth', CODES_MISSING_DOUBLE)
cloudCoverTot=row["CloudCoverTot"]
codes_set(ibufr, 'cloudCoverTotal', cloudCoverTot)
codes_set(ibufr, '#1#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
codes_set(ibufr, '#1#cloudAmount', CODES_MISSING_LONG)
codes_set(ibufr, '#1#cloudType', CODES_MISSING_LONG)
codes_set(ibufr, '#2#attributeOfFollowingValue', CODES_MISSING_LONG)
codes_set(ibufr, '#1#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
codes_set(ibufr, '#2#cloudAmount', CODES_MISSING_LONG)
codes_set(ibufr, '#2#cloudType', CODES_MISSING_LONG)
codes_set(ibufr, '#3#attributeOfFollowingValue', CODES_MISSING_LONG)
codes_set(ibufr, '#2#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
codes_set(ibufr, '#3#cloudAmount', CODES_MISSING_LONG)
codes_set(ibufr, '#3#cloudType', CODES_MISSING_LONG)
codes_set(ibufr, '#4#attributeOfFollowingValue', CODES_MISSING_LONG)
codes_set(ibufr, '#3#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#4#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
codes_set(ibufr, '#4#cloudAmount', CODES_MISSING_LONG)
codes_set(ibufr, '#4#cloudType', CODES_MISSING_LONG)
codes_set(ibufr, '#5#attributeOfFollowingValue', CODES_MISSING_LONG)
codes_set(ibufr, '#4#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'presentWeather', CODES_MISSING_LONG)
codes_set(ibufr, '#1#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'pastWeather1', CODES_MISSING_LONG)
codes_set(ibufr, 'pastWeather2', CODES_MISSING_LONG)
codes_set(ibufr, '#1#timeSignificance', CODES_MISSING_LONG)
codes_set(ibufr, '#2#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'precipitationIntensityHighAccuracy', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'sizeOfPrecipitatingElement', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#timeSignificance', CODES_MISSING_LONG)
codes_set(ibufr, '#3#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'precipitationType', CODES_MISSING_LONG)
codes_set(ibufr, 'characterOfPrecipitation', CODES_MISSING_LONG)
codes_set(ibufr, 'durationOfPrecipitation', CODES_MISSING_LONG)
codes_set(ibufr, 'otherWeatherPhenomena', CODES_MISSING_LONG)
codes_set(ibufr, 'intensityOfPhenomena', CODES_MISSING_LONG)
codes_set(ibufr, 'obscuration', CODES_MISSING_LONG)
codes_set(ibufr, 'characterOfObscuration', CODES_MISSING_LONG)
codes_set(ibufr, '#4#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#4#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#3#timeSignificance', CODES_MISSING_LONG)
codes_set(ibufr, '#4#timePeriod', CODES_MISSING_LONG)
windDir=row["WindDir"]
codes_set(ibufr, '#1#windDirection', float(windDir))
windSpeed=row['WindSpeed']
codes_set(ibufr, '#1#windSpeed', float(windSpeed))
codes_set(ibufr, '#4#timeSignificance', CODES_MISSING_LONG)
codes_set(ibufr, '#5#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#1#maximumWindGustDirection', CODES_MISSING_LONG)
windGust=row["WindGust"]
codes_set(ibufr, '#1#maximumWindGustSpeed', windGust)
codes_set(ibufr, '#6#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#2#maximumWindGustDirection', CODES_MISSING_LONG)
codes_set(ibufr, '#2#maximumWindGustSpeed', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#7#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'extremeCounterclockwiseWindDirectionOfAVariableWind', CODES_MISSING_LONG)
codes_set(ibufr, 'extremeClockwiseWindDirectionOfAVariableWind', CODES_MISSING_LONG)
codes_set(ibufr, '#5#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#5#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#8#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'maximumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#1#minimumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#6#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#9#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#2#minimumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#6#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#7#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'methodOfPrecipitationMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, 'methodOfLiquidContentMeasurementOfPrecipitation', CODES_MISSING_LONG)
codes_set(ibufr, '#10#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'totalPrecipitationOrTotalWaterEquivalent', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#8#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'methodOfEvaporationMeasurement', CODES_MISSING_LONG)
codes_set(ibufr, '#11#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'evaporation', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#12#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'totalSunshine', CODES_MISSING_LONG)
codes_set(ibufr, '#13#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'longWaveRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'shortWaveRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'netRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'globalSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'diffuseSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, 'directSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#14#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, 'numberOfFlashesThunderstorm', CODES_MISSING_LONG)
codes_set(ibufr, '#15#timePeriod', CODES_MISSING_LONG)
codes_set(ibufr, '#1#firstOrderStatistics', CODES_MISSING_LONG)
codes_set(ibufr, '#2#nonCoordinatePressure', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#windDirection', CODES_MISSING_LONG)
codes_set(ibufr, '#2#windSpeed', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#airTemperature', CODES_MISSING_DOUBLE)
codes_set(ibufr, '#2#relativeHumidity', CODES_MISSING_LONG)
codes_set(ibufr, '#2#firstOrderStatistics', CODES_MISSING_LONG)
codes_set(ibufr, 'qualityInformationAwsData', CODES_MISSING_LONG)
codes_set(ibufr, 'internalMeasurementStatusInformationAws', CODES_MISSING_LONG)
#Encode the keys back in the data section
codes_set(ibufr, 'pack', 1)
def main():
'''
main program reads the command line and encodes the messages into the output filename
to run the program
prog -a Ascii_input_file -b Bufr_output_file -w wigosIDs_csvfile -l logFileName
'''
cmdLine = read_cmdline()
inputFilename = cmdLine.ascii
outFilename = cmdLine.bufr
stationFile=cmdLine.wigoscodes
logFile=cmdLine.logging
logging.basicConfig(filename=logFile,format="%(asctime)s %(levelname)s:%(message)s ",level=logging.DEBUG)
logging.info("Using ecCodes version {0}".format(codes_get_api_version()))
fout = open(outFilename, 'w')
dfEstacoes=read_estacoes_data(stationFile)
message_encoding(inputFilename, fout,dfEstacoes)
fout.close()
logging.info(' output file {0}'.format(outFilename))
if __name__ == '__main__':
main()
A quality control of the observations is needed to avoid encoding wrong values in the output BUFR.
#!/usr/bin/python
'''
# Copyright 2005-2018 ECMWF.
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction
This is a test program to encode Wigos Synop
requires
1) ecCodes version 2.8 or above (available at https://confluence.ecmwf.int/display/ECC/Releases)
2) python2.7
To run the program
./test_wigos_NEW.py -a Ascii_csv_file -b output_Bufr -w WIGOS_TABLE_FILE -l logFile
Uses BUFR version 4 template 301150 307091
Author : Roberto Ribas Garcia ECMWF 27 Sep 2018
Modifications
Author : Roberto Ribas Garcia ECMWF 7 Nov 2018
added new command line option to read the station codes
changed logic to read station codes and update bufr accordingly
added logging information
Roberto Ribas Garcia, ECMWF 24 Jan 2019 changed the encoding function
to have one header.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!-------------------------------------- CAVEATS ---------------------------
!!!!!!!!!!!!!!!!!!!!!!!!! A QUALITY CHECK IMPLEMENTATION IS NEEDED
!!!!!!!!!!!!!!!!!!!!!!!!! IT MAY BE INCOMPLETE CHECK AND TEST CAREFULLY
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
'''
import sys
#sys.path.append('/usr/local/lib64/python2.7/site-packages/')
# adapt the path to your installation
sys.path.append("/usr/local/apps/eccodes/2.10.0/GNU/6.3.0/lib/python2.7/site-packages/")
import argparse
#import ipdb
import pandas as pd
import numpy as np
import logging
from datetime import datetime
from eccodes import (codes_set, codes_set_array, codes_bufr_new_from_samples,codes_write,codes_get_api_version,
codes_release, CodesInternalError,CODES_MISSING_DOUBLE,CODES_MISSING_LONG)
def read_cmdline():
'''
reads the command line to get the input ascii filename and the output bufr file
usage
prog -a Ascii_input_file -b Bufr_output_file -l logfile -w WIGOS_TABLE_FILE
'''
p = argparse.ArgumentParser()
p.add_argument('-a', '--ascii', help = 'input Ascii filename')
p.add_argument('-b', '--bufr', help = 'output Bufr filename')
p.add_argument("-w", "--wigoscodes", help = "csv with the station codes")
p.add_argument("-l","--logging", help = "log file")
args = p.parse_args()
return args
def bufr_encode_new(dfMeteo,dfEst,fout):
'''
encodes the bufr file with one BUFR header and loops over the
the dfMeteo dataframe.
dfEst : is the station WiGOS info read from the WIGOS_TABLE_FILE
dfMeteo : contains the meteo information
fout : is an file object opened for writing in the main program
'''
ibufr = codes_bufr_new_from_samples('BUFR4')
nSubsets=len(dfMeteo.index)
oneSubsetDelayed=np.array([0, 0, 0, 0, 1, 1, 0, 0, 1, 1,1, 1, 0, 0,],dtype=np.int)
DelayedDesc=np.tile(oneSubsetDelayed,nSubsets)
### this DelayedDesc array is a replication of the oneSubsetDelayed array as many
### times as subsets are present in the message. The number of subsets is found
### as the number of lines of the dfMeteo dataframe
### this is used to populate the key inputShortDelayedDescriptorReplicationFactor
codes_set_array(ibufr, 'inputShortDelayedDescriptorReplicationFactor', DelayedDesc)
codes_set(ibufr, 'edition', 4)
codes_set(ibufr, 'masterTableNumber', 0)
codes_set(ibufr, 'bufrHeaderCentre', 43)
codes_set(ibufr, 'bufrHeaderSubCentre', 0)
codes_set(ibufr, 'updateSequenceNumber', 0)
codes_set(ibufr, 'dataCategory', 0)
codes_set(ibufr, 'internationalDataSubCategory', 7)
codes_set(ibufr, 'dataSubCategory', 7)
codes_set(ibufr, 'masterTablesVersionNumber', 28)
codes_set(ibufr, 'localTablesVersionNumber', 0)
### check if this suits your system
procTime=datetime.now()
codes_set(ibufr, 'typicalYear', procTime.year)
codes_set(ibufr, 'typicalMonth', procTime.hour)
codes_set(ibufr, 'typicalDay', procTime.day)
codes_set(ibufr, 'typicalHour', procTime.hour)
codes_set(ibufr, 'typicalMinute', procTime.minute)
codes_set(ibufr, 'typicalSecond', procTime.second)
codes_set(ibufr, 'numberOfSubsets', nSubsets)
codes_set(ibufr, 'observedData', 1)
codes_set(ibufr, 'compressedData', 0)
ivalues = ( 301150, 307091,)
# Create the structure of the data section
codes_set_array(ibufr, 'unexpandedDescriptors', ivalues)
## these are labels to update the different keys MAY NEED MORE LABELS
lblHOS1=1
lblHOS2=1
lblHOSMP=1
lblTS=1
lblTP=1
### loops over the dfMeteo
nValidMsg=0
nWrongMsg=0
for imsg,row in dfMeteo.iterrows():
stationID=row["station"].strip()
dfEstInfo=dfEst[dfEst["CODIGO"]==stationID]
if not dfEstInfo.empty:
nValidMsg+=1
key="#{0}#wigosIdentifierSeries".format(imsg+1)
codes_set(ibufr, key,0 )
key="#{0}#wigosIssuerOfIdentifier".format(imsg+1)
codes_set(ibufr, key, 76)
key="#{0}#wigosIssueNumber".format(imsg+1)
codes_set(ibufr, key, 0)
wigosLocalID=dfEstInfo["WIGOS"].values[0]
WigosCode=dfEstInfo["WIGOS"].values[0]
wigosIdChar=WigosCode.split("-")[-1]
key='#{0}#wigosLocalIdentifierCharacter'.format(imsg+1)
codes_set(ibufr,key ,wigosIdChar)
key="#{0}#stateIdentifier".format(imsg+1)
codes_set(ibufr, key, CODES_MISSING_LONG)
key='#{0}#nationalStationNumber'.format(imsg+1)
codes_set(ibufr,key , CODES_MISSING_LONG)
block=dfEstInfo["BLOCK"].values[0]
key='#{0}#blockNumber'.format(imsg+1)
codes_set(ibufr, key, block)
stationNumber=dfEstInfo["STATION"].values[0]
key='#{0}#stationNumber'.format(imsg+1)
codes_set(ibufr, key, stationNumber)
stationName=dfEstInfo["NOME"].values[0]
key='#{0}#stationOrSiteName'.format(imsg+1)
codes_set(ibufr,key ,stationName)
key='#{0}#stationType'.format(imsg+1)
codes_set(ibufr,key , CODES_MISSING_LONG)
key="#{0}#year".format(imsg+1)
codes_set(ibufr,key , row["year"])
key='#{0}#month'.format(imsg+1)
codes_set(ibufr, key, row["month"])
key="#{0}#day".format(imsg+1)
codes_set(ibufr, key, row["day"])
key="#{0}#hour".format(imsg+1)
codes_set(ibufr, key, row["ObsHour"])
key="#{0}#minute".format(imsg+1)
codes_set(ibufr, key, 0)
stationLat=dfEstInfo["LATITUDE"].values[0]
stationLon=dfEstInfo["LONGITUDE"].values[0]
stationAltitude=dfEstInfo["ALTITUDE"].values[0]
key="#{0}#latitude".format(imsg+1)
codes_set(ibufr,key,stationLat)
key="#{0}#longitude".format(imsg+1)
codes_set(ibufr,key,stationLon)
key="#{0}#heightOfStationGroundAboveMeanSeaLevel".format(lblHOS1)
codes_set(ibufr,key,stationAltitude)
lblHOS1+=1
altitude = float(stationAltitude) + 1.5
key="#{0}#heightOfBarometerAboveMeanSeaLevel".format(lblHOS2)
codes_set(ibufr,key,altitude)
lblHOS2+=1
key="#{0}#surfaceQualifierForTemperatureData".format(imsg+1)
codes_set(ibufr,key,3)
key='#{0}#mainPresentWeatherDetectingSystem'.format(imsg+1)
codes_set(ibufr, key, CODES_MISSING_LONG)
key='#{0}#supplementaryPresentWeatherSensor'.format(imsg+1)
codes_set(ibufr, key, CODES_MISSING_LONG)
key='#{0}#visibilityMeasurementSystem'.format(imsg+1)
codes_set(ibufr,key , CODES_MISSING_LONG)
key='#{0}#cloudDetectionSystem'.format(imsg+1)
codes_set(ibufr,key, CODES_MISSING_LONG)
key='#{0}#lightningDetectionSensorType'.format(imsg+1)
codes_set(ibufr,key , CODES_MISSING_LONG)
key='#{0}#skyConditionAlgorithmType'.format(imsg+1)
codes_set(ibufr,key , CODES_MISSING_LONG)
key='#{0}#capabilityToDetectPrecipitationPhenomena'.format(imsg+1)
codes_set(ibufr,key , CODES_MISSING_LONG)
key='#{0}#capabilityToDetectOtherWeatherPhenomena'.format(imsg+1)
codes_set(ibufr,key , CODES_MISSING_LONG)
key='#{0}#capabilityToDetectObscuration'.format(imsg+1)
codes_set(ibufr,key , CODES_MISSING_LONG)
key='#{0}#capabilityToDiscriminateLightningStrikes'.format(imsg+1)
codes_set(ibufr, key, CODES_MISSING_LONG)
key="#{0}#nonCoordinatePressure".format(imsg+1)
codes_set(ibufr,key,CODES_MISSING_DOUBLE)
pressure=row["PresInst"]
pressure = float(pressure) * 100
key="#{0}#pressureReducedToMeanSeaLevel".format(imsg+1)
codes_set(ibufr,key,pressure)
key="#{0}#3HourPressureChange".format(imsg+1)
codes_set(ibufr,key,CODES_MISSING_DOUBLE)
key="#{0}#characteristicOfPressureTendency".format(imsg+1)
codes_set(ibufr,key,2)
key="#{0}#pressure".format(imsg+1)
codes_set(ibufr, key, CODES_MISSING_DOUBLE)
key='#{0}#nonCoordinateGeopotentialHeight'.format(imsg+1)
codes_set(ibufr, key, CODES_MISSING_LONG)
key='#{0}#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform'.format(lblHOSMP)
codes_set(ibufr,key ,CODES_MISSING_DOUBLE)
lblHOSMP+=1
airTInst=row["airTinst"]
if airTInst!=CODES_MISSING_DOUBLE:
temperature = airTInst + 273.15
else:
temperature=airTInst
key="#{0}#airTemperature".format(imsg+1)
codes_set(ibufr,key,temperature)
dewPInst=row["dewPInst"]
dewPInst=dewPInst+273.15
key="#{0}#dewpointTemperature".format(imsg+1)
codes_set(ibufr,key,dewPInst)
relHinst=row["relHinst"]
key="#{0}#relativeHumidity".format(imsg+1)
codes_set(ibufr,key , relHinst)
key="#{0}#timeSignificance".format(lblTS)
codes_set(ibufr, key, 2)
lblTS+=1
key="#{0}#timePeriod".format(lblTP)
codes_set(ibufr, key, -10)
lblTP+=1
windDir=row["WindDir"]
windSpeed=row['WindSpeed']
key="#{0}#windDirection".format(imsg+1)
codes_set(ibufr,key, windDir)
key='#{0}#windSpeed'.format(imsg+1)
codes_set(ibufr,key , windSpeed)
key="#{0}#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform".format(lblHOSMP)
codes_set(ibufr,key,10)
key="#{0}#timePeriod".format(lblTP)
codes_set(ibufr,key,-10)
lblTP+=1
key="#{0}#timePeriod".format(lblTP)
codes_set(ibufr,key,-60)
key='#{0}#totalPrecipitationOrTotalWaterEquivalent'.format(imsg+1)
precip=row["Precip"]
codes_set(ibufr,key , precip)
key="#{0}#netRadiationIntegratedOverPeriodSpecified".format(imsg+1)
rad=row["Rad"]
codes_set(ibufr,key,rad)
else:
logging.info("Station {0} not found in the WIGOS TABLE FILE".format(stationID))
nWrongMsg+=1
#######################################################
###### at the end pack, write and release ibufr
codes_set(ibufr, 'pack', 1)
codes_write(ibufr, fout)
codes_release(ibufr)
return nValidMsg,nWrongMsg
def read_ascii(inputFilename):
'''
function to read the Ascii data into a pandas dataframe,
args:
inputFilename : full path of the Ascii file for example /tmp/data/rema_20180918.txt
uses white spaces as column delimiters
index_col = False avoids using the first column (Station) as index
names is the list of names from the excel it can be changed but this affects the dataframe
returns the pandas dataframe dfM (for Measurement)
'''
dfM = pd.read_csv(inputFilename,
header = None,
index_col = False,
delim_whitespace = True,
names = ['station', 'year', 'month', 'day', 'ObsHour', 'TensBat', 'TempCpu', 'airTinst',
'airTmax', 'airTmin', 'relHinst', 'relHmax', 'relHmin', 'dewPInst', 'dewPmax',
'dewPmin', 'PresInst', 'PresMax', 'PresMin', 'WindSpeed', 'WindDir', 'WindGust',
'Rad', 'Precip', 'CloudCoverTot', 'CloudCODE', 'CloudBase', 'Visib'])
cols = dfM.columns.drop('station')
dfM[cols] = dfM[cols].apply(pd.to_numeric, errors = 'coerce')
return dfM
# column 1 = station column 2 = year column 3 = month column 4 = day column 5 = ObsHour column 6 = TensBat column 7 = TempCpu
# column 8 = airTinst column 9 = airTmax column 10 = airTmin column 11 = relHinst column 12 = relHmax column 13 = relHmin
# column 14 = dewPInst column 15 = dewPmax column 16 = dewPmin column 17 = PresInst column 18 = PresMax column 19 = PresMin
# column 20 = WindSpeed column 21 = WindDir column 22 = WindGust column 23 = Rad column 24 = Precip column 25 = CloudCoverTot
# column 26 = CloudCODE column 27 = CloudBase column 28 = Visib
#
#
# column 1 = station column 2 = Name column 3 = Latitude column 4 = Longitude column 5 = Elevation column 6 = UF
# column 7 = Region column 8 = Date column 9 = ID_WIGOS column 10 = status
def read_estacoes_data(StationCodeFile):
'''
reads the Stations Code File ( with information such as name,lat, lon,alt, WigosId etc)
the name is read through the command line argument -c
'''
dfEst = pd.read_csv(StationCodeFile,
header = 0,
index_col = False)
return dfEst
def main():
'''
main program reads the command line and encodes the messages into the output filename
to run the program
program_name.py -a Ascii_input_file -b Bufr_output_file -w WIGOS_TABLE_FILE -l logFile
'''
cmdLine = read_cmdline()
inputFilename = cmdLine.ascii
##### changed this to make it work, change back according to your requirements
# FileHeader
#data_hora = inputFilename.split('_')
#data_hora = data_hora[1].split('.aut')
#data_hora = data_hora[0]
#data_hora = list(data_hora)
#dia = data_hora[6] + data_hora[7]
#hora = data_hora[8] + data_hora
dia="20"
hora="12"
FileHeader = 'ISAI99 SBBR ' + dia + hora + '00\r\r\n'
outFilename = cmdLine.bufr
stationFile=cmdLine.wigoscodes
logFile = cmdLine.logging
logging.basicConfig(filename=logFile,format = "%(asctime)s %(levelname)s:%(message)s ",level=logging.DEBUG)
logging.info("Using ecCodes version {0}".format(codes_get_api_version()))
fout = open(outFilename, 'w')
fout.write(FileHeader)
dfEstacoes = read_estacoes_data(stationFile)
dfMeteo=read_ascii(inputFilename)
nValidMsg,nWrongMsg=bufr_encode_new(dfMeteo,dfEstacoes,fout)
fout.close()
logging.info( "number of valid Messages {0} wrong messages {1}".format(nValidMsg,nWrongMsg))
logging.info(' output file {0}'.format(outFilename))
if __name__ == '__main__':
main()
Changed the bufr structure at request from Brazil. Now, there is only one bufr header and the different stations appear as subsets of the message.
The new program is not complete. Some work is requiered on the user side to fully adapt to the bufr structure and perhaps simplify the bufr_encode_new function.
DISCLAIMER: This software is intended for testing purposes only. Feedback would be appreciated but technical support can only be given if our workload permits.