Page History

...

This command generates a python code script that contains the code to update the bufr_encode_new function.

...

ibufr = codes_bufr_new_from_samples('BUFR4')

is not needed This function message_encoding generates handles to bufr messages using a template BUFR4_local that is part of ecCodes installation ( can be seen by using codes_info)as the message handle is provided by the message_encoding function.

Code Block

language	py
title	message_encode

def message_encoding(FullInputFileName,fout):
    '''
    Message encoding function 
    FullInputFilename      :     full path of the Ascii file for example /tmp/data/rema_20180918.txt
    fout                   :     file Object to write the output bufr file( obtained by a call to open )
    
    Requires ecCodes and the BUFR4_local template on  
                 ECCODES_PATH/share/eccodes/samples

    '''
    TEMPLATE='BUFR4_local'
    
    # reads the Ascii file into a pandas Dataframe
    dfFull=read_ascii(FullInputFileName)
  
    # loops over the rows of the dataFrame dfFull  
    for _,row in dfFull.iterrows():
        bid=codes_bufr_new_from_samples(TEMPLATE)
        try:
            bufr_encode_new(bid,row)
            codes_write(bid,fout)
        except CodesInternalError as ec:
            print ec
        codes_release(bid)

...

Code Block

language	py
title	write_block

    outfile = open('outfile.bufr', 'w')
    codes_write(ibufr, outfile)
    print ("Created output BUFR file 'outfile.bufr'")
    codes_release(ibufr)

The tool bufr_dump can generate code in Fortran90, Python, C and ecCodes filter language ( that can be executed and tested with bufr_filter command line tool). More information, examples etc available at

ecCodes Home

UPDATES

Updated the code to encode BUFR and fixed dataframes. Includes new command line options to provide the Wigos Identifier tables and log file, Jose Mauro Resende provided the Wigos Identifier csv file and contributed to the code.

Code Block

language	py
title	test_wigos5.py

#!/usr/bin/env python
'''
# Copyright 2005-2018 ECMWF.
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction

This is a test program to encode Wigos Synop 
requires 

1) ecCodes version 2.8 or above (available at https://confluence.ecmwf.int/display/ECC/Releases)
2) python2.7

To run the program

   ./test_wigos4.py -a Ascii_csv_file  -b output_Bufr
   
Uses BUFR version 4 template  301150 307091

Author : Roberto Ribas Garcia ECMWF 27 Sep 2018
Modifications
Author : Roberto Ribas Garcia ECMWF 7 Nov 2018
         added new command line option to read the station codes
         added simplified quality control 
         changed logic to read station codes and update bufr accordingly
         added logging information 
'''
import argparse 
#import ipdb
import pandas as pd
import numpy as np
from datetime import datetime
from eccodes import (codes_set, codes_set_array, codes_bufr_new_from_samples,codes_write,codes_get_api_version, 
                     codes_release, CodesInternalError,CODES_MISSING_DOUBLE,CODES_MISSING_LONG) 
import logging 



    
def read_cmdline():
    '''
    reads the command line to get the input ascii filename and the output bufr file
        usage
            prog  -a Ascii_input_file  -b Bufr_output_file -w wigosIDs_csvfile -l logFileName
    '''
    p = argparse.ArgumentParser()
    p.add_argument('-a', '--ascii', help = 'input Ascii filename')
    p.add_argument('-b', '--bufr', help = 'output Bufr filename')
    p.add_argument("-w", "--wigoscodes",help="csv with the station codes")
    p.add_argument("-l","--logging", help="log file")
    args = p.parse_args()
    return args


def read_ascii(inputFilename):
    '''
    function to read the Ascii data into a pandas dataframe, 
        args:
            inputFilename : full path of the Ascii file for example /tmp/data/rema_20180918.txt
     
    uses white spaces as column delimiters
    index_col = False avoids using the first column (Station) as index
    names is the list of names from the excel it can be changed but this affects the dataframe
    returns the pandas dataframe dfM (for Measurement)
    '''
    dfM = pd.read_csv(inputFilename,
                     header = None, 
                     index_col = False,
                     delim_whitespace = True,
                     names = ['station', 'year', 'month', 'day', 'ObsHour', 'TensBat', 'TempCpu', 'airTinst',
                              'airTmax', 'airTmin', 'relHinst', 'relHmax', 'relHmin', 'dewPInst', 'dewPmax',
                              'dewPmin', 'PresInst', 'PresMax', 'PresMin', 'WindSpeed', 'WindDir', 'WindGust',
                              'Rad', 'Precip', 'CloudCoverTot', 'CloudCODE', 'CloudBase', 'Visib'])
    cols=dfM.columns.drop("station")
    dfM[cols]=dfM[cols].apply(pd.to_numeric,errors="coerce")
    return dfM


# column  1 = station    column 2 = year     column  3 = month     column 4 = day       column  5 = ObsHour  column  6 = TensBat  column 7 = TempCpu
# column  8 = airTinst   column 9 = airTmax  column 10 = airTmin  column 11 = relHinst  column 12 = relHmax  column 13 = relHmin
# column 14 = dewPInst  column 15 = dewPmax  column 16 = dewPmin  column 17 = PresInst  column 18 = PresMax  column 19 = PresMin
# column 20 = WindSpeed column 21 = WindDir  column 22 = WindGust column 23 = Rad       column 24 = Precip   column 25 = CloudCoverTot
# column 26 = CloudCODE column 27 = CloudBase column 28 = Visib
#
#     
# column 1 = station  column 2 = Name  column 3 = Latitude  column  4 = Longitude  column 5 = Elevation  column 6 = UF
# column 7 = Region   column 8 = Date  column 9 = ID_WIGOS  column 10 = status





def message_encoding(FullInputFileName, fout,dfEst):
    '''
    Message encoding function 
    FullInputFilename      :     full path of the Ascii file for example /tmp/data/rema_20180918.txt
    fout                   :     file Object to write the output bufr file( obtained by a call to open )
    
    Requires ecCodes and the BUFR4_local template on  
                  ECCODES_PATH/share/eccodes/samples
    '''
    TEMPLATE = 'BUFR4_local'

    # reads the Ascii file into a pandas Dataframe of measurements dfM
    dfM = read_ascii(FullInputFileName)

    # Header
    fout.write('ISAI99 SBBR ' + str(dfM['day'][0]) + str(dfM['ObsHour'][0]) + '00\r\r\n')

    # loops over the rows of the dataFrame dfFull
    nValidMessages=0
    nWrongMessages=0  
    for imsg, mrow in dfM.iterrows():
        #here it checks that the measurement station is in the WigosCodes file otherwise,
        #does not encode the message  
        
        stationToEncode=mrow["station"]
        dfStationInfo=dfEst.query("CODIGO=='{0}'".format(stationToEncode))
        
        if dfStationInfo.empty:
            logging.warn("message for station {0} could not be encoded ".format(mrow["station"]))
            nWrongMessages+=1
        else:
            logging.info( "encoding message {0}".format(imsg+1))
            bid = codes_bufr_new_from_samples(TEMPLATE)
            try:
                bufr_encode_new(bid, mrow,dfStationInfo)
                codes_write(bid, fout)
            except CodesInternalError as ec:
                print ec
            codes_release(bid)
            nValidMessages+=1
            
    logging.info(" number of valid messages {0} number of invalid messages {1}".format(nValidMessages,nWrongMessages))
def read_estacoes_data(StationCodeFile):
    '''
    reads the Stations Code File ( with information such as name,lat, lon,alt, WigosId etc)
    the name is read through the command line argument -c
    '''
    dfEst = pd.read_csv(StationCodeFile,
                      header = 0,
                      index_col = False)
                      
    return dfEst


def bufr_encode_new(ibufr, row,dfStationInfo):


    '''
    encodes the new SYNO 307091 adding the  1125, 1126, 1127, 1128 wigos keys before.
    '''
    ivalues = (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,)
    codes_set_array(ibufr, 'inputShortDelayedDescriptorReplicationFactor', ivalues)
    codes_set(ibufr, 'edition', 4)
    codes_set(ibufr, 'masterTableNumber', 0)
    codes_set(ibufr, 'bufrHeaderCentre', 43)
    codes_set(ibufr, 'bufrHeaderSubCentre', 0)
    codes_set(ibufr, 'updateSequenceNumber', 0)
    codes_set(ibufr, 'dataCategory', 0)
    codes_set(ibufr, 'internationalDataSubCategory', 255)
    codes_set(ibufr, 'dataSubCategory', 170)
    codes_set(ibufr, 'masterTablesVersionNumber', 29)
    codes_set(ibufr, 'localTablesVersionNumber', 0)
    # set the YMD
    codes_set(ibufr, 'typicalYear', row['year'])
    codes_set(ibufr, 'typicalMonth', row['month'])
    codes_set(ibufr, 'typicalDay', row['day'])
    codes_set(ibufr, 'typicalHour', row['ObsHour'])
    codes_set(ibufr, 'typicalMinute', 0)
    codes_set(ibufr, 'typicalSecond', 0)
    # Encodes  the Section 2 of the BUFR used internally at ECMWF (start here)
    # codes_set(ibufr, 'rdbType', 1)
    # codes_set(ibufr, 'oldSubtype', 176)
    # codes_set(ibufr, 'localYear', row['year'])
    # codes_set(ibufr, 'localMonth', row['month'])
    # codes_set(ibufr, 'localDay', row['day'])
    # codes_set(ibufr, 'localHour', row['ObsHour'])
    # codes_set(ibufr, 'localMinute', 0)
    # codes_set(ibufr, 'localSecond', 0)
    # procTime = datetime.now()
    # codes_set(ibufr, 'rdbtimeDay', procTime.day)
    # codes_set(ibufr, 'rdbtimeHour', procTime.hour)
    # codes_set(ibufr, 'rdbtimeMinute', procTime.minute)
    # codes_set(ibufr, 'rdbtimeSecond', procTime.second)
    # codes_set(ibufr, 'rectimeDay', procTime.day )
    # codes_set(ibufr, 'rectimeHour', procTime.hour)
    # codes_set(ibufr, 'rectimeMinute', procTime.minute)
    # codes_set(ibufr, 'rectimeSecond', procTime.second)
    # codes_set(ibufr, 'correction1', 0)
    # codes_set(ibufr, 'correction1Part', 0)
    # codes_set(ibufr, 'correction2', 0)
    # codes_set(ibufr, 'correction2Part', 0)
    # codes_set(ibufr, 'correction3', 0)
    # codes_set(ibufr, 'correction3Part', 0)
    # codes_set(ibufr, 'correction4', 0)
    # codes_set(ibufr, 'correction4Part', 0)
    # codes_set(ibufr, 'qualityControl', 70)
    # codes_set(ibufr, 'newSubtype', 0)
    # codes_set(ibufr, 'numberOfSubsets', 1)
    # codes_set(ibufr, 'localLatitude', data['lat'])
    # codes_set(ibufr, 'localLongitude', data['lon'])
    #### End of encoding local section 2 
    codes_set(ibufr, 'observedData', 1)
    codes_set(ibufr, 'compressedData', 0)
    ivalues = (301150, 307091)
    codes_set_array(ibufr, 'unexpandedDescriptors', ivalues)
    codes_set(ibufr, 'wigosIdentifierSeries',0 )
    codes_set(ibufr, 'wigosIssuerOfIdentifier', 76)
    codes_set(ibufr, 'wigosIssueNumber', 0)
    WigosCode=dfStationInfo["WIGOS"].values[0]
    wigosIdChar=WigosCode.split("-")[-1]
    codes_set(ibufr, 'wigosLocalIdentifierCharacter',wigosIdChar )
    
    codes_set(ibufr, 'stateIdentifier', CODES_MISSING_LONG)
    codes_set(ibufr, 'nationalStationNumber', CODES_MISSING_LONG)
    
    block=dfStationInfo["BLOCK"].values[0]
    codes_set(ibufr, 'blockNumber', block)
    
    stationNumber=dfStationInfo["STATION"].values[0]
    codes_set(ibufr, 'stationNumber', stationNumber)
    
    stationName=dfStationInfo["NOME"].values[0]
    codes_set(ibufr, 'stationOrSiteName', stationName)
    
    codes_set(ibufr, 'stationType', 0)
    codes_set(ibufr, 'year', row['year'])
    codes_set(ibufr, 'month', row['month'])
    codes_set(ibufr, 'day', row['day'])
    codes_set(ibufr, 'hour', row['ObsHour'])
    codes_set(ibufr, 'minute', 0)
    
    stationLat=dfStationInfo["LATITUDE"].values[0]
    stationLon=dfStationInfo["LONGITUDE"].values[0]
    stationAltitude=dfStationInfo["ALTITUDE"].values[0]
    
    codes_set(ibufr, 'latitude', stationLat)
    codes_set(ibufr, 'longitude', stationLon)
    codes_set(ibufr, 'heightOfStationGroundAboveMeanSeaLevel', stationAltitude)
    altitude = float(stationAltitude) + 1.5
    codes_set(ibufr, 'heightOfBarometerAboveMeanSeaLevel', altitude)
    
    codes_set(ibufr, 'surfaceQualifierForTemperatureData', CODES_MISSING_LONG)
    codes_set(ibufr, 'mainPresentWeatherDetectingSystem', CODES_MISSING_LONG)
    codes_set(ibufr, 'supplementaryPresentWeatherSensor', CODES_MISSING_LONG)
    codes_set(ibufr, 'visibilityMeasurementSystem', CODES_MISSING_LONG)
    codes_set(ibufr, 'cloudDetectionSystem', CODES_MISSING_LONG)
    codes_set(ibufr, 'lightningDetectionSensorType', CODES_MISSING_LONG)
    codes_set(ibufr, 'skyConditionAlgorithmType', CODES_MISSING_LONG)
    codes_set(ibufr, 'capabilityToDetectPrecipitationPhenomena', CODES_MISSING_LONG)
    codes_set(ibufr, 'capabilityToDetectOtherWeatherPhenomena', CODES_MISSING_LONG)
    codes_set(ibufr, 'capabilityToDetectObscuration', CODES_MISSING_LONG)
    codes_set(ibufr, 'capabilityToDiscriminateLightningStrikes', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#nonCoordinatePressure', CODES_MISSING_DOUBLE)
    pressure=row["PresInst"]
  
    pressure = float(pressure) * 100
    codes_set(ibufr, 'pressureReducedToMeanSeaLevel', pressure)
    codes_set(ibufr, '3HourPressureChange', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'characteristicOfPressureTendency', CODES_MISSING_LONG)
    codes_set(ibufr, 'pressure', pressure)
    codes_set(ibufr, 'nonCoordinateGeopotentialHeight', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#1#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
    airTInst=row["airTinst"]
    
    if airTInst!=CODES_MISSING_DOUBLE:
        temperature = airTInst + 273.15
    else:
        temperature=airTInst
    codes_set(ibufr, '#1#airTemperature', temperature)
    dewPInst=row["dewPInst"]
    
    dewPInst=dewPInst+273.15
    codes_set(ibufr, 'dewpointTemperature', dewPInst)
    relHinst=row["relHinst"]
    
    codes_set(ibufr, '#1#relativeHumidity', relHinst)
    codes_set(ibufr, '#1#depthBelowLandSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#1#soilTemperature', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#depthBelowLandSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#soilTemperature', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#3#depthBelowLandSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#3#soilTemperature', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#4#depthBelowLandSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#4#soilTemperature', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#5#depthBelowLandSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#5#soilTemperature', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#6#depthBelowLandSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#1#attributeOfFollowingValue', CODES_MISSING_LONG)
    
    visib=row["Visib"]
    codes_set(ibufr, 'horizontalVisibility', visib)
    codes_set(ibufr, '#3#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#3#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'iceDepositThickness', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'rateOfIceAccretionEstimated', CODES_MISSING_LONG)
    codes_set(ibufr, 'methodOfWaterTemperatureAndOrOrSalinityMeasurement', CODES_MISSING_LONG)
    codes_set(ibufr, 'oceanographicWaterTemperature', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'wavesDirection', CODES_MISSING_LONG)
    codes_set(ibufr, 'periodOfWaves', CODES_MISSING_LONG)
    codes_set(ibufr, 'heightOfWaves', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'methodOfStateOfGroundMeasurement', CODES_MISSING_LONG)
    codes_set(ibufr, 'stateOfGround', CODES_MISSING_LONG)
    codes_set(ibufr, 'methodOfSnowDepthMeasurement', CODES_MISSING_LONG)
    codes_set(ibufr, 'totalSnowDepth', CODES_MISSING_DOUBLE)
    cloudCoverTot=row["CloudCoverTot"]
    codes_set(ibufr, 'cloudCoverTotal', cloudCoverTot)
    codes_set(ibufr, '#1#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#cloudAmount', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#cloudType', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#attributeOfFollowingValue', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#cloudAmount', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#cloudType', CODES_MISSING_LONG)
    codes_set(ibufr, '#3#attributeOfFollowingValue', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#3#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
    codes_set(ibufr, '#3#cloudAmount', CODES_MISSING_LONG)
    codes_set(ibufr, '#3#cloudType', CODES_MISSING_LONG)
    codes_set(ibufr, '#4#attributeOfFollowingValue', CODES_MISSING_LONG)
    codes_set(ibufr, '#3#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#4#verticalSignificanceSurfaceObservations', CODES_MISSING_LONG)
    codes_set(ibufr, '#4#cloudAmount', CODES_MISSING_LONG)
    codes_set(ibufr, '#4#cloudType', CODES_MISSING_LONG)
    codes_set(ibufr, '#5#attributeOfFollowingValue', CODES_MISSING_LONG)
    codes_set(ibufr, '#4#heightOfBaseOfCloud', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'presentWeather', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'pastWeather1', CODES_MISSING_LONG)
    codes_set(ibufr, 'pastWeather2', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#timeSignificance', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'precipitationIntensityHighAccuracy', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'sizeOfPrecipitatingElement', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#timeSignificance', CODES_MISSING_LONG)
    codes_set(ibufr, '#3#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'precipitationType', CODES_MISSING_LONG)
    codes_set(ibufr, 'characterOfPrecipitation', CODES_MISSING_LONG)
    codes_set(ibufr, 'durationOfPrecipitation', CODES_MISSING_LONG)
    codes_set(ibufr, 'otherWeatherPhenomena', CODES_MISSING_LONG)
    codes_set(ibufr, 'intensityOfPhenomena', CODES_MISSING_LONG)
    codes_set(ibufr, 'obscuration', CODES_MISSING_LONG)
    codes_set(ibufr, 'characterOfObscuration', CODES_MISSING_LONG)
    codes_set(ibufr, '#4#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#4#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#3#timeSignificance', CODES_MISSING_LONG)
    codes_set(ibufr, '#4#timePeriod', CODES_MISSING_LONG)

    windDir=row["WindDir"]
    codes_set(ibufr, '#1#windDirection', float(windDir))
    
    windSpeed=row['WindSpeed']
    codes_set(ibufr, '#1#windSpeed', float(windSpeed))
    
    codes_set(ibufr, '#4#timeSignificance', CODES_MISSING_LONG)
    codes_set(ibufr, '#5#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#maximumWindGustDirection', CODES_MISSING_LONG)
    
    windGust=row["WindGust"]
    codes_set(ibufr, '#1#maximumWindGustSpeed', windGust)
    codes_set(ibufr, '#6#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#maximumWindGustDirection', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#maximumWindGustSpeed', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#7#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'extremeCounterclockwiseWindDirectionOfAVariableWind', CODES_MISSING_LONG)
    codes_set(ibufr, 'extremeClockwiseWindDirectionOfAVariableWind', CODES_MISSING_LONG)
    codes_set(ibufr, '#5#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#5#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#8#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'maximumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#1#minimumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#6#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#9#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#minimumTemperatureAtHeightAndOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#6#heightOfSensorAboveWaterSurface', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#7#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'methodOfPrecipitationMeasurement', CODES_MISSING_LONG)
    codes_set(ibufr, 'methodOfLiquidContentMeasurementOfPrecipitation', CODES_MISSING_LONG)
    codes_set(ibufr, '#10#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'totalPrecipitationOrTotalWaterEquivalent', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#8#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'methodOfEvaporationMeasurement', CODES_MISSING_LONG)
    codes_set(ibufr, '#11#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'evaporation', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#12#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'totalSunshine', CODES_MISSING_LONG)
    codes_set(ibufr, '#13#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'longWaveRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'shortWaveRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'netRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'globalSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'diffuseSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, 'directSolarRadiationIntegratedOverPeriodSpecified', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#14#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, 'numberOfFlashesThunderstorm', CODES_MISSING_LONG)
    codes_set(ibufr, '#15#timePeriod', CODES_MISSING_LONG)
    codes_set(ibufr, '#1#firstOrderStatistics', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#nonCoordinatePressure', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#windDirection', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#windSpeed', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#airTemperature', CODES_MISSING_DOUBLE)
    codes_set(ibufr, '#2#relativeHumidity', CODES_MISSING_LONG)
    codes_set(ibufr, '#2#firstOrderStatistics', CODES_MISSING_LONG)
    codes_set(ibufr, 'qualityInformationAwsData', CODES_MISSING_LONG)
    codes_set(ibufr, 'internalMeasurementStatusInformationAws', CODES_MISSING_LONG)
    #Encode the keys back in the data section
    codes_set(ibufr, 'pack', 1)

def main():
    '''
    main program reads the command line and encodes the messages into the output filename
        to run the program 
        prog  -a Ascii_input_file  -b Bufr_output_file -w wigosIDs_csvfile -l logFileName
    '''
    
    cmdLine = read_cmdline()
    inputFilename = cmdLine.ascii 
    outFilename = cmdLine.bufr 
    stationFile=cmdLine.wigoscodes
    logFile=cmdLine.logging 
    logging.basicConfig(filename=logFile,format="%(asctime)s %(levelname)s:%(message)s ",level=logging.DEBUG)
    logging.info("Using ecCodes version {0}".format(codes_get_api_version()))
    fout = open(outFilename, 'w')
    dfEstacoes=read_estacoes_data(stationFile)
    message_encoding(inputFilename, fout,dfEstacoes)
    fout.close()
    logging.info(' output file {0}'.format(outFilename))
    
if __name__ == '__main__':
    main()

A quality control of the observations is needed to avoid encoding wrong values in the output BUFR.

Code Block

#!/usr/bin/python
'''
# Copyright 2005-2018 ECMWF.
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction

This is a test program to encode Wigos Synop 
requires 

1) ecCodes version 2.8 or above (available at https://confluence.ecmwf.int/display/ECC/Releases)
2) python2.7

To run the program

   ./test_wigos_NEW.py -a Ascii_csv_file  -b output_Bufr -w WIGOS_TABLE_FILE -l logFile
   
Uses BUFR version 4 template  301150 307091

Author : Roberto Ribas Garcia ECMWF 27 Sep 2018
Modifications
Author : Roberto Ribas Garcia ECMWF 7 Nov 2018
         added new command line option to read the station codes
         changed logic to read station codes and update bufr accordingly
         added logging information 
         Roberto Ribas Garcia, ECMWF 24 Jan 2019 changed the encoding function
         to have one header. 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!-------------------------------------- CAVEATS ---------------------------
!!!!!!!!!!!!!!!!!!!!!!!!! A QUALITY CHECK IMPLEMENTATION IS NEEDED
!!!!!!!!!!!!!!!!!!!!!!!!! IT MAY BE INCOMPLETE CHECK AND TEST CAREFULLY
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
'''
import sys
#sys.path.append('/usr/local/lib64/python2.7/site-packages/')
# adapt the path to your installation
sys.path.append("/usr/local/apps/eccodes/2.10.0/GNU/6.3.0/lib/python2.7/site-packages/")

import argparse 
#import ipdb
import pandas as pd
import numpy as np
import logging 
from datetime import datetime
from eccodes import (codes_set, codes_set_array, codes_bufr_new_from_samples,codes_write,codes_get_api_version, 
                     codes_release, CodesInternalError,CODES_MISSING_DOUBLE,CODES_MISSING_LONG) 

    
def read_cmdline():
    '''
    reads the command line to get the input ascii filename and the output bufr file
        usage 
            prog  -a Ascii_input_file  -b Bufr_output_file -l logfile -w WIGOS_TABLE_FILE
    '''
    p = argparse.ArgumentParser()
    p.add_argument('-a', '--ascii', help = 'input Ascii filename')
    p.add_argument('-b', '--bufr', help = 'output Bufr filename')
    p.add_argument("-w", "--wigoscodes", help = "csv with the station codes")
    p.add_argument("-l","--logging", help = "log file")
    args = p.parse_args()
    return args



                

def bufr_encode_new(dfMeteo,dfEst,fout):
    '''
    encodes the bufr file with one BUFR header and loops over the 
    the dfMeteo dataframe. 
    
    dfEst    : is the station WiGOS info read from the WIGOS_TABLE_FILE
    dfMeteo  : contains the meteo information 
    fout     : is an file object opened for writing in the main program 
    
    '''
    
    ibufr = codes_bufr_new_from_samples('BUFR4')
   
    nSubsets=len(dfMeteo.index)
    oneSubsetDelayed=np.array([0, 0, 0, 0, 1, 1, 0, 0, 1, 1,1, 1, 0, 0,],dtype=np.int)
    DelayedDesc=np.tile(oneSubsetDelayed,nSubsets)
    ### this DelayedDesc array is a replication of the oneSubsetDelayed array as many
    ### times as subsets are present in the message. The number of subsets is found
    ### as the number of lines of the dfMeteo dataframe
    ### this is used to populate the key inputShortDelayedDescriptorReplicationFactor 
    codes_set_array(ibufr, 'inputShortDelayedDescriptorReplicationFactor', DelayedDesc)
    codes_set(ibufr, 'edition', 4)
    codes_set(ibufr, 'masterTableNumber', 0)
    codes_set(ibufr, 'bufrHeaderCentre', 43)
    codes_set(ibufr, 'bufrHeaderSubCentre', 0)
    codes_set(ibufr, 'updateSequenceNumber', 0)
    codes_set(ibufr, 'dataCategory', 0)
    codes_set(ibufr, 'internationalDataSubCategory', 7)
    codes_set(ibufr, 'dataSubCategory', 7)
    codes_set(ibufr, 'masterTablesVersionNumber', 28)
    codes_set(ibufr, 'localTablesVersionNumber', 0)
    ### check if this suits your system
    procTime=datetime.now()
    codes_set(ibufr, 'typicalYear', procTime.year)
    codes_set(ibufr, 'typicalMonth', procTime.hour)
    codes_set(ibufr, 'typicalDay', procTime.day)
    codes_set(ibufr, 'typicalHour', procTime.hour)
    codes_set(ibufr, 'typicalMinute', procTime.minute)
    codes_set(ibufr, 'typicalSecond', procTime.second)
    codes_set(ibufr, 'numberOfSubsets', nSubsets)
    codes_set(ibufr, 'observedData', 1)
    codes_set(ibufr, 'compressedData', 0)
    ivalues = ( 301150, 307091,)
    # Create the structure of the data section
    codes_set_array(ibufr, 'unexpandedDescriptors', ivalues)
    ## these are labels to update the different keys  MAY NEED MORE LABELS
    lblHOS1=1
    lblHOS2=1
    lblHOSMP=1
    lblTS=1
    lblTP=1
    ### loops over the dfMeteo 
    nValidMsg=0
    nWrongMsg=0
    for imsg,row in dfMeteo.iterrows():
        stationID=row["station"].strip()
        dfEstInfo=dfEst[dfEst["CODIGO"]==stationID]
        if not dfEstInfo.empty:
            nValidMsg+=1
            key="#{0}#wigosIdentifierSeries".format(imsg+1)
          
            codes_set(ibufr, key,0 )
            key="#{0}#wigosIssuerOfIdentifier".format(imsg+1)
            codes_set(ibufr, key, 76)
            key="#{0}#wigosIssueNumber".format(imsg+1)
            codes_set(ibufr, key, 0)
            
            wigosLocalID=dfEstInfo["WIGOS"].values[0]
            WigosCode=dfEstInfo["WIGOS"].values[0]
            wigosIdChar=WigosCode.split("-")[-1]
            key='#{0}#wigosLocalIdentifierCharacter'.format(imsg+1)
            codes_set(ibufr,key ,wigosIdChar)
            
            key="#{0}#stateIdentifier".format(imsg+1)
            codes_set(ibufr, key, CODES_MISSING_LONG)
            key='#{0}#nationalStationNumber'.format(imsg+1)
            codes_set(ibufr,key , CODES_MISSING_LONG)
            block=dfEstInfo["BLOCK"].values[0]
            key='#{0}#blockNumber'.format(imsg+1)
            codes_set(ibufr, key, block)
            stationNumber=dfEstInfo["STATION"].values[0]
            key='#{0}#stationNumber'.format(imsg+1)
            codes_set(ibufr, key, stationNumber)
            stationName=dfEstInfo["NOME"].values[0]
            key='#{0}#stationOrSiteName'.format(imsg+1)
            codes_set(ibufr,key ,stationName)
            key='#{0}#stationType'.format(imsg+1)
            codes_set(ibufr,key , CODES_MISSING_LONG)
            
            key="#{0}#year".format(imsg+1)
            codes_set(ibufr,key , row["year"])
            key='#{0}#month'.format(imsg+1)
            codes_set(ibufr, key, row["month"])
            key="#{0}#day".format(imsg+1)
            codes_set(ibufr, key, row["day"])
            key="#{0}#hour".format(imsg+1)
            codes_set(ibufr, key, row["ObsHour"])
            key="#{0}#minute".format(imsg+1)
            codes_set(ibufr, key, 0)
            
            stationLat=dfEstInfo["LATITUDE"].values[0]
            stationLon=dfEstInfo["LONGITUDE"].values[0]
            stationAltitude=dfEstInfo["ALTITUDE"].values[0]
            key="#{0}#latitude".format(imsg+1)
            codes_set(ibufr,key,stationLat)
            key="#{0}#longitude".format(imsg+1)
            codes_set(ibufr,key,stationLon)
            key="#{0}#heightOfStationGroundAboveMeanSeaLevel".format(lblHOS1)
            codes_set(ibufr,key,stationAltitude)
            lblHOS1+=1
            altitude = float(stationAltitude) + 1.5
            key="#{0}#heightOfBarometerAboveMeanSeaLevel".format(lblHOS2)
            codes_set(ibufr,key,altitude)
            lblHOS2+=1
            key="#{0}#surfaceQualifierForTemperatureData".format(imsg+1)
            codes_set(ibufr,key,3)
            key='#{0}#mainPresentWeatherDetectingSystem'.format(imsg+1)
            codes_set(ibufr, key, CODES_MISSING_LONG)
            key='#{0}#supplementaryPresentWeatherSensor'.format(imsg+1)
            codes_set(ibufr, key, CODES_MISSING_LONG)
            key='#{0}#visibilityMeasurementSystem'.format(imsg+1)
            codes_set(ibufr,key , CODES_MISSING_LONG)
            key='#{0}#cloudDetectionSystem'.format(imsg+1)
            codes_set(ibufr,key, CODES_MISSING_LONG)
            key='#{0}#lightningDetectionSensorType'.format(imsg+1)
            codes_set(ibufr,key , CODES_MISSING_LONG)
            key='#{0}#skyConditionAlgorithmType'.format(imsg+1)
            codes_set(ibufr,key , CODES_MISSING_LONG)
            key='#{0}#capabilityToDetectPrecipitationPhenomena'.format(imsg+1)
            codes_set(ibufr,key , CODES_MISSING_LONG)
            key='#{0}#capabilityToDetectOtherWeatherPhenomena'.format(imsg+1)
            codes_set(ibufr,key , CODES_MISSING_LONG)
            key='#{0}#capabilityToDetectObscuration'.format(imsg+1)
            codes_set(ibufr,key , CODES_MISSING_LONG)
            key='#{0}#capabilityToDiscriminateLightningStrikes'.format(imsg+1)
            codes_set(ibufr, key, CODES_MISSING_LONG)
            key="#{0}#nonCoordinatePressure".format(imsg+1)
            codes_set(ibufr,key,CODES_MISSING_DOUBLE)
            pressure=row["PresInst"]
            pressure = float(pressure) * 100
            key="#{0}#pressureReducedToMeanSeaLevel".format(imsg+1)
            codes_set(ibufr,key,pressure)
            key="#{0}#3HourPressureChange".format(imsg+1)
            codes_set(ibufr,key,CODES_MISSING_DOUBLE)
            key="#{0}#characteristicOfPressureTendency".format(imsg+1)
            codes_set(ibufr,key,2)
            key="#{0}#pressure".format(imsg+1)
            codes_set(ibufr, key, CODES_MISSING_DOUBLE)
            key='#{0}#nonCoordinateGeopotentialHeight'.format(imsg+1)
            codes_set(ibufr, key, CODES_MISSING_LONG)
            key='#{0}#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform'.format(lblHOSMP)
            codes_set(ibufr,key ,CODES_MISSING_DOUBLE)
            lblHOSMP+=1
            airTInst=row["airTinst"]
            if airTInst!=CODES_MISSING_DOUBLE:
                temperature = airTInst + 273.15
            else:
                temperature=airTInst
            key="#{0}#airTemperature".format(imsg+1)
            codes_set(ibufr,key,temperature)
            dewPInst=row["dewPInst"]
            dewPInst=dewPInst+273.15
            key="#{0}#dewpointTemperature".format(imsg+1)
            codes_set(ibufr,key,dewPInst)
            relHinst=row["relHinst"]
            key="#{0}#relativeHumidity".format(imsg+1)
            codes_set(ibufr,key , relHinst)
            key="#{0}#timeSignificance".format(lblTS)
            codes_set(ibufr, key, 2)
            lblTS+=1
            key="#{0}#timePeriod".format(lblTP)
            codes_set(ibufr, key, -10)
            lblTP+=1
            windDir=row["WindDir"]
            windSpeed=row['WindSpeed']
            key="#{0}#windDirection".format(imsg+1)
            codes_set(ibufr,key, windDir)
            key='#{0}#windSpeed'.format(imsg+1)
            codes_set(ibufr,key , windSpeed)
            
            key="#{0}#heightOfSensorAboveLocalGroundOrDeckOfMarinePlatform".format(lblHOSMP)
            codes_set(ibufr,key,10)
            
            key="#{0}#timePeriod".format(lblTP)
            codes_set(ibufr,key,-10)
            lblTP+=1
            key="#{0}#timePeriod".format(lblTP)
            codes_set(ibufr,key,-60)
            key='#{0}#totalPrecipitationOrTotalWaterEquivalent'.format(imsg+1)
            precip=row["Precip"]
            codes_set(ibufr,key , precip)
            
            key="#{0}#netRadiationIntegratedOverPeriodSpecified".format(imsg+1)
            rad=row["Rad"]
            codes_set(ibufr,key,rad)
        else:
            logging.info("Station {0} not found in the WIGOS TABLE FILE".format(stationID))
            nWrongMsg+=1
            

#######################################################             
###### at the end pack, write and release ibufr
    codes_set(ibufr, 'pack', 1)
    codes_write(ibufr, fout)
    codes_release(ibufr)
    return nValidMsg,nWrongMsg

    

def read_ascii(inputFilename):
    '''
    function to read the Ascii data into a pandas dataframe, 
        args:
            inputFilename : full path of the Ascii file for example /tmp/data/rema_20180918.txt
     
    uses white spaces as column delimiters
    index_col = False avoids using the first column (Station) as index
    names is the list of names from the excel it can be changed but this affects the dataframe
    returns the pandas dataframe dfM (for Measurement)
    '''
    dfM = pd.read_csv(inputFilename,
                     header = None, 
                     index_col = False,
                     delim_whitespace = True,
                     names = ['station', 'year', 'month', 'day', 'ObsHour', 'TensBat', 'TempCpu', 'airTinst',
                              'airTmax', 'airTmin', 'relHinst', 'relHmax', 'relHmin', 'dewPInst', 'dewPmax',
                              'dewPmin', 'PresInst', 'PresMax', 'PresMin', 'WindSpeed', 'WindDir', 'WindGust',
                              'Rad', 'Precip', 'CloudCoverTot', 'CloudCODE', 'CloudBase', 'Visib'])
    cols = dfM.columns.drop('station')
    dfM[cols] = dfM[cols].apply(pd.to_numeric, errors = 'coerce')
    return dfM


# column  1 = station    column 2 = year     column  3 = month     column 4 = day       column  5 = ObsHour  column  6 = TensBat  column 7 = TempCpu
# column  8 = airTinst   column 9 = airTmax  column 10 = airTmin  column 11 = relHinst  column 12 = relHmax  column 13 = relHmin
# column 14 = dewPInst  column 15 = dewPmax  column 16 = dewPmin  column 17 = PresInst  column 18 = PresMax  column 19 = PresMin
# column 20 = WindSpeed column 21 = WindDir  column 22 = WindGust column 23 = Rad       column 24 = Precip   column 25 = CloudCoverTot
# column 26 = CloudCODE column 27 = CloudBase column 28 = Visib
#
#     
# column 1 = station  column 2 = Name  column 3 = Latitude  column  4 = Longitude  column 5 = Elevation  column 6 = UF
# column 7 = Region   column 8 = Date  column 9 = ID_WIGOS  column 10 = status





def read_estacoes_data(StationCodeFile):
    '''
    reads the Stations Code File ( with information such as name,lat, lon,alt, WigosId etc)
    the name is read through the command line argument -c
    '''
    dfEst = pd.read_csv(StationCodeFile,
                        header = 0,
                        index_col = False)
    return dfEst



def main():
    '''
    main program reads the command line and encodes the messages into the output filename
        to run the program 
            program_name.py -a Ascii_input_file -b Bufr_output_file -w WIGOS_TABLE_FILE -l logFile 
    '''
    cmdLine = read_cmdline()
    inputFilename = cmdLine.ascii
##### changed this to make it work, change back according to your requirements
    # FileHeader
    #data_hora = inputFilename.split('_')
    #data_hora = data_hora[1].split('.aut')
    #data_hora = data_hora[0]
    #data_hora = list(data_hora)
    #dia = data_hora[6]  + data_hora[7]
    #hora = data_hora[8]  + data_hora
    dia="20"
    hora="12"
    FileHeader = 'ISAI99 SBBR ' + dia + hora + '00\r\r\n'

    outFilename = cmdLine.bufr 
    stationFile=cmdLine.wigoscodes
    logFile = cmdLine.logging 
    logging.basicConfig(filename=logFile,format = "%(asctime)s %(levelname)s:%(message)s ",level=logging.DEBUG)
    logging.info("Using ecCodes version {0}".format(codes_get_api_version()))
    fout = open(outFilename, 'w')
    fout.write(FileHeader)
    
    dfEstacoes = read_estacoes_data(stationFile)

    dfMeteo=read_ascii(inputFilename)
  

    nValidMsg,nWrongMsg=bufr_encode_new(dfMeteo,dfEstacoes,fout)
                        
    fout.close()
    logging.info( "number of valid Messages {0}  wrong messages {1}".format(nValidMsg,nWrongMsg))
    logging.info(' output file {0}'.format(outFilename))
    
if __name__ == '__main__':
    main()

Changed the bufr structure at request from Brazil. Now, there is only one bufr header and the different stations appear as subsets of the message.

The new program is not complete. Some work is requiered on the user side to fully adapt to the bufr structure and perhaps simplify the bufr_encode_new function.

DISCLAIMER: This software is intended for testing purposes only. Feedback would be appreciated but technical support can only be given if our workload permits.

Page tree

Versions Compared

Old Version 8

New Version Current

Key

UPDATES