Page History
...
| Info | ||
|---|---|---|
If the sample BUFR file cannot be found, please check the path used for samples (see the following command to be run for this, and example output):
|
Now we can use command line BUFR tools to see the content of TM307092.bufr:
| Code Block |
|---|
user@host:~> bufr_dump -p TM307092.bufr > TM307092.plain |
The above given command dumps the content of the BUFR file in plain (key=value) format. Please refer to bufr_dump for more information and different options to use bufr_dump tool. The output will be:
| Expand | ||
|---|---|---|
| ||
|
3. Preparing CSV file
TM307092.plain tells us what parameters (keys) we can encode with the template 307092. Now we will create a file containing our data which will be in CSV format. In this step, we need to consider both the content of 307092.plain and associated WMO tables to provide key-value pairs appropriately (units, scale, etc.). Please note that although CSV file is chosen as an example here, you may use any format to provide your data as input to your python program for BUFR encoding as long as you are able to make the program read your data.
| Code Block | ||
|---|---|---|
| ||
year|month|day|hour|minute|blockNum|stationNum|stationName|lat|lon|height|pressure|presMeanSeaLev|temp|relHum|totalPrep|windDir|windSpeed
2020|09|14|14|00|17|100|NameOfStation|41.55|28.47|300|100860|102730|301.25|43|0.1|62|1.7
2020|09|14|14|00|17|101|NameOfStation1|40.55|27.47|401|100850|102700|304.25|50|0.0|70|2 |
The SYNOP data provided above is created just to serve as an example for this tutorial. As you may realise, it provides values for a limited number of parameters. While you are creating your input files, you should include as much information as possible from the measurements of your station and change the below given csv2bufr.py accordingly to read from the correct column and write to the correct key.
4. BUFR encoding the data
| Code Block | ||||
|---|---|---|---|---|
| ||||
#!/usr/bin/env python
# (C) Copyright 1996- ECMWF.
#
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
#
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation
# nor does it submit to any jurisdiction.
from eccodes import *
import csv, argparse
# read command line to get input filename
def read_cmdline():
p = argparse.ArgumentParser()
p.add_argument("--i",help=" input Ascii filename")
args = p.parse_args()
return args
# read data from CSV file into a list
def csv_read(filename):
data = []
try:
with open(filename) as csvfile:
reader = csv.reader(csvfile, delimiter='|')
for row in reader:
data.append(row)
except IOError as error:
print(error)
sys.exit(1)
else:
csvfile.close()
return data[1:]
# Encode the data from CSV into BUFR
def message_encoding(FileName, fout):
# reads the CSV file into a python list
dataIn = csv_read(FileName)
# loops over the rows of the csv file (one BUFR message for each row)
for row in dataIn:
bid = codes_bufr_new_from_samples('BUFR4')
for ele in range(len(row)):
row[ele] = row[ele].strip()
try:
bufr_encode(bid, row)
codes_write(bid, fout)
except CodesInternalError as ec:
print ec
codes_release(bid)
def bufr_encode(ibufr, row):
# set header keys and values
codes_set(ibufr, 'edition', 4)
codes_set(ibufr, 'masterTableNumber', 0)
codes_set(ibufr, 'bufrHeaderCentre', 98) # 98: centre is ecmf
codes_set(ibufr, 'bufrHeaderSubCentre', 0)
codes_set(ibufr, 'updateSequenceNumber', 0)
codes_set(ibufr, 'dataCategory', 0) # 0: Surface data - land
codes_set(ibufr, 'internationalDataSubCategory', 7) # 7: n-min obs from AWS stations
codes_set(ibufr, 'dataSubCategory', 7)
codes_set(ibufr, 'masterTablesVersionNumber', 31)
codes_set(ibufr, 'localTablesVersionNumber', 0)
codes_set(ibufr, 'observedData', 1)
codes_set(ibufr, 'compressedData', 0)
codes_set(ibufr, 'typicalYear', int(row[0]))
codes_set(ibufr, 'typicalMonth', int(row[1]))
codes_set(ibufr, 'typicalDay', int(row[2]))
codes_set(ibufr, 'typicalHour', int(row[3]))
codes_set(ibufr, 'typicalMinute', int(row[4]))
codes_set(ibufr, 'typicalSecond', 0)
ivalues=(307092)
codes_set(ibufr, 'unexpandedDescriptors', ivalues)
# set data keys and values
codes_set(ibufr, 'year', int(row[0]))
codes_set(ibufr, 'month', int(row[1]))
codes_set(ibufr, 'day', int(row[2]))
codes_set(ibufr, 'hour', int(row[3]))
codes_set(ibufr, 'minute', int(row[4]))
codes_set(ibufr, 'blockNumber', int(row[5]))
codes_set(ibufr, 'stationNumber', int(row[6]))
codes_set(ibufr, 'longStationName',row[7].strip())
codes_set(ibufr, 'latitude', float(row[8]))
codes_set(ibufr, 'longitude', float(row[9]))
codes_set(ibufr, 'heightOfStationGroundAboveMeanSeaLevel', float(row[10]))
codes_set(ibufr, 'pressure', float(row[11]))
codes_set(ibufr, 'pressureReducedToMeanSeaLevel', float(row[12]))
codes_set(ibufr, 'airTemperature', float(row[13]))
codes_set(ibufr, '#1#relativeHumidity', float(row[14]))
codes_set(ibufr, '#2#timePeriod', -10) # -10: Period of precipitation observation is 10 minutes
codes_set(ibufr, 'totalPrecipitationOrTotalWaterEquivalent', float(row[15]))
codes_set(ibufr, '#1#timeSignificance', 2) # 2: Time averaged
codes_set(ibufr, '#3#timePeriod', -10) # -10: Period of wind observations is 10 minutes
codes_set(ibufr, 'windDirection', float(row[16]))
codes_set(ibufr, 'windSpeed', float(row[17]))
codes_set(ibufr, 'pack', 1) # Required to encode the keys back in the data section
def main():
cmdLine = read_cmdline()
inputFilename = cmdLine.i
print inputFilename
outFilename = str(inputFilename.split('.')[0]+'.bufr')
fout = open(outFilename, "w")
message_encoding(inputFilename, fout)
fout.close()
print " output file {0}".format(outFilename)
if __name__ == '__main__':
main() |
Please pay attention to the comments within the code and make changes accordingly. When the above given code is run, we expect to have datain.bufr created as:
| Code Block |
|---|
user@host:~> ./csv2bufr.py --i datain.csv
datain.csv
output file datain.bufr |
datain.bufr will have 2 BUFR messages since the input file has 2 rows and csv2bufr.py creates a bufr message for each row:
| Code Block |
|---|
user@host:~> bufr_ls datain.bufr
datain.bufr
centre masterTablesVersionNumber localTablesVersionNumber typicalDate typicalTime numberOfSubsets
ecmf 31 0 20200914 140000 1
ecmf 31 0 20200914 140000 1
2 of 2 messages in datain.bufr |
Related articles
| Content by Label | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...