ODB API

Change history

0.14.0

• First beta version of new Python API, see examples in subdirectory src/api/  The intention is to provide an API conforming with PEP 249 Python Database API Specification v2.0 https://www.python.org/dev/peps/pep-0249/  The new API, unlike the old one, allows for writing files (using INSERT, see examples).

0.13.0

• Preliminary, beta version of new API for C/C++ and Fortran, see api subdirectory for examples. All comments, suggestions and ideas for improvement will be much appreciated!

• New SQL syntaxes:
  • CREATE TABLE for associating SQL tables with files and setting metadata of new files
  • 'SELECT ALL *' for reading heterogeneous files (different columns on different parts of a file), available via the new API (ODBQL)
  • INSERT for writing files, available via ODBQL
  • MATCH '(' expression_list ')' IN QUERY '(' select_statement ')'
• ECML, the embedded scripting language based on MARS language syntax, has been merged into ODB API
• ODB API has been integrated with MarsKit, which allows direct retrieval of data from MARS or ODB Server
• ODB Server's functionality for local files available as part of the library
• A tarball with ODB API and all its dependencies is available, see file odb_api_bundle-0.13.0-Source.tar.gz. To build MarsKit it is necessary to pass -DENABLE_GRIBPP=OFF to cmake

0.10.5

• Added file group.txt to source code's etc directory. The file is needed for unit testing. An up to date version of the file is in http://apps.ecmwf.int/odbgov/group/csv/ , it is also installed in /usr/local/apps/odb_api/codes on ECMWF systems, and is used by MARS client when archiving ODB data. ODB_API_CODES environment variable can be used to change default location.

• Added cmake variable EXTRA_ODB_MIGRATOR_LIBS. It can be used to pass additional libraries needed for building odb_migrator, like gomp when building with gfortran.
• eckit is not distributed with this version. It is available for download here: ecKit Releases

0.10.4

• Use /usr/local/apps/odb_api/codes/group.txt ( http://data-portal.ecmwf.int/odbgov/csv/Group/ ) instead of deprecated obsgroups.txt (MARS client support code),

• Fix odb_migrator for cases when SQL SELECT defining data to be converted contained  functions or expressions other than simple column names on the select list,
• When writing output files create subdirectories if present in path name,
• Put back changes reverted by accident in change 6389d281bfb (this regression was released with 0.10.3): ODB-75 (LIKE/RLIKE, atoi), ODB-85 (hex bitfields formatting), ODB-108 (WHERE 0).

0.10.3

• Improve memory consumption of odb split tool (ODB-121)

0.10.2

• Fix odb_migrator, it was crashing when invoked with no parameters (ODB-91)

• Fix AIX build issues.

0.10.1

• Updated to use eckit 0.6.0 and ecbuild 1.3.2
• Table in the SELECT's INTO clause can be defined with a CREATE TABLE statement (ODB-67)

• odb sql tool has new flags (ODB-85):

  [--binary]            Print bitfields in binary notation

  [--no_alignment]      Do not align columns

• Python API for legacy ODB added (documentation to be written)
• ...

0.9.32

• "ORDER BY 1" was not working correctly (ODB-49)
• oda2request tool broken in ODB API 0.9.31 (wrong default path to configuration) (ODB-70)
• CSV import tool was not working correctly for some bitfield columns (ODB-76)
• ODB_DOUBLE constant added to Fortran bindings (ODB-77)
• Python API for accessing old ODB databases (ODB-74)
• Added odb_migrator option to specify missing value indicators different than default ones (only for REAL and INTEGER currently, DOUBLE always the same as REAL, BITFIELD always 0) (ODB-68)

            Syntax: 
               -mdi <type1:MDI1,type2:MDI2,...>
            e.g.:
              -mdi REAL:2147483647,INTEGER:2147483647

0.9.31

• Fixed a bug introduced in 0.9.30, where operator >= was behaving like =.
• Now ORDER BY clause accepts also column numbers, for example: SELECT int(lat/10), int(lon/10), count( * ) ORDER BY 3 DESC
• Support for #include directive

0.9.30

• Public header oda.h renamed to odb_api.h
• Added functions for reading bitfields' metadata: odb_read_iterator_get_bitfield  and odb_select_iterator_get_bitfield to Fortran API
• Disabled again fix for codec short_real, (ODB-15). The new codec short_real2 will be used by default when all ODB API applications at ECMWF are upgraded to use an ODB API version >= 0.9.24
• Added syntax for range expressions in SQL WHERE clause, e.g.: <value1> <=  <expression> <=  <value2>

0.9.29

• Fixed encoding of files containing columns of type DOUBLE (no packing) only.

0.9.28

• Improvements to the CSV import tool, now it can import bitfields
• Fixed queries with shift operator (#) appearing as arguments of functions
• Fixed option -T (do not print header with column names) of the sql tool

0.9.27

• Upgraded to eclib 7.3.0 - now wrapped in namespace eclib
• Fixed ODB-26: some expressions with mixed aggregated and non-aggregated functions, e.g.: sum(obsvalue)+rms(obsvalue), were calculated incorrectly

0.9.26

• Introduced in-memory data tables

0.9.25

•  Introduced row shift operator (#) (ODB-2)

0.9.24

• Fix an issue in SELECT DISTINCT ... INTO (some rows contained NULLs instead of correct values)
• Fix odb header tool so it works properly when string columns contain empty strings (ODB-19)
• Asynchronous IO can be used optionally when writing files by setting ODB_API_USE_AIO
• Fix the split tool (ODB-17)

• Added new codec short_real2 which should replace codec short_real in the future. NOTE: files created with this codec cannot be decoded with versions 0.9.23 and earlier. The new codec short_real2 is not used for encoding by default (ODB-15)

• CodecOptimizer can be configured by setting ODB_DEFAULT_CODEC

0.9.23

• Fixed a memory leak

•  Add options -offset and -length to the sql tool:

          [-offset <offset>]    Start processing file at a given offset

          [-length <length>]    Process only given number of bytes of data

• Add option -offsets to odb header tool, for printing basic information about blocks of data on a file:

          offset length numberOfRows

• Fix for ODB-8: Missing values were not handled correctly by SQL formulae if results of SELECT saved on file
• Semantics of SQL multiplication was different than in the old ODB: now 0 * NULL == 0, not NULL (ODBSUP-13)
• Result of most SQL functions is now double precision (ODB-11)
• Improved odb compare tool
• Added verification to the split tool