Processing Configuration

Basic configuration paramters, in particular directory paths, are stored in a configuration file used by both data extraction script and the XML interpreter.

The file is UTF-8 encoded and follows the MS INI format:

[RawData]
raw_data_path       = ../raw-data/
mask                = CR1000_*.dat
logger_time_zone    = UTC+1

[Metadata]
Project      = My project name
Web Page     = http://my-project.org/
File Content = Data acquired during My Project
Owner        = Myself
Contact      = me@my-email.com
Comment      = Data are provided without warranty of fitness for a particular purpose.

[Files]
data_path           = ./
xml_dtd_out         = data_map.dtd
raw_data            = consolidated_raw_data.npy
processed_data_xlsx = processed_data.xlsx
processed_data_npy  = processed_data.npy
xml_map_path        = ./
xml_map             = data_map.xml

RawData

The RawData section is used only by the data extraction tool.

  • raw_data_path specifies the path to the Campbell Scientific TOA5 raw data files generated by LoggerNet.
  • mask is the file name mast (glob) of the TOA5 raw data files that should be imported.
  • logger_time_zone is a string that is copied into the output file

Metadata

The Metadata section is used by the XML interpreter. The keys and values are copied into the output XLSX file ahead of the data table.

Files

The Files section is used by both raw data extraction tool and XML interpreter:

  • data_path specifies where intermediate and final files will be stored.
  • raw_data specifies the name of the file generated by the TOA5 raw data extraction tool.
  • processed_data_xlsx and processed_data_npy are the names of the files that store the result of XML interpreter.
  • xml_dtd_out names the file the XML interpreter stores the DTD in.
  • xml_map_path (optional) is the path to the XML Data File. If not specified, data_path is assumed.
  • xml_map is the input XML Data File for the XML interpreter.