Konferenzbeitrag
Dafit – a new work flow oriented approach for time efficient data preparation, validation and flagging of time series data from environmental monitoring
Vorschaubild nicht verfügbar
Volltext URI
Dokumententyp
Text/Conference Paper
Zusatzinformation
Datum
2013
Autor:innen
Zeitschriftentitel
ISSN der Zeitschrift
Bandtitel
Verlag
Shaker Verlag
Zusammenfassung
Standardized quality assurance according to UN/WMO Global Atmosphere Watch (GAW) data quality
objectives is essential for a homogeneous high level of data quality in GAW world data centers. However
data processing and data preparation often times is done individually and in a non-standardized way. As
an additional problem interactive data validation can be a very time consuming step on a GAW measurement
station.
In front of this background a new approach for a software solution with a set of time efficient and standardized
methods is proposed which also can be used as a tool for future standardization of data quality assurance
on GAW measurement stations.
In general in order to prepare measurement data a set of methods is required. In the framework of this program
the methods are ranked in two groups: first simpler methods for managing structural changes and
corrections in the time series data and second higher developed methods for assuring a correct time
structure, graphical control and flagging non representative data and for the calculation of differently or
higher aggregated mean values and statistical values.
By fulfilling the auxiliary condition that data treatment at first has to be finished with application of methods
of group one the user is practically free in finding and selecting its way for a solution. Once the user
has found an ordered set of methods and parameters which is a sufficient solution for the preparation of
the actual data, the solution can be stored as a set of parameters for the individual project. This enables the
user for repeating the solution at another time to another set of time series data which are produced with
the same data format from the same instrument. This structure is characteristic and useful for continuous
environmental monitoring which produces a high amount of time series data with a constant data format.
By saving all intermediate results in a hierarchical way with a fixed naming convention, in a later time the
effect of all steps of the application of methods can be traced back in the workflow history.
Because the proposed methodology separates methods and its parameters, it can be a good substitution to
many situations in data preparation whereas up to now normally for each little change the program source
would have to be changed and compiled. This gives an essential facilitation to the whole process of data
preparation of time series data in the daily monitoring. The set of methods works with ascii data files, in
case each line has a time stamp with own date and time. The program works on files with arbitrary minute
data or higher aggregated time data and provides precision up to the second. Dafit runs on Windows XPWin
8.