Dafit – a new work flow oriented approach for time efficient data preparation, validation and flagging of time series data from environmental monitoring
|Ries, Ludwig Christian
|Fleischer, Andreas G.
|Standardized quality assurance according to UN/WMO Global Atmosphere Watch (GAW) data quality objectives is essential for a homogeneous high level of data quality in GAW world data centers. However data processing and data preparation often times is done individually and in a non-standardized way. As an additional problem interactive data validation can be a very time consuming step on a GAW measurement station. In front of this background a new approach for a software solution with a set of time efficient and standardized methods is proposed which also can be used as a tool for future standardization of data quality assurance on GAW measurement stations. In general in order to prepare measurement data a set of methods is required. In the framework of this program the methods are ranked in two groups: first simpler methods for managing structural changes and corrections in the time series data and second higher developed methods for assuring a correct time structure, graphical control and flagging non representative data and for the calculation of differently or higher aggregated mean values and statistical values. By fulfilling the auxiliary condition that data treatment at first has to be finished with application of methods of group one the user is practically free in finding and selecting its way for a solution. Once the user has found an ordered set of methods and parameters which is a sufficient solution for the preparation of the actual data, the solution can be stored as a set of parameters for the individual project. This enables the user for repeating the solution at another time to another set of time series data which are produced with the same data format from the same instrument. This structure is characteristic and useful for continuous environmental monitoring which produces a high amount of time series data with a constant data format. By saving all intermediate results in a hierarchical way with a fixed naming convention, in a later time the effect of all steps of the application of methods can be traced back in the workflow history. Because the proposed methodology separates methods and its parameters, it can be a good substitution to many situations in data preparation whereas up to now normally for each little change the program source would have to be changed and compiled. This gives an essential facilitation to the whole process of data preparation of time series data in the daily monitoring. The set of methods works with ascii data files, in case each line has a time stamp with own date and time. The program works on files with arbitrary minute data or higher aggregated time data and provides precision up to the second. Dafit runs on Windows XPWin 8.
|Proceedings of the 27th Conference on Environmental Informatics - Informatics for Environmental Protection, Sustainable Development and Risk Management
|Dafit – a new work flow oriented approach for time efficient data preparation, validation and flagging of time series data from environmental monitoring
|Environmental Management Information Systems