Model For GCOS, GOOS, and GTOS End-to-End Data Management Systems

Facilitating Access to Global Observing Systems Data and Information

The figure below is a model for the Global Observing Systems (GCOS, GOOS and GTOS) end-to-end data management systems. It was developed by the Joint Data and Information Management Panel (JDIMP). Each box represents a function. Click the boxes for further information on the functions. The functions in red are the responsibility of the science panels developing the observing systems. The functions in blue are the responsibilities of the developers and managers of the data management systems.

Global Observing System end-to-end Data management System

Services to Users

Although not shown in the figure, services to users will be provided from all the boxes on the diagram, both red and blue. The Science Policy function will provide guidance and coordination to national and international activities seeking to address the problems created by global and regional change. The science development bodies define and coordinate the data collection activities to satisfy the requirements for data and derived information and to identify the relevant "best" historical datasets for users.

All the blue boxes will provide data and information as appropriate at various time scales to a variety of users. The users will include national and international researchers, operational agencies, engineers, economists, those studying social impacts, managers of the observing systems, etc. The blue boxes will not provide interpretations, assessments, or predictions but rather the data and information from which others can do so.

The Science Policy Function

The science policy function for purposes of the Global Observing Systems (GCOS, GOOS and GTOS) are at the intergovernmental level and are the responsibilities of bodies such as SCOR, IOC, WMO, ICSU, etc. These bodies identify the need for and sponsor cooperative international science programs such as the WCRP, WOCE, and IGBP. Bodies such as these have also specified the need for and are sponsoring operational data collection programs to meet the needs of present and future international scientific and operational programs. Specifically this function has defined the need for the GCOS, GOOS, and GTOS.

National Collection Centers

Data for international programs are collected by national activities. The collection center function for the Global Observing Systems (GCOS, GOOS and GTOS) is generally a national activity. The data assembly center collects together specific data and metadata from national activities for one or more datasets identified by the scientific and technical panels as required for global observing systems purposes. Generally speaking, a nation will collect data to a standard that meets its national needs. This national standard will not necessarily meet the needs of a program such as the GOOS, GCOS, or GTOS. For example, for detection of climate change it may be necessary to have a period of simultaneous operation for calibration purposes between old and new sensors at a station. Additional meta data. may also be required. Thus for the global observing systems, the national collection center may be asked to undertake special processing or handling for global observing systems data.

In any case, the national collection center will have certain responsibilities for assembling national data and meta data, for processing the data in a certain way, and for forwarding the data to one or more international centers for further assembly, processing, distribution, and archiving according to the details of the end-to-end data management plan for the variables in question.

One country may have several collection centers for the various components of the Global Observing Systems (GCOS, GOOS and GTOS), and for the different variables being collected. However, it is not necessary for each country to have its own collection centers. It would make sense for, say, groups of developing or developed countries with common interests to share collection centers according to their capabilities and expertise.

The Global Observing Systems (GCOS, GOOS and GTOS) Operational or Research Centers

The next required function in the Global Observing Systems (GCOS, GOOS and GTOS) data management is the Global Observing Systems Operational or Research Center. This center takes on responsibilities for specific activities in support of the Global Observing Systems (GCOS, GOOS and GTOS). Examples include special processing functions, acquisition and quality control of data, generation of products, monitoring of data flows, or compilation and distribution of specific datasets or data products to selected users.

The needs for these centers are identified by science panels or data management groups and are based on the tasks required to prepare the datasets, generate the data products, and deliver them to the users. Data will generally be moved by electronic network between a number of such centers carrying out different steps in the processing. It is more effective to move the data to the place where the necessary expertise and capacity exists to perform a function on a dataset rather than to try and develop parallel capabilities at the site where the data are located. It is also a way of sharing the workload and not over-burdening a few centers.

Once the needs are identified in developing the end-to-end data management plan a center can volunteer for a task or the data management planners can solicit one with the necessary expertise. This helps to share the workload minimize the development of parallel data management expertise. It is easier to deal with an increased volume in data that are already being handled at a center, than to develop new software systems and handle data not seen before.

Operational centers will be expected to commit to a task for the medium to long term. If a center is carrying out an important step in the end-to-end management of a data flow, the whole flow will collapse if the center withdraws. Thus there would need to be adequate notice of termination of a service to allow time for an alternate center to be established.

The Science Development Function

The development of the science aspects of the international science programs and supporting global observing systems are accomplished through the vehicles of scientific steering groups (SSGs), steering committees (SCs), science panels, and groups of experts. These various groups are composed of nominated scientific experts in the fields relevant to the observation systems being developed or managed.

For the Global Observing Systems (GCOS, GOOS and GTOS) data management the groups that are defining the requirements for data and data management are the scientific and technical advisory panels. There are several such panels:

Atmospheric Observation Panel for Climate (AOPC)
Ocean Observation Panel for Climate (OOPC)
Terrestrial Observation Panel for Climate (TOPC)
The Coastal Ocean Observing Panel (COOP)

The science panels of the Global Observing Systems (GCOS, GOOS and GTOS) are responsible for defining the observations required in the future for the observing systems to achieve their goals, for identifying the historical data sets that are relevant and important, and for specifying the meta data to be collected and maintained with both the historical and new data. There are two types of metadata. The meta data referred to here is that required for a user to understand the limitations and circumstances of the data in order to use it properly. For example this type of metadata would include instrument type, method of analysis, and quality flags for the data points. The other type of meta data is the dataset metadata which would include such information as formats, holders of the data, and the media on which it can be made available.

The science panels are also responsible for specifying at least the necessary elements of the end-to-end data management systems to be developed to process, quality control, deliver the data to users, and provide for final archival of the data and metadata.

Final Archives

The purpose of the final archives is to preserve the data for future research and engineering users. The World Data Center (WDC) System is an example of such a final archive and much of the Global Observing Systems data will be archived there. In cases where a WDC does not exist for a particular data set, an alternate permanent archive must be found, usually in a government center that is prepared to accept the responsibilities.

The minimum responsibilities for a permanent archive would include most of the following

accept the data and all available supporting meta data
store it either in its original form or in a form from which all the original data and meta data can be recovered
refresh or update the medium on which the data and metadata are stored so that both are readable in the future, and
provide the data and all supporting metadata to users on request, free of charge or at the cost of reproduction.

There is an additional minimum requirement for the Global Observing Systems (GCOS, GOOS and GTOS) datasets.

To store the data so that it can be distinguished and recovered separately from similar data that are not the Global Observing Systems (GCOS, GOOS and GTOS) data and therefore have not necessarily had the same care taken in collection, quality control, and processing.

Final Archives should be identified for all the Global Observing Systems (GCOS, GOOS and GTOS) datasets in establishing the end-to-end data management systems.

Information Center

The Global Observing System Information Center (GOSIC) is operational at the NOAA National Climatic Data Center (NCDC) in Asheville, NC .

The Information Center provides on-line information about the collection, flow, and availability of the Global Observing Systems (GCOS, GOOS and GTOS) data. This information is intended for use by Global Observing Systems data collectors and data managers, by those evaluating the effectiveness and performance of the systems, and most importantly, by those who need the data for research, operations and engineering.

Evaluation and Feedback

As with the information center, this function applies to all aspects of the data collection and data flow up to final archival. Evaluation and feedback are an essential part of an end-to-end data management system. There are many aspects to evaluation of a system. Each element is required to perform one or more functions on the data and information and pass one or more deliverables on to the next step according to some schedule. Data and information flow monitoring will endeavor to ensure the tasks are completed and the flows are on schedule. Information about bottlenecks or delays will be fed back to managers and the centers involved for action to correct.

Data will be collected according to a "best practice" and evaluation of the quality of the data will occur at several points in the life cycle of a dataset. There will be a requirement to evaluate the performance of the quality control activities to ensure they are achieving the established goals. Information on problems, particularly recurring problems, will be fed back to facilitate resolution of the problems.

Included in the evaluation and feedback function is the monitoring of user satisfaction. In this context "user" includes the science bodies defining the requirements, the managers of the data collection and management systems, and the scientific, technical, and operational users of the datasets and products.

There will be a requirement to ask and answer the question "is the system working"? Evaluation and feedback are the means to answer that question. Scientific and technical advisory panels will define the evaluation and feedback mechanisms necessary to ensure "the system is working". In many cases centers will be asked to assist by implementing and operating some of the evaluation and feedback mechanisms in support of the Global Observing Systems (GCOS, GOOS and GTOS) data management.