The WIGOS Data Quality Monitoring System webtool (WDQMS) is the cornerstone of the World Meteorological Organization’s (WMO) observations monitoring. The system calculates the metrics certifying payments of the Systematic Observations Financing Facility (SOFF), assists Regional WIGOS Centers (RWC) with their quality monitoring functions, helps Numerical Weather Prediction (NWP) centers benchmarking their use of observations, and provides the data allowing WMO to calculate metrics about the evolution of its observing systems, for example the Global Basic Observing System (GBON). WDQMS has modues for surface, radiosonde, marine and climate observations and has started the migration from GTS to WIS 2.0.

But how did WDQMS come to be the system it is? This post examines the history of WDQMS from its beginnings as a concept, to its current state.

The beginnigs

The WIGOS Data Quality Monitoring (WDQMS) concept was developed in the Inter Commission Coordination Group on the WMO Integrated Global Observing System (WIGOS) starting from 2014, under the leadership of co-chairs Sue Barrel and Bertrand Calpini, and WIGOS project manager Lars Peter Riishoojgaard. The concept consisted of a monitoring function, the focus of this post, and an Incident Management System function and Regional WIGOS centers (RWC). The monitoring function for availability, quality and timeliness was designed to be based on output provided by NWP centers.

The NWP-focused approach reflected NWP centers’ unique perspective as principal users of observations exchanged in real-time. NWP centers are best suited to asses which observational data exchanged through the network is available, usable and in-time for assimilation in numerical weather prediction. While the webtool showing the results, is the most visible part of the system, the NWP centers providing data, the Experts responsible for the design of the data-exchange templates and the community interpreting the information play equally important roles in the WDQMS ecosystem.

The need for a quality monitorintg function was first identified by the Task Team on WIGOS Quality Management (TT-WQM) in 2014. The request to review and modernise the NWP-based monitoring within the WIGOS framework was addressed by the two workshops on Quality Monitoring and Incident Management held, in December 2014 and December 2015. This led to the creation of the Task Team on the WIGOS Data Quality Monitorint System (TT-WDQMS), which further developed the concept under the leadership of the late Stuart Goldstraw.

Multiple NWP centers routinely providing feedback on observations using a standardized template makes it possible to compare results and results in a global view independent of any potential biases in individual NWP centers. Initially the European Center for Medium-range Weather Forecasts (ECMWF) and the National Centers for Environmental Prediction (NCEP) agreed to take part in a pilot project for surface (SYNOP) and upper-air (TEMP) observation in 2015, with Deutscher Wetterdienst (DWD) and Japan Meteorological Agency (JMA) joining the Pilot Project in 2016.

WDQMS concept WDQMS data-flow, from the WDQMS User Manual

While the reported data was initially only used to generate ad-hoc plots, by 2017 a beta-version of a webbased tool, later to become synonymous with WDQMS, had been developed by the WMO Secretariat. It allowed to view SYNOP and TEMP observation aggregated by six-hour and daily intervalls and to see results for each center individually, as well as a combined view. The initial system was based on a single rented Linux Infomaniak server, running a PostGIS database, a Tomcat based Geoserver for generating the maps, first a PHP and later a Django backend, and a simple HTML and Jquery based web-frontend. Most of the ETL proccesses transforming the raw NWP data provided as CSV files into aggregated results ran directly in the database as complex SQL queries, generated by Python scripts. The initial choices of a map-centric system with colored dots representing stations, a calendar based navigation and popups for details have endured and are still recognizable in the current system.

WDQMS in 2017 WDQMS webtool beta-version display of the status of upper-air observations for 7 November 2017 at UTC 00 centered on the Netherlands.

Meanwhile, TT-WDQMS, supported by Luis Nunes at the WMO Secretariat, further developped the WDQMS concept by adding templates for other observation types, the yearly December meetings of TT-WDQMS becoming a fixture of the community. In 2017 Marine Quality and Aircraft templates were developpped, under the leadership of ECMWF’s Cristina Prates, and a framework for including non NWP-centered observation types discussed. The WDQMS webtool continued in beta-version.

Towards operationalization and GBON

In 2018, the 70th WMO Executive Council (EC) endorsed a WDQMS based quality monitoring function as a core WIGOS component, and took steps to include the WDQMS concept into the WIGOS regulatory material. At the same time, ECMWF and the WMO Secretariat started working on an agreement for ECMWF to take over the development and operations of the WDQMS webtool in its operational phase. A Memorandum of Understanding was signed in 2019.

Starting from 2018, and as a result of the WDQMS webtool putting into evidence the relatively poor state of international data exchange, the Commission for Basic Systems (CBS) Management Group, CBS-TECO, and later the WMO EC decided to take immediate action to improve the situation, eventually leading to the development of the WMO Global Basic Observing Network (GBON) and WMO’s new data-policy.

WDQMS in 2017 WDQMS webtool prototype visible in the WMO “video-wall” in 2017. The video-wall was intstrumental in creating momentum for revising WMO’s data-policy.

In the meantime the ECMWF web-development team, freshly reinforced by lead developer Pierre Vernier, was busy working on the production system. A complete re-write, the initial production system was based on a cloud-native infrastructure running in a Kubernetes cluster, with Django and Django REST framework as backend application server, Nginx load-balancers, a React based frontend and a PostGIS database as storage. Periodic ETL jobs aggregating the data were implemented as Django management commands and ran as Kubernetes cron-jobs in dedicated pods. Map-tiles were produced asynchronously by Django worker-nodes orchestrated by a Celery and RabbitMQ combination. This latter design choice, while in line with engineering principles in force at the time, was later simplified by introducing a dedicated tileserver based on pg_tileserv.

WDQMS in 2017 Inital WDQMS high-level architecture and stack (credit Amro Abouelenein)

The first production version of WDQMS hosted by ECMWF was launched on 17 March 2020, the system having been in pre-operational mode since 29 November 2019.

WDQMS in 2020 WDQMS webtool on 11 March 2020, shortly before its operational launch, showing surface data for all centers for 18 UTC.

New constellations

After the WMO Reform in 2020, TT-WDQMS, now chaired by Jeff Ator from NCEP, became part of the Standing Committee of Information Management and Technology (SC-IMT) under the new WMO Structure.

WDQMS continued to be enhanced in the 2020-21 timeframe. The most visible result of these efforts were the new Global Climate Observing System (GCOS) module, for GSN (Surface) and GUAN (Upper-air). But the team also implemented numerous smaller features, such as a download link, adding DWD and NCEP upper-air reporting, station search functions and an interface to provide the monthly assessed reporting status to OSCAR/Surface.

In 2022 and co-inciding with WDQMS lead-developer Pierre leaving ECMWF, WMO and ECMWF agreed a new Memorandum of Understanding whereby ECMWF would continue operating WDQMS, and WMO was going to take over responsiblity for the development of the system. Amro Abouelenein joined WMO as a full-stack developer in 2022.

Major releases since this time were the implementation of the GBON module, lauched in May 2023 just before the GBON regulations coming into force, monthly aggregations of SYNOP and TEMP, as well as the Marine Quality module. Other developments were support for stations reporting observations using multiple WIGOS Station Identifiers, aligning quality functions between the NWP and GBON modules, and functions in support of Regional WIGOS centers. The GBON compliance app was launched as a separate software, building on the newly released GBON module of WDQMS.

Based on the experiences of the handover, the development team started a phased approach to modernize WDQMS in 2023. On the one hand the handover process had shown a steep learning curve due to complexity of both code and development platform. On the other hand, WDQMS’s role of certifying SOFF payments called for a traceability, accountabilty and quality not only in observatinonal data but also in the algorithms aggregating them. After a review, the team found that while the overall Kubernetes platform and system architecture were still fit for purpose, there was a need to rationalize the map related components of the system architecture, to upgrade the development platform, and to introduce a software architecture to rationalize the numerous scripts processing the different data types that had been added since 2020. This resulted in a four phase refactoring process. First, to simplify the map tile production. Second, to modernize the development platform by leveraging docker compose instead of bespoke scripts. Third and forth, to introduce a software architecture and software development best practices and to modularize the code of data processing and aggregation algorithms. A comprehensive end-to-end integration testing, developed by the team with the help of Benjamin Pire, made sure that the refactored system still processed data as before. The details of this refactoring process will be described in a separate series of posts.

In 2024, the TT-WDQMS became the Expert Team on the Observations Data Quality Monitoring System (ET-ODQMS) and moved under the Standing Committee on WMO Integrated Processing and Prediction System (SC-WIPPS), for better alignment with other NWP center related activites. The Tokyo meeting of ET-ODQMS in October 2024 was the first time the team saw each other in person, having met only virtually since 2019. China Meteorological Administration (CMA) and The Bureau of Meteorology (BoM) nominated experts to the team and expressed their intention to become WDQMS monitoring centers in 2023 and 2024 respectively. At the Tokyo meeting, important decisions on a timeliness template definition, the harmonization of the legacy CBS-led observations quality monitoring and the WDQMS process as well as the formalization of WDQMS monitoring centers as WIPPS centers were taken.

ET-ODQMS-family-picture

Meeting of ET-ODQMS in Tokyo in October 2024

WDQMS, the future

The development roadmap of WDQMS forsees several new features and components to be released.

The release of timeliness, planned for June 2025, will complete the existing availability and quality monitoring categories in giving a detailed insight into how quickly observations reach NWP. With the release of timeliness, WDQMS will monitor all three vectors in terms of which requirements for Global NWP are defined. The implementation of timeliness, will also open up the possibility to complete the GBON module in a subsequent release, by adding timeliness and quality, which are currently not considered, to the GBON module.

To cover all WIGOS component observing systems, additional observation types and variables for the Aircraft Based Observing System (ABOS), Cryosphere and Global Atmosphere Watch (GAW), will have to be added to WDQMS. A data exchange template for snow-depth having been agreed at the 2024 Tokyo meeting, snow-depth is planned to be released once at least two centers can operationally provide the snow-depth variable. The snow-depth release is expected to happen in Q3 2025 and will likely be implemented by adding snow-depth as an additional variable in the NWP surface module. While a data exchange template for ABOS was already developed back in 2017, an implementation in the webtool will require significant conceptual work to visualize the the numerous in-flight and ascent/descent observations in this system. An ABOS module of WDQMS therefore not be released before 2026.

The implementation of WIS 2.0 by the WMO community will bringing old questions about the meaning of availability back to the foreground. Whereas availability based on observations available to NWP centers data assimilation provided for a relatively uncontested availability definition under the GTS with its multiple different vantage points and less observable design, WIS 2.0, featuring an open-design provides a alternative definition of availability, potentially different from the NWP center one, particularly during the transition time where not all WDQMS monitoring Centers have yet established an operational connection to WIS 2.0.

However, there are reasons to believe the choice of an NWP center centric and thus end-user based approach leaveraging NWP centers unique view of observations, will prove to be lasting. First, only NWP centers are able to provide a measure of the quality of observations, as they can compare actual observations with expected values from their models. Likewise, threy are also in a unqiue position to measure timeliness, since timeliness requirements for global NWP measure the time observations take to reach NWP. Second, only NWP centers have the technical means to remove duplicate observations in the network, through a mix of data-assimilation and model based deduplication. Third, as operational entities, NWP centers have unique capabilities to not only monitor data in theory, but doing the hard job of routinely providing CSV files to WDQMS four times per day, every day.