Earth sciences and the study of climate change require substantial computational power and cutting-edge networks. An example of this is the deployment of an end-to-end, ultra-large bandwidth connection between the Swiss National Supercomputing Centre (CSCS) in Lugano, which hosts the MeteoSwiss weather prediction model, and the European Centre for Medium-Range Weather Forecasts (ECMWF) in Bologna. This link spans three different domains: the two national research networks (GARR in Italy and Switch in Switzerland) and the pan-European GÉANT network interconnecting the two.
This interconnection is part of the SwissTwins project, an international collaboration to facilitate, among other things, access to ECMWF forecast products and meteorological archive for the training of Artificial Intelligence (AI) and Machine Learning (ML) models that run on the Alps HPC system. ECMWF has one of the largest meteorological archives in the world and also provides other datasets that are widely used for model training, like meteorological reanalysis. These models have shown great potential in forecasting atmospheric conditions and weather patterns, and a lot of research is aimed at applying these methods to climate scenarios. As part of this project, a dedicated Data Hypercube—a high-performance storage and data processing cluster—will be deployed at the centre in Bologna and connected to the CSCS Alps data centre infrastructure in Lugano. This cluster will support data access with low latency and the extraction of specific datasets to train AI and ML models to benefit the Swiss and international numerical weather and climate prediction community.
As mentioned by DR Oliver Fuhrer, the Head of the Numerical Prediction Department at MeteoSwiss: “Connecting the AI and ML capability of the Alps infrastructure with the unique data archive hosted at ECMWF in Bologna will open up research opportunities that nobody would have thought possible.”
The end-to-end, ultra-large bandwidth infrastructure will enable massive data transfers from ECMWF meteorological data archives to CSCS supercomputing facilities. Specifically, in its pilot phase, this initiative aims to demonstrate the possibility of transferring an entire ensemble forecast dataset in less than one hour between ECMWF and CSCS. In subsequent phases, multiple AI/ML models in the weather and climate domain will consume data from the Data Hypercube simultaneously, thus increasing the total aggregate throughput.
Thanks to the collaboration between GARR, Switch and GÉANT, the two centres are now interconnected by a state-of-the-art, ultra-large bandwidth dedicated connection that seamlessly traverses the three network domains. But there is more. In designing this link, the requirements included extremely high capacity and scalability and significant resilience, reliability, security, and direct user access to the optical infrastructure.
GARR interviewed Ahmed Benallegue (ECMWF), Sabrina Tomassini and Alessandro Inzerilli (GARR) to explain how the adopted technological solutions help meet such requirements.
A dedicated multi-domain service
The connection comprises two diverse and dedicated 100 Gbps links, independently managed by the three operators. New resources have been deployed to ensure the service’s security and reliability. At the ECMWF site, Lepida, GARR’s local partner in the Emilia Romagna region, has installed two new fibre links to access the backbone.
From there, the two optical light paths have been configured to reach GARR Points of Presence (PoPs) in Milan. Here, the Italian network connects with the GÉANT network, which carries the traffic to its PoP located at CERN in Geneva. From Geneva, Switch takes over and delivers the data to its final destination at CSCS in Lugano.
The implementation methods differ between the networks: GARR and GÉANT have chosen an optical domain approach, while Switch uses a Layer 2 Ethernet-over-MPLS service.
“This type of service needs a high level of collaboration and trust among operators and would not be possible outside the research network ecosystem,” commented Sabrina Tomassini. When it comes to implementation, management, and monitoring each network is responsible for its domain, therefore a high level of communication is necessary. National Research and Education Networks offer multi-domain services, transcending their domains to harmonise services across networks. This capability is unique to NRENs, as commercial operators are confined to their domains of competence.
Security and resilience
To ensure a high level of security and resilience, a deep technical analysis was conducted, and diversification, shelters, and robust pathways were employed to identify and address potential faults, thereby ensuring the service’s highest possible resilience.
In Bologna, as in Lugano
“This service is one of the first to leverage the capabilities of the new next-generation GARR-T network for optical infrastructure access,” stated Sabrina Tomassini. “The advantage is that the entire 100 Gbps channel capacity is wholly dedicated to the user, without statistical multiplexing.”
“Moreover, a peculiarity of this interconnection is that the sites are linked at the layer 2 level. This enables CSCS to have full visibility of the Data hypercube machines located at ECMWF as if they were located in its local network in Lugano. Also, this is a feature enabled by the new GARR-T network,” says Alessandro Inzerilli.
Looking to the future…
The project, currently planned for two years, serves as a pilot. If successful, it could lead to further developments, including the possibility of increasing capacity to 400 Gbps in the future. “One of the advantages of the optical link with GARR is that any potential upgrade to 400 Gbps could be performed relatively quickly with minimal technical effort from ECWMF,” commented Ahmed Benallgque (ECMWF).
This high-performance connection opens new perspectives for research in meteorology and scientific computing. It enables closer interaction between ECMWF and the Swiss numerical weather and climate prediction community. It facilitates the sharing of large volumes of data and potentially accelerates progress in climate modelling and weather forecasting.
Watch the presentation by Ahmed Benallegue (ECMWF) at the Workshop GARR 2024
This article was originally published in Italian in the magazine of the Italian NREN GARR (GARR News 30): https://www.garrnews.it/caffe-scientifico/tre-reti-al-servizio-della-terra
