AI That Guides Researchers Through Complex Datasets

What if artificial intelligence could detect unusual behaviour or uncover trends in climate data usage before they become critical issues?

Imagine managers of the Earth System Grid Federation (ESGF) nodes, part of the ENES RI, overseeing millions of daily downloads. Detecting infrastructure issues or changes in download patterns in real time is challenging, and this can affect researchers across Europe relying on climate data. But what if AI could automatically flag anomalies and usage patterns, enabling proactive intervention?

Thanks to RI-SCALE’s Data Exploitation Platforms (DEPs), AI-based models can now be trained on historical data provided by the ESGF Data Statistics service. By learning from such download patterns, the AI model could then be applied to real-time data usage streams to detect irregular behaviour or potential infrastructure issues, such as sudden drops on data access, unexpected traffic spikes, or incomplete data transfers, or to identify the most used data in a given period.

The result? Infrastructure managers can react quickly to anomalies, implement preventive measures, and perform downstream actions on the DEP platform, thus improving the overall operation settings and maintaining reliable data access for the climate research community.

RI-SCALE provides federated compute, secure data pipelines, and AI frameworks that allow the design and development of AI models to support monitoring and potential improvement of ESGF data nodes operations and federation-wide data management. Pipelines convert raw download statistics into structured time series, models are trained on historical usage, and real-time streams can be continuously monitored and used to react to high loads and changes occurring in the data download streams.

By integrating AI within ENES RI, RI-SCALE enables predictive maintenance, optimizes system performance, and improves overall service reliability - all while adhering to FAIR principles and meeting data privacy and security constraints.

“RI-SCALE allows us to monitor climate data usage across the ENES RI, enabling more advanced data mining and deeper insights into usage patterns and system performance”, says Fabrizio Antonio, Computer Scientist, CMCC. “AI-powered anomaly detection could ensure that ESGF components remain efficient and reliable for the entire research community.”