This use case applies AI-based anomaly detection methods to climate data download statistics to identify irregularities and prevent infrastructure issues. It enhances infrastructure monitoring for Earth System Grid Federation (ESGF) nodes, enabling proactive data access management.
Smart Detection of Anomalies in Climate Data Usage
RI: ENES
Challenge
With millions of downloads daily from ESGF data nodes, infrastructure managers struggle to detect usage issues and failures in real time. This use case leverages AI for early detection of anomalies in download behaviour, allowing managers to proactively address infrastructural problems and improve service reliability.
Target
Apply AI models to identify unusual data access patterns, detect download anomalies, and support predictive maintenance and optimization of ESGF infrastructure.
Development Steps
-
Evaluate historical ESGF download data to construct training datasets
-
Build pipelines to convert download logs into structured time series data
-
Train anomaly detection models on historical patterns
-
Deploy models to analyse live data streams and detect anomalies in real time
Target Stakeholders:
-
ENES RI managers
-
ESGF data node administrators
-
Infrastructure monitoring and operations teams
Impact
Real-time anomaly detection improves infrastructure reliability and reduces service downtime. It enables data node operators to:
-
React promptly to excessive load or failures
-
Implement preventive actions
-
Improve user satisfaction and overall system efficiency
