ECMWF AIFS Ensemble Forecasting with GPU Cloud Computing
ECMWF AIFS: AI-Powered Weather Forecasting
ECMWF’s Artificial Intelligence Forecasting System (AIFS) represents a paradigm shift in numerical weather prediction. Built on transformer architectures and trained on ERA5 reanalysis data, AIFS generates ensemble forecasts at a fraction of the computational cost of traditional physics-based models. The AIFS ENS v1.0 model produces 50-member ensemble forecasts extending to 33 days (792 hours).
GPU Cloud Computing Workflow
Running AI weather models requires significant GPU resources. The ea-aifs repository implements a cost-efficient workflow using Coiled for on-demand GPU provisioning:
Seven-Step Pipeline
| Step | Environment | Purpose |
|---|---|---|
| 1 | CPU | Download ECMWF open data, preprocess to pickle format |
| 2 | GPU | Transfer input states from cloud storage |
| 3 | GPU | Execute AIFS model for 50 ensemble members |
| 4 | GPU | Upload GRIB outputs, shutdown expensive GPU |
| 5 | CPU | Regrid from N320 to 1.5° resolution |
| 6 | CPU | Ensemble quintile analysis |
| 7 | CPU | Forecast submission |
Cost Optimization Strategy
The workflow minimizes GPU costs by:
- Preprocessing data on cheap CPU instances before GPU execution
- Using cloud storage (GCS) as intermediate buffer between environments
- Multi-threaded uploads to reduce GPU idle time
- Immediately shutting down GPU after model execution
Infrastructure
# CPU environment for data preparation
coiled notebook start --vm-type n2-standard-2 --software aifs-etl-v2
# GPU environment for AI model execution
coiled notebook start --vm-type a2-ultragpu-1g --software flashattn-dockerv1
Application for East Africa
This workflow enables ICPAC to generate AI-based ensemble forecasts for the East Africa region, complementing traditional dynamical models with GenAI approaches for improved early warning systems.