CY-Bench

Forecasting crop yields before harvest is done with machine learning as well as process-based models. However, studies are often case-based, using different countries, data sources and evaluation procedures. This makes results difficult to compare and progress difficult to measure. CY-Bench provides a standardised infrastructure for this task. It compiles official sub-national yield statistics for maize in 38 countries and for wheat in 29 countries, spanning six continents. These statistics are paired with growing-season weather, soil moisture, satellite vegetation indicators and soil properties. All of these data are aligned to the administrative units in which the yields are reported and to each crop’s growing season. A Python library loads the data, aligns the predictors with the crop seasons and forecast lead times, performs the necessary preprocessing for each model, runs the prescribed evaluation and reports the standard metrics. This makes a domain-curated forecasting task accessible to machine learning researchers and provides a common basis for comparing methods under conditions designed to resemble real pre-harvest forecasting.

The CY-Bench project was conceptualised and developed by the Machine Learning team at the Agricultural Model Intercomparison and Improvement Project (AgML), which brings together agronomists, climate scientists, and machine learning researchers. This interdisciplinary collaboration enabled the evaluation of potential weather, soil, vegetation and crop-mask datasets. These judgements are explicit: the alternatives, trade-offs and reasons for selecting particular sources are documented, and each selected dataset is accompanied by a data card linking to its sources and related documentation.

The selection of sources was guided by five principles: global coverage, public accessibility, regular updating, near-real-time availability, and demonstrated relevance to crop growth. Near-real-time availability is important because the benchmark is intended to replicate operational forecasting conditions rather than retrospective prediction alone.

The evaluation procedure addresses a known source of overly optimistic results. Yields in neighbouring regions in the same year are correlated, meaning random train-test splits could place related observations on either side of the boundary. CY-Bench uses a different approach: leave-one-year-out evaluation. This involves withholding a complete year so that information from that season cannot be used for training, and extreme years remain intact during testing. The predictor time series are also aligned to crop seasons and can be cut at specified lead times before harvest, thus preserving the information boundary of an actual forecast. The dataset and Python library are released under EUPL-1.2, which was chosen to be compatible with the licences of the incorporated sources.

CY-Bench is an open, analysis-ready benchmark covering maize in 38 countries and wheat in 29 countries across six continents. The dataset and code have separate persistent identifiers, and the accompanying Python package loads the data and runs the benchmark. Data Cards document each constituent source, and preparation scripts and notebooks are released to enable coverage to be extended as new national crop statistics become available. As the predictor datasets are globally available, the authors identify sub-national yield statistics as the main obstacle to including more countries.

The coverage deliberately includes diverse agricultural systems and regions that are underrepresented in agricultural machine learning research, enabling models to be tested beyond well-studied settings. Any remaining gaps are made visible; notable omissions include Ukraine, Uganda and Tanzania for maize, and Canada, Ukraine and Russia for wheat.

Limitations are documented in detail. Pests, diseases, farm management and socioeconomic conditions are not globally represented; irrigated and rainfed systems are not separated; and yield statistics provide no uncertainty estimates. Crop masks and calendars are static. Rather than assuming that the resulting mask error is negligible, the authors tested it across more than 12,000 administrative regions and found that mask-driven variability accounted for around 2% of the natural interannual variability in the vegetation signal. The authors also note that leave-one-year-out evaluation assumes independence between years, which may be too strong an assumption.

Benchmark results are included as illustrative examples rather than to determine the best method, and systematic model comparison is left to separate work. Therefore, the resulting contribution is not a single forecasting model, but rather reusable infrastructure for the transparent, geographically broad and leakage-aware evaluation of crop yield forecasting methods.

CY-Bench has subsequently been used to evaluate a range of crop-yield prediction models. The results are presented in a public dashboard and leaderboard, covering statistical baselines, process-based crop models, feature-engineered machine-learning models, tabular foundation models and deep sequence models. Performance is reported by country and disaggregated into overall, spatial, temporal and anomaly skill. This provides evidence that the benchmark can support systematic and transparent model evaluation beyond individual case studies.

·     Kallenberg, M. et al. (2026). CY-Bench: a comprehensive benchmark dataset for sub-national crop yield forecasting. Earth System Science Data 18, 3997–4018. https://doi.org/10.5194/essd-18-3997-2026

·     Dataset: https://doi.org/10.5281/zenodo.11502142

·     Code: https://doi.org/10.5281/zenodo.20456375

·     https://cybench.agml.org/

·     Interactive results for Benchmarking the State of AI for Crop Yield Forecasting on CY-Bench: https://wur-ai.github.io/AgML-CY-Bench-dashboard/