My cookie preferences
From SAFE to Zarr: The EOPF Sample Service Initiative
Briese C, Reimer C, Reck C, Papadakis D, Claus M, Brandt G, Fouilloux A, Odaka T
Over the past decade, the operational Copernicus Sentinels Data Processors have generated vast amounts of Earth observation data, supporting various scientific and commercial applications. However, the current format used by ESA to provide Copernicus data, known as SAFE (Standard Archive Format for Europe), has become outdated. To address this, ESA has initiated the transition to a new Zarr-based data format. The Earth Observation Processing Framework (EOPF) Sample Service is ESA’s official initiative to support this transition by providing early access to the new format for users. This shift is essential for creating a cloud-native and interoperable solution that enhances data accessibility and integration with modern processing frameworks. The primary goal is to standardize data formats across Sentinel missions, enable scalable processing on cloud platforms, and ensure compatibility with contemporary data science tools. This initiative is crucial for minimizing disruption and ensuring continuity for users, applications, and services built around existing data formats.
The EOPF Sample Service comprises several key components. The EOPF Core Platform re-formats ingested SAFE data products into the new cloud-optimized EOPF Zarr data products and provides data access via STAC API and S3 API. To ensure timely conversion, the platform utilizes Argo Events and the Copernicus Data Space Ecosystem's subscription service. This platform is maintained by experts from EODC and DLR. The EOPF User Platform offers additional user services, including JupyterHub (BinderHub), Dask, and a STAC Browser, which are essential for supporting user adoption by lowering the entrance barrier to cloud applications and data discovery capabilities. The service is designed to make use of advanced technologies such as Kubernetes for container orchestration and Dask for parallel computing. User and identity management is achieved in cooperation with the Copernicus Data Space Ecosystem.
User adoption is further facilitated through Jupyter Notebooks designed by experts within the consortium, including members from the Pangeo community. These notebooks showcase the use of the new format within the community and are continuously improved by incorporating user feedback. In addition, enhancements are made to widely-used software tools like GDAL to support the new format, with practical demonstrations available through Jupyter Notebooks. The consortium selected by ESA to carry out this implementation includes experts from Brockmann Consult, DLR, Ifremer, EURAC, Evenflow, Simula, and EODC, each contributing their specialized knowledge in Earth observation, data management, and user engagement.
This contribution aims to present the EOPF Sample Service initiative and the current status of its implementation. The first Jupyter Notebooks demonstrating the new format will also be showcased, providing users with an intuitive and user-friendly interface for accessing and processing sample data in the new EOPF format.
DOI: 10.5194/egusphere-egu25-21202
