A European HPC-centric Benchmark Framework

Expected Outcome:The Project is expected to contribute to the following outcomes:Enhanced decision-making through comprehensive system comparisons that improves the procurement process for exascale, post-exascale supercomputers and supercomputers with dedicated AI capabilities. This will enable more informed choices regarding the acquisition of new systems and upgrades of existing onesCompetent HPC application developers and end-users in selecting systems that best meet their needs, balancing quality factors like accuracy with considerations of cost, such as time-to-solutionOverall improved operation and fine-tuning of HPC and HPC-AI systems leading to improved performance, throughput and energy optimization, and improved end-user experienceA unified, extensible and well-documented benchmarking framework to easily accommodate new, community-contributed benchmarks with common standards, versioning and controlA well-maintained and continuously updated benchmarking suite for exascale and post-exascale HPC, incl. set of apps, as well as AI models. Scope:A. Deployment of a benchmarking framework for designing, developing and executing exascale HPC and HPC-AI benchmarks. The envisioned benchmarking framework will:offer a fine grained and fair comparison methodology among different HPC systems, i.e. all benchmarks, benchmark run rules[1] and benchmark submission rules must be designed to ensure reproducibility, repeatability and replicability of metrics on the same system, ensuring fairness and comparability of metrics across different systemsdefine precise performance metrics with a clear focus on energy-related performance indicatorsstandardise all benchmarking input- and output formatscollect and report all benchmarking results while offering statistically sound result analysesensure that all benchmarks are executable on the respective target environment(s)offer a standardized structured workflow capturing and streamlining the entire benchmarking processoffer a standard