HPC and Big Data Technologies for Global Challenges

Dive into the HiDALGO2 project's latest steps in this 2^nd edition of the newsletter. HiDALGO2 continues to move forward with crucial interdisciplinary research and studies by all working teams, participating in relevant events and conferences and nurturing synergies and collaborations with other European projects and initiatives.

We invite you to follow us on our channels to stay updated on all the project’s activities. Enjoy your reading!

www.hidalgo2.eu

WELCOME

Welcome from the Project Coordinator

Dear Readers of the newsletter, Colleagues,

The HiDALGO2 project is an important undertaking for European science addressing important key challenges for the entire international community, namely climate change. They have long been an undeniable phenomenon observed in many places on Earth. Climate change affects our everyday lives and, as a result, increasingly affects our life quality. One of the key reasons is the dynamic development of societies, which has had a significant impact on the natural environment for decades. This can be dramatically experienced nowadays through violent weather phenomena (e.g. storms, rainfall, fires) or the air quality in cities.

These problems are addressed through Global Challenges (GC), the core of HiDALGO2 activity. GC requires interdisciplinary expertise and large-scale solutions due to its inherent complexity. The aim of the HiDALGO2 project is therefore to explore synergies between modeling, data collection, simulation, data analysis, and visualization, and to achieve better scalability of current and future HPC and AI infrastructures in order to deliver highly scalable solutions that can effectively exploit pre-exascale systems.

Our project focuses on four environmental application cases: improving air quality in urban agglomerations, energy efficiency of buildings, renewable energy sources and forceful fires. A common feature of the modeling of the above simulations is the use of numerical analysis of fluid flows using the computational fluid dynamics (CFD) method, which usually requires very high computing power. Therefore, the ability to use the most powerful HPC machines in Europe and in the world offered by EuroHPC JU for our work is an undeniable advantage that strengthens our capabilities.

However, it should be remembered that HiDALGO2 is not only about the development of simulation models. In our project, we place great emphasis on issues related to the scalability of solutions and the best possible adjustment of software to the infrastructure (co-design) through the use of appropriate benchmarking methodology and algorithmic optimization methods. This enables the effective use of top-class HPC systems to simulate complex structures with much greater accuracy unattainable in the case of calculations using cloud solutions. The quality of our solutions is assessed through uncertainty analysis performed in ensemble runs.

Last but not least, HiDALGO2's activity is to contribute to reducing the skills gap among the EU user community by sharing knowledge through organized specialized workshops and trainings.

The first period of the project is mainly devoted to the installation of applications on EuroHPC JU machines, integration with the central CI/CD system offered by CASTIEL2, setting up of services that are components of the HiDALGO2 architecture, and numerous dissemination activities.

Stay tuned by following us on social media.

Marcin Lawenda

Poznań Supercomputing and Networking Center

USE CASES & TOOLS

The USE CASES of the HiDALGO2 project

HiDALGO2 comprises four use cases focused on environmental aspects that improve the quality of life of urban and rural residents. These are Urban Air Project (UAP), Urban Buildings (UB), Renewable Energy Sources (RES), and Wildfires (WF). UAP and UB deal with urban development issues, focusing on air quality considering the weather conditions, car traffic and the emission of pollutants by buildings.
 

Urban Air Project (UAP): In this use case, we work around the evolution of air quality in urban areas, considering pollution, wind, comfort, and planning. The core of our work here is the Urban Air Flow computational model based on modern massive HPC, mathematical models, and AI technologies.

Urban Building (UB): This use case will develop building energy and indoor air quality models to simulate the contribution of the buildings at the city scale in terms of heat, greenhouse gas emissions (GHG), and NOx emissions. Information about gas emission enriches the UAP pilot with new, never-before-analysed data.

Renewable Energy Sources (RES): This use case will develop more advanced models for wind and solar energy and study the impact of uncertainties in the model, which was never done before. It answers the question of how much energy from wind and sun can be produced considering specific weather conditions. It is with noticing that RES has dedicated modules that consider urban and extra-urban needs. 

Wildfires (WF): This use case will provide a computational model for the simulation of wildfire-atmosphere interactions and smoke dispersion at several scales (a new approach) aimed at assessing risk and potential impacts over populated areas. Moreover, a new model for probabilistic fire spread in Wildland Urban Interface (WUI) areas will be delivered along with an adaptation of the software FDS and algorithms for the visual simulation of animated volumetric smoke.

Data analytics, AI, and visualisation in the HiDALGO2 project

The development of data analytics and AI methods is driven by the requirements of the project’s use cases. Vast amounts of data produced by simulations must be tackled to find dependencies, relations, and implications. In many cases, AI methods must be implemented to empower data analytics solutions. HiDALGO2 is developing ways to increase the performance of the analysis, considering efficient algorithms and the corresponding implementations for serial and parallel computations. Consequently, this objective will address the parallelisation weaknesses of modern data analytics frameworks and procedures by applying message-passing-oriented functionality for in-situ data processing. The tests of the aspects mentioned above will be carried out using EuroHPC Joint Undertaking’s infrastructure.

NEWS

HiDALGO2 in SC23, 12-17 November 2023

Photo credit: SC23 website

HiDALGO2 participates in SC23 on 12-17 November 2023 in Denver under the “umbrella” of the EuroHPC Joint Undertaking with a demonstration video that unfolds the aims and goals of the project and presents the central facts, including the partners, the duration, and the funding. The EuroHPC Joint Undertaking will present various European projects – ΗIDALGO2 being among them- in a booth in the exhibition area, close to other European partners.

HiDALGO2 in ICIAM 2023, Japan, August 2023

Photo credit: ICIAM 2023 highlights video

Zoltán Horváth from our partner Széchenyi István Egyetem (SZE) participated at the ICIAM 2023 conference in Tokyo in August 2023 as an invited speaker in the mini-symposium "Physics-based and data-driven modeling for digital twins" with joint work of Mátyás Constans. He presented the RedSIM solver created in HiDALGO2.

Watch two interesting videos:
On exhaust pipe acoustics problem – for FOM
On urban air flow computation for the city of Gyor – for FOM and ROM

HiDALGO2 partners met in Athens, June 20 & 21, 2023

On June 20 and 21, 2023, we had a productive meeting in Athens, where all project partners gathered for a 2-day extended discussion on our project’s progress and the following essential steps. We were also introduced to our Advisory Board members, who will review and provide their input on our next steps.

JOINED ACTIONS

Collaboration with ENCCS - EuroCC National Competence Centre Sweden & SLB-analys

In the past months, several communication activities were on. First, we had a fruitful meeting with ENCCS - EuroCC National Competence Centre Sweden and SLB-analys (Stockholm Sta), a company associated with the City of Stockholm. After an introduction about the project from the project coordinator, Marcin Lawenda from Poznan Supercomputing and Networking Center, and Zoltan Horvath from Széchenyi István Egyetem, we got great feedback with a similar presentation from SLB-Analys. Both parties are interested in cooperating with us in developing simulation models and using HiDALGO2 project outcomes.

 Sharing knowledge with Indian scientists

As wildfires are a tragic environmental phenomenon of our times, our project’s use case about this issue seems to attract particular interest. More specifically, in an invitation we got from the European Commission and Mr. Juan Pelegrin (HPC and quantum technologies), scientists from India (as part of the INCO call of the European Commission) showed great interest in our case led by MeteoGRID due to similar research they are conducting. Our project coordinator, Marcin Lawenda, presented the goals to the other party. We will stay in touch with them, sharing our knowledge and exchanging results from our work.

 Collaboration with National Competence Centres

In the past months, our collaboration efforts with several National Competence Centres and MSEs leveled up! We joined several National Competence Centres at an online meeting, where our first seeds of cooperation were sown. The topics that attracted the most interest were the ones related to the portal and collaboration with HPC systems. We will keep in touch and share our work outcomes and valuable ideas.

 Workshop on OpenFoam

HiDALGO2 participated in September 2023 along with other CoEs at the joint workshop on OpenFoam. The workshop was attended by EXCELLERAT 2, exaFOAM Project, Exasim, HPC CoE, and CASTIEL2 project.

BLOG

Addressing the challenge of complex computing scenarios on large-scale resources

How to support the execution of challenging scenarios on high-end computing resources is one of the critical objectives of the HiDALGO2 project. With a set of carefully selected software components, we aim to ensure both sufficient performance of parallel computations and offer good flexibility and User experience. Here, we highlight QCG-Portal and QCG-PilotJob that are being integrated into the project’s ecosystem to comprehensively support the management and efficient execution of complex computing scenarios on large-scale resources.

Apply an uncertainty quantification (UQ) framework to enhance model credibility

Read how we aim to address the pressing question of how well our models and simulations perform by applying an uncertainty quantification (UQ) framework, specifically focusing on its application to environmental challenges that require computational fluid dynamics (CFD) simulations.

INDUSTRY PAPERS OF HiDALGO

HiDALGO2 scientists work hard on simulations to increase their functional capabilities and scalability. The effects of their work are expected soon in the form of numerous publications and reports. However, recently, we can boast of publishing two papers on the results of work from the previous project, HiDALGO. 

The first one published in Future Generation Computer Systems is entitled: “Profiling and optimization of Python-based social sciences applications on HPC systems by means of task and data parallelism” and authored by Łukasz Szustak, Marcin Lawenda, Sebastian Arming, Gregor Bankhamer, Christoph Schweimer and Robert Elsaesser. The article presents optimisation techniques for two Python-based large-scale social sciences applications: SN (Social Network) Simulator and KPM (Kernel Polynomial Method). These applications use MPI technology to transfer data between computing processes, which in the regular implementation leads to load imbalance and performance degradation. To avoid this effect, we propose a 2-stage optimization (tasks scheduling and division) and facilitated use of multiple NUMA domains. 

The second paper “Large-Scale Parallelization of Human Migration Simulation” published in IEEE Transactions on Computational Social Systems is authored by Derek Groen, Nikela Papadopoulou, Petros Anastasiadis, Marcin Lawenda, Lukasz Szustak, Sergiy Gogolenko, Hamid Arabnejad, and Alireza Jahani. This work focuses on an agent-based modelling tool (the Flee simulation code) that can forecast population displacements in civil war settings. However, performing accurate simulations requires non-negligible computational capacity. We present an approach parallelization for fast execution on multicore platforms and discuss the algorithm's computational complexity and implementation.

The findings of those two contain general conclusions on optimisation and co-design techniques that can also be used in the current work.

What’s coming next in HiDALGO2?

HiDALGO2 is going to participate in major HPC events like SC23 in Denver (at EuroHPC Joint Undertaking booth), and in HiPEAC 2024 in Munich along with ESiWACE3 with a joined workshop. Also, important collaborations with other Centres of Excellence and National Competence Centres are underway with an exchange of tools, studies, dissemination material, and joined webinars. Last but not least, do not forget to join the EuroHPC User Day that will take place on 11 December 2023, in Brussels, Belgium. Stay tuned!

PARTNERS

To keep up to date with our project’s progress, subscribe to our newsletter through our website and follow our social media channels.

www.hidalgo2.eu

Co-funded by the European Union. This work has received funding from the European High Performance Computing Joint Undertaking (JU) and Poland, Germany, Spain, Hungary, and France under grant agreement number: 101093457.

View in browser | Unsubscribe