Parallel I/O and Portable Data Formats

4 Nov 2024, 09:00 → 6 Nov 2024, 16:30 Europe/Berlin

This course will take place as an on-site and in-person event. It is not possible to attend online.

The focus of this course is on the concept of performing I/O operations (reading/writing from/to file) in parallel, as well as demonstrating the usage of common libraries to perform such parallel I/O tasks. This course will start with an introduction of different parallel I/O strategies. It will then cover the netCDF-4 and HDF5 libraries. The course will have a dedicated session on I/O guidelines for Machine/Deep Learning with PyTorch. Lastly, optimization potential and best practices for parallel I/O are discussed.

The three day course consist of lectures as well as practical exercises on implementing the techniques presented in the course.

The time table is preliminary. Final agenda will be shared via a separate email to registered participants approximately two weeks before the course.

Prerequisites

• Knowledge about Linux shell (usage of terminal commands)
• Experience in one of the following programming languages C/C++/Fortran/Python
• Knowledge in MPI parallelisation or have participated in Introduction to parallel programming with MPI and OpenMP

Language:

This course is given in English.

Date:

4-6 November 2024, 09:00-16:30 each day

Instructors

Ilya Zhukov, Jolanta Zjupa, Junxian Chew, Aravind Sankaran, Jan Ebert (JSC)

Contact 

For any questions concerning the course, please send an e-mail to i.zhukov@fz-juelich.de, j.zjupa@fz-juelich.de, j.chew@fz-juelich.de
 

Monday, 4 November

09:00 → 10:30 MPI Onboarding

Speaker: Aravind Sankaran (Juelich Supercomputing Center)

10:30 → 10:45 Break

10:45 → 12:15 Introduction to Parallel I/O

Speaker: Ilya Zhukov (JSC)

12:15 → 13:15 Lunch Break

13:15 → 14:45 NetCDF-4

Speaker: Junxian Chew (Jülich Supercomputer Centre, Forschungszentrum Jülich)

14:45 → 15:00 Break

15:00 → 16:30 NetCDF-4

Speaker: Junxian Chew (Jülich Supercomputer Centre, Forschungszentrum Jülich)

Tuesday, 5 November

09:00 → 10:30 HDF5

Speaker: Jolanta Zjupa (JSC/FZJ)

10:30 → 10:45 Break

10:45 → 12:15 HDF5

Speaker: Jolanta Zjupa (JSC/FZJ)

12:15 → 13:15 Lunch Break

13:15 → 14:45 MPI-IO

Speaker: Junxian Chew (Jülich Supercomputer Centre, Forschungszentrum Jülich)

14:45 → 15:00 Break

15:00 → 16:30 Python Machine Learning I/O

Speaker: Jan Ebert (Forschungszentrum Jülich)

Wednesday, 6 November

09:00 → 10:30 I/O profiling and optimisation

Speaker: Ilya Zhukov (JSC)

10:30 → 10:45 Break

10:45 → 12:15 Best Practices

Speaker: Aravind Sankaran

12:15 → 13:15 Lunch Break

13:15 → 16:30 Hands-on