HDF5 is a data model, file format, and I/O library that has become a de facto standard for HPC applications to achieve scalable I/O and for storing and managing big data from computer modeling, large physics experiments, and observations. Several ECP applications are currently using or planning to use HDF5 for I/O. Many new features, such as caching and prefetching, asynchronous I/O, log structured I/O, etc., have been developed in ECP for HDF5 applications’ taking advantage of exascale storage subsystems.
The tutorial will cover various best practices for using HDF5 efficiently, including performance profiling of HDF5, I/O patterns that obtain good I/O performance, and using new features such as asynchronous I/O, caching, prefetching effectively, log-structured I/O, and DAOS. The tutorial also covers UnifyFS to use the distributed node-local storage as a single file system. The tutorial presenters will use code examples and demonstrate performance benefits with efficient HDF5 usage.
