37th SPEEDUP Workshop on High-Performance Computing



Sponsored by: Speedup   D-INFK   COLAB   IBM Zurich Research Center


HDF5 Tutorial

Elena Pourmal, Albert Cheng

ETHZ: Tuesday, September 9
or
EPFL: Friday, September 12, Room MA 12

08:00 - 9:00
Introduction to HDF5 Data, Programming Models and Tools
This Tutorial is designed for new HDF5 users. We will cover basic HDF5 Data Model objects and their properties; we will give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C (C++) examples will be used to illustrate HDF5 concepts. The main HDF5 utilities such as h5dump, h5diff, h5repack, h5stat, h5repart and HDF5 Java browsing and editing tool HDFView will be touched.
9:00 - 9:30
Advanced HDF5 features
This Tutorial is designed for the HDF5 users with some HDF5 experience.It will cover advanced features of the HDF5 library for achieving better I/O performance and efficient storage.

The following HDF5 features will be discussed: partial I/O, compression and other filters including new n-bit and scale+offset filters, and data storage options. Significant time will be devoted to the discussion of complex HDF5 datatypes such as strings, variable-length datatypes, array and compound datatypes.
9:30 - 10:00
Coffee break
10:00 - 12:00
Introduction to Parallel HDF5
This Tutorial is designed for the users who have exposure to MPI I/O and basic concepts of HDF5 and would like to learn about Parallel HDF5 Library. The Tutorial will cover Parallel HDF5 design and programming model. Several C and Fortran examples will be used to illustrate the basic ideas of the Parallel HDF5 programming model. Some performance issues including collective chunked I/O will be discussed.
12:00 - 13:15
Lunch break
13:15 - 14:15
HDF5: Life cycle of data
In this talk we will discuss what happens to data when it is written from the HDF5 application to an HDF5 file. This knowledge will help developers to write more efficient applications and to avoid performance bottlenecks.
14:15 - 14:45
Coffee break
14:45 - 16:45
New features in HDF5 1.8.0
This Tutorial targets HDF5 application developers and anyone who is interested in the new HDF5 Library features. The following new features available in 1.8.0 will be discussed:
  • Group revisions: We will introduce new features of the HDF5 Group object that include creation ordering, compact storage, and support of Unicode for the HDF5 object's names and datatypes. The Tutorial will also cover new APIs for copying HDF5 objects between HDF5 files.
  • HDF5 cache: New metadata cache to improve performance and memory usage in the HDF5 library has been implemented. We will give overview of the implementation and introduce new APIs that can be used by HDF5 application programmers to analyze and tune cache performance.
  • Error handling: Error handling mechanism was redesigned to give HDF5 application developers better control over the HDF5 Library error stack. New Error APIs that allow integration of application's error reporting with the HDF5 error reporting will be introduced in this part of Tutorial.
  • Links: This part of Tutorial discusses how HDF5 Library implements and handles linking mechanism in HDF5 including new type of links & external links to HDF5 objects.
  • Backward-Forward Compatibility issues: We will educate HDF5 users about backward and forward compatibility issues across releases of the HDF5 Library and versions of the HDF5 file format. We will discuss changes in the file format that were done to support new HDF5 features such as object creation order, compact groups, efficient access to the variable length data, UTF-8 encoding, external links, etc., and their implications on the HDF5 Library and users?? applications.
  • HDF5 and NetCDF4: We will give update on the project and show some performance results
16:45 - 17:30
Swiss HDF5 Power Users are Reporting ...
Benedikt Oswald and Achim Gsell, Paul Scherrer Institut (PSI), Villigen, Switzerland
The portable, open and scalable data storage standard H5Fed - Tranparent finite element data storage for tetrahedral meshes and associated data, slides
Aaron Ponti and Patrick Schwarb
High resolution in microscopy & large field of vie = very big datasets!
Cyril Flaig, ETHZ
Usage of HDF5 in Finite Element Analysis of Bone Structures