Fast Analysis & Storage of Big Data Using HDF5

Data is growing at a tremendous rate with an overall increase in the digital universe with petabytes of data flowing from different sources.  The simplest explanation of this big data phenomenon is that, on one hand it’s about large amounts of data, while on the other hand it is the difficulty to analyze these large data sets. The analytics require some specific data to be retrieved. This usually involves methods with large time and space complexities. In this paper, we describe the HDF5 data model and some of its performance enhancing capabilities that minimize the complexities involved in large data handling.