Design and Implementation of GPU-based File Similarity Evaluation System

Background/Objectives: Recently, storage systems and backup systems are popularly used and the number of duplicated data is increased drastically. To minimize data storage size and efficient use of network bandwidth, we proposed deduplication systems and file similarity measurement schemes with GPGPU scheme. The GPGPUs are applied to file similarity measurement for computation speedup. Methods/Statistical Analysis: To cope with the problem accompanying the parallelization of the measurement, we compare two implementations with shared memory and preprocessing. In addition, we propose an alternative to Rabin fingerprinting algorithm to lessen the computational burden of the algorithm to the GPUs. We compare the performance of the systems in time elapsed for several files. Findings: First, we found through experiments that the preprocessing was slightly faster than the shared memory scheme for the overlapped region of consecutive data segments which were assigned to different cores. This region should be shared by two cores for fingerprinting. By adapting GPGPU parallelization with the preprocessing technique for file similarity measurement, the proposed system outperformed the systems with a multi-core CPU. Also, it gets faster for the bigger file. In addition, we made the system three times faster by adapting an alternative to Rabin fingerprinting algorithm. It eliminates the computational burden of the algorithm and provides comparable results to the system with the latter. Improvements: The procedure will be beneficial to de-duplication system in determining file similarity and finding duplicated regions of two files. We achieved speedup in the measurement of file similarity by parallelization on GP-GPUs with two methods for overlaps of consecutive data segments and an alternative fingerprinting algorithm.