Using Subfiling to Improve Programming Flexibility and Performance of Parallel Shared-file I/O

摘要: There are two popular parallel I/O programming styles used by modern scientific computational applications: unique-file and shared-file. Unique-file usually gives satisfactory performance, but its major drawback is that managing a large number of files can overwhelm the task post-simulation data processing. Shared-file produces fewer allows arrays partitioned among processes to be saved in canonical order. As processors on machines increases into thousands more, problem size turn global array also increase proportionally. It not practical manage each larger than few hundreds GB. Hence, seek middle ground between these styles, we propose subfiling scheme divides multi-dimensional smaller subarrays, file, named subfile. Subfiling implemented top MPI-IO. We incorporate it netCDF library order preserve partitioning information file header, so later reconstructed. In addition, since decreases sharing reduce overhead system's consistency control. Our experimental results with several benchmarks show provide improved performance.