APEnet+: a 3D Torus network optimized for GPU-based HPC Systems

摘要: In the supercomputing arena, strong rise of GPU-accelerated clusters is a matter fact. Within INFN, we proposed an initiative — QUonG project whose aim to deploy high performance computing system dedicated scientific computations leveraging on commodity multi-core processors coupled with latest generation GPUs. The inter-node interconnection based point-to-point, performance, low latency 3D torus network which built in framework APEnet+ project. It takes form FPGA-based PCIe card exposing six full bidirectional links running at 34 Gbps each that implements RDMA protocol. order enable significant access reduction for data transfer, direct network-to-GPU interface was built. specialized hardware blocks, integrated board, provide support GPU-initiated communications using so called peer-to-peer (P2P) transactions. This development made close collaboration GPU vendor NVIDIA. final shape complete deployment assembly standard 42U racks, one capable 80 TFLOPS/rack peak cost 5 k€/T F LOPS and estimated power consumption 25 kW/rack. this paper report status rack R&D activities 2012 will focus enhancement through adoption new 28 nm FPGAs allowing implementation Gen3 host addition fault tolerance-oriented capabilities.