Constructing collaborative desktop storage caches for large scientific datasets

摘要: High-end computing is suffering a data deluge from experiments, simulations, and apparatus that creates overwhelming application dataset sizes. This has led to the proliferation of high-end mass storage systems, area clusters, centers. These facilities offer large range choices in terms capacity access rate, as well strong availability consistency support. However, for most end-users, “last mile” their analysis pipeline often requires processing visualization at local computers, typically desktop workstations. End-user workstations---despite having more power than ever before---are ill-equipped cope with such demands due insufficient secondary space I/O rates. Meanwhile, portion unused.We propose FreeLoader framework, which aggregates unused bandwidth into shared cache/scratch space, hosting large, immutable datasets exploiting locality. article presents architecture, component design, performance results based on our proof-of-concept prototype. Its architecture comprises contributing benefactor nodes, steered by management layer, providing services integrity, high performance, load balancing, impact control. Our experiments show an appealing low-cost solution storing massive delivering higher rates traditional facilities, namely, or remote file Internet repositories. In particular, we present novel striping techniques allow efficiently aggregate workstation's network communication bandwidth. addition, native workload donor machines small can be effectively controlled. Further, security features encryptions integrity checks easily added filters interested clients. Finally, demonstrate how legacy applications use API store retrieve datasets.