Clustered Storage and File Systems

In modern clusters, I/O is quickly emerging as the main bottleneck limiting performance. The need for scalable parallel I/O and file systems is becoming more and more urgent. The performance of network storage systems is often limited by overhead in the I/O path, such as memory copying, network access costs, and protocol overhead. High speed networking architectures such as InfiniBand and Quadrics create opportunities to address these issues without changing fundamental principles of production operating systems.

We have examined the feasibility of leveraging InfiniBand and Quadrics technology to improve I/O performance and scalability of cluster file systems. By taking advantage of user-level networking and Remote Direct Memory Access (RDMA) we have implemented The Parallel Virtual File System (PVFS) over InfiniBand and the second generation of Parallel Virtual File System (PVFS2) over Quadrics. In addition, we have suggested Unifier to unify file cache management and communication buffer management for PVFS over InfiniBand. Moreover, we have exploited InfiniBand RDMA to improve iSCSI. We plan to extend these research activities to provide high performance storage solution for InfiniBand clusters.

Broad research issues being investigated in this direction:



  • R. Noronha, and D. K. Panda, IMCa: A High Performance Caching Frontend for GlusterFS on InfiniBand. , Int'l Conference on Parallel Processing, Portland, OR, 2008.
  • Sundeep Narravula, Hari Subramoni, Ping Lai, Bhargavi Rajaraman, Ranjit Noronha and Dhabaleswar K Panda, Performance of HPC Middleware over InfiniBand WAN. International Conference on Parallel Processing (ICPP) 2008. (accepted for publication) [Technical Report pdf]
  • W. Yu, R. Noronha, S. Liang, and D.K. Panda, Benefits of High Speed Interconnects to Cluster File Systems: A Case Study with Lustre, Communication Architecture for Clusters (CAC) Workshop, to be held in conjunction with Int'l Parallel and Distributed Processing Symposium (IPDPS '06), April 2006.
  • W. Yu, S. Liang and D.K. Panda. High Performance Support of Parallel Virtual File System (PVFS2) over Quadrics. The 19th ACM International Conference on Supercomputing (ICS '05). Cambridge, Massachusetts, June, 2005.

  • J. Wu, P. Wyckoff, D. K. Panda, and Rob Ross. Unifier: Unifying Cache Management and Communication Buffer Management for PVFS over InfiniBand. IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid 04), 19-22 April, Chicago, Illinois.
  • J. Liu, D. K. Panda, and M. Banikazemi. Evaluating the Impact of RDMA on Storage I/O over InfiniBand. SAN-03 Workshop (in conjunction with HPCA), Feb. 2004. Conference Slides .
  • J. Wu, P. Wyckoff, and D. K. Panda. Supporting Efficient Noncontiguous Access in PVFS over InfiniBand. IEEE Cluster Computing 2003, Hong Kong, Dec. 1-4, 2003. Technical Report,
  • Jiesheng Wu, Pete Wyckoff, and Dhabaleswar K. Panda Demotion-Based Exclusive Caching through Demote Buffering: Design and Evaluations over Different Networks Workshop on Storage Network Architecture and Parallel I/O (SNAPI) held in conjunction with Parallel Architecture and Compilation Techniques (PACT) '03 conference, Sept. 28, \2003, New Orleans, LA.
  • J. Wu, P. Wyckoff, and D. K. Panda. PVFS over InfiniBand: Design and Performance Evaluation. International Conference on Parallel Processing (ICPP 03). Oct. 2003. Technical Report.
  • J. Wu and D.K. Panda, MPI/IO on DAFS Over VIA: Implementation and Performance Evaluation , Communication Architecture for Clusters (CAC'02) Workshop, held in conjunction with IPDPS '02, April 2002.
  • Technical Reports

    PhD Dissertations

    Graduate Student Researchers:

    Jiesheng Wu, Jiuxing Liu, Weikuan Yu, Ranjit Noronha, Shuang Liang

    D. K. Panda
    Last modified: Sun Jun 12 13:34:17 EDT 2005