OpenSolaris NFS over RDMA Project
- Security: The current Read-Read based design allows the client to RDMA Read data from the servers buffers. This has a number of security and resource vulnerabilities. A malicious client may attempt to read data which it is not authorized to do so. In addition, the client is now in control of the server buffers and may attempt to tie up several of these buffers for an inordinate amount of time, denying service to other clients. The solution to both these problems lies in disallowing the client from issuing any RDMA operations to the server. The server instead uses a combination of RDMA Read and Write operations (Read-Write design) to fulfill bulk data transfer operations. This effectively eliminates the threat from malicious clients by allowing the server to not expose it buffers, in addition to maintaining complete server control over the allocation and usage of its buffers.
- Performance: Moving from a Read-Read design to a Read-Write design allows us to tackle some of the security and resource vulnerabilities present in the current design. One of the limitations of InfiniBand RDMA operations is the need to register buffers with the the HCA before issuing the operation. Because of the nature of the NFS protocol, these registrations must be performed in the critical path of each NFS operation. As a result, they constitute a substantial overhead. In this research, we have explored reducing the overhead of these registration operations through the use of Fast Memory Registration (FMR) and a registration cache.
- Interoperability: The final objective of this project is allow it to interoperate with other RDMA enabled implementations such as the Linux NFS over RDMA. The Linux implementation of NFS/RDMA uses additional techniques to further reduce registration overhead and directly place data into its target buffers. These techniques include the use of physical registration in combination with a larger number of NFS operations in the long reply phase of the RPC/RDMA protocol. These technique require design changes to accommodate these difference efficiently and effectively in the Read-Write design. Finally, connection setup management differences between the InfiniBand stacks on the two systems require additional mechanisms to seamlessly manage these objectives.
- draft-ietf-nfsv4-rpcrdma-06 RDMA Transport for ONC RPC (July 2007)
- draft-ietf-nfsv4-nfsdirect-06 NFS Direct Data Placement (July 2007)
- rfc3530.txt Network File System (NFS) version 4 Protocol (April 2003)
- draft-ietf-nfsv4-minorversion1-12 NFSv4 Minor Version 1 (June 2007)
- NFS/RDMA Talk (Tom Talpey, Connectathon 2006)
- NFS over RDMA Paper (Brent Callaghan, FAST '02)
Our testbed consists of
dual Opteron x2100's with 2GB memory and Single Data Rate
(SDR) x8 PCI-Express InfiniBand Adapters.
These systems were running OpenSolaris build version 33.
The back-end file system used was tmpfs which is a memory based
Conferences & Workshops (1)