NFS over InfiniBand

by | November 15, 2017

The steps to make NFS work over Infiniband.

I have a QLogic 12300 InfiniBand switch and a bunch of computers with QLogic Corp. IBA7322 QDR InfiniBand card. I wanted to use this network to speedup my fileserver shares.

The resources:

https://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html

The subnet manager runs on the switch. Therefore, the setup of the network is simple: just plug the cables to the switch. I did not configure anything special on the switch.

IP over InfiniBand

The operating system on the computers is Debian 9. The necessary packages are:

apt install infiniband-diags firmware-qlogic ibutils

Also, I had to load some modules by adding the following lines to /etc/modules

rdma_ucm
ib_umad
ib_uverbs
ib_ipoib

Then assign an IP address to the interface. Edit the file /etc/network/interfaces and add based on the IPs you want to use:

auto ib0
iface ib0 inet static
  address 10.53.105.2
  netmask 255.255.255.0
  broadcast 10.53.105.255

 

The last step is to mount the NFS share. I modified the export on my fileserver to publish the share only on subnet of the IB network:

zfs set sharenfs="rw=@10.53.105.0/24" storage/home

And then mount the directory as usual. The following command can be used to show which exports are available on a machine. It has to be executed as root.

showmount -e localhost

 

Leave a Reply

Your email address will not be published. Required fields are marked *