NFS over InfiniBand

by | November 15, 2017

The steps to make NFS work over Infiniband.

I have a QLogic 12300 InfiniBand switch and a bunch of computers with QLogic Corp. IBA7322 QDR InfiniBand card. I wanted to use this network to speedup my fileserver shares.

The resources:

The subnet manager runs on the switch. Therefore, the setup of the network is simple: just plug the cables to the switch. I did not configure anything special on the switch.

IP over InfiniBand

The operating system on the computers is Debian 9. The necessary packages are:

apt install infiniband-diags firmware-qlogic ibutils

Also, I had to load some modules by adding the following lines to /etc/modules


Then assign an IP address to the interface. Edit the file /etc/network/interfaces and add based on the IPs you want to use:

auto ib0
iface ib0 inet static


The last step is to mount the NFS share. I modified the export on my fileserver to publish the share only on subnet of the IB network:

zfs set sharenfs="rw=@" storage/home

And then mount the directory as usual. The following command can be used to show which exports are available on a machine. It has to be executed as root.

showmount -e localhost


Leave a Reply

Your email address will not be published. Required fields are marked *