The steps to make NFS work over Infiniband.
I have a QLogic 12300 InfiniBand switch and a bunch of computers with QLogic Corp. IBA7322 QDR InfiniBand card. I wanted to use this network to speedup my fileserver shares.
The subnet manager runs on the switch. Therefore, the setup of the network is simple: just plug the cables to the switch. I did not configure anything special on the switch.
IP over InfiniBand
The operating system on the computers is Debian 9. The necessary packages are:
apt install infiniband-diags firmware-qlogic ibutils
Also, I had to load some modules by adding the following lines to /etc/modules
rdma_ucm ib_umad ib_uverbs ib_ipoib
Then assign an IP address to the interface. Edit the file /etc/network/interfaces and add based on the IPs you want to use:
auto ib0 iface ib0 inet static address 10.53.105.2 netmask 255.255.255.0 broadcast 10.53.105.255
The last step is to mount the NFS share. I modified the export on my fileserver to publish the share only on subnet of the IB network:
zfs set sharenfs="firstname.lastname@example.org/24" storage/home
And then mount the directory as usual. The following command can be used to show which exports are available on a machine. It has to be executed as root.
showmount -e localhost