Setting up Lustre Router - Hosts and Clients

by Jesse Stroik — last modified Mar 20, 2014 09:53 PM


Important Note: The definitive source for Lustre documentation is the Lustre Operations Manual available at

These documents are copied from internal SSEC working documentation that may be useful for some, but be we provide no guarantee of accuraccy, correctness, or safety. Use at your own risk.

These notes are for hosts bridgng between networks, such as infinband to ethernet. Significant portions of the notes have been removed as they are specific to SSEC procedures.

Install Software on the Router

The router only requires lustre client software, not server.

Setting up the Lustre Networks

This sets up the lnet module to load and start the proper networks to communicate on. it needs to at least include all networks the router needs. The most common examples will be the current IB network it is connected to, and the SSEC public network. Int his example, I'll use the SSEC IB and SSEC public nets.
options lnet 'ip2nets="o2ib0 172.16.*.*; tcp0"'

Routing configuration

Set the /etc/modprobe.d/lustre.conf files accordingly. Create the file if it does not exist.


options lnet ip2nets="tcp0 128.104.*.*; o2ib0 172.16.*.*" forwarding=enabled
 NOTE: ORDER MATTERS with tcp0 and o2ib0 definitions



Need to have lustre client installed


 Simple example for client:
options lnet networks="tcp" routes="o2ib0"
 After creating lustre.conf:

service --add lustre  service --add lnet  chkconfig lustre on  chkconfig lnet on  reboot

After the reboot, you can create mount points and add lustre mounts. Here is an example of adding fjord to /etc/fstab. /fjord                  lustre defaults,_netdev,user_xattr      0 0


options lnet ip2nets="o2ib0 172.16.*.*" routes="tcp0 172.16.24.[220-229]@o2ib0"

 where is your router. multiple can be speicifed, and you can use the syntax:

routes="network-type1 N.N.N.[N-N]@network-type2

 Where N is a number.




options lnet ip2nets="o2ib0 172.16.*.*" routes="tcp0"



After you attempt to mount, run:
lfs df -h

And see if every OST is active and available.

It may take a few minutes for the mount points to work properly after you get everything else up, because it needs to determine which OSTs are up and sane, and it needs to determine which routers are active.