Configuring FCoE on IBM Switches - Sun, Jan 5, 2014

I've been lucky enough recently to work with FCoE a fair bit, using IBM Systems Networking switches both rack-mount and integrated into the IBM Flex/PureFlex chassis. What I've found is that getting FCoE running with IBM equipment is actually really easy. I thought I'd try and condense the wealth of information in the Application Guides down to a blog post.

Through talking with people about FCoE, and especially the IBM equipment, I get the impression that FCoE is a fairly common method these days for implementing storage fabric, however not everyone I speak with has an understanding of how it all hangs together, so I'll attempt to also cover the general theory behind FCoE where necessary.

I think everybody has at least heard of FCoE by now. If you are fairly unfamiliar with it, FCoE is basically a way to carry Fibre-Channel storage traffic using Ethernet switching equipment. The equipment needs to support FCoE, namely CEE/DCB for transmitting frames, and FIP snooping (or FIPS) for inspecting FCoE FIP packet contents to ensure packets are only transmitted between valid FCoE endpoints.

You also require an FCF (Fibre Channel Forwarder) somewhere in your FCoE fabric (storage network), which dissects the FCoE packets for the purposes of WWN zoning or NPIV (N-Port Identification Virtualisation). The FCF is then connected to your storage via plain old FC, or FCoE, completing the fabric.

Let's consider an example scenario, with an IBM EN4093 Flex chassis switch, connected to an 8264CS converged fabric switch/FCF. We'll assume the Omni (switchable Ethernet/FC SFP+ ports) on the 8264CS switch are connected to existing Fibre-Channel storage.

All of the below commands require your log on to the switch CLI via SSH, Telnet or using a serial cable. Setting up these connections is covered in the installation documentation for the switches.

You may notices that subtle differences (and some overt ones) exist between the EN4094 and 8264CS configuration syntax on the switches you are trying this on. The CLI used on IBM switches is in the process of transitioning between IBMNOS (the old BNT style configs) and ISCLI - the new "industry standard" (read: Cisco) style configuration. The syntax changes slightly each firmware release to make it more intuitive, and more familiar to people without IBM/BNT switch experience. Check the firmware versions of your switches if any of the below does not work, I recommend updating to at least version 7.7. The updates can be downloaded from IBM FixCentral.

Start by enabling CEE on the Chassis switch. This will allow DCB/CEE capable servers connected to the switch to negotiate via DCBX to use DCB/CEE instead of standard Ethernet.

cee enable

You will then need to create a VLAN for FCoE traffic. I suggest using VLAN 1002 unless you have a reason to change it, as most FCoE HBAs will default to VLAN 1002, and you might save yourself some configuration on the server's HBA and OS.

#enter the vlan configuration context for VLAN 1002
vlan 1002

#Just a descriptive name to make the config easier to read, not required
name "FCoE"

#An example of adding a single physical port to the FCoE network.
#Not required if you are only using UFP or vNICs to connect your servers.
member INTA1

#Replace with the ports your 8264CS switch are connected to (uplinks)
member EXT13,EXT14

#An example of adding a virtual NIC (UFP or vNIC) to the FCoE network.
#Not required if you are not using vNICs/UFP
vmember INTA2.2

#exit VLAN configuration context
exit

Enable tagging on your uplink ports, as you will likely want to carry other VLANs. It is also possible to have dedicated uplinks for FCoE traffic. If you are doing that - you can skip this step.

#Enter the configuration context for the two
#external uplink ports. Substitute EXT13,EXT14
#for your uplink ports.
interface EXT13,EXT14

#A descriptive name for the ports, again, not required.
name "Uplink"

#Enable tagging (trunking in Cisco-speak)
tagging

#Exit the configuration context
exit

Then enable FIP Snooping, so that FCoE sessions can be detected and established on the switch.

#Our FCoE VLAN, which are very familiar with by now.
vlan 1002

#Enable FCF for traffic in this VLAN
fcf enable

#Exit configuration context
exit

The next step, is to create your zone. You will need the WWNs of your HBAs, and the WWNs of your storage controllers. Create separate zones if your servers have multiple paths to the storage controllers, for example if they have multiple HBAs you would like to connect to the same LUNs, in order to provide multi-pathing.

#First, create aliases for your WWNs. Not required,
#but much easier in the long-run if you need to change
#WWNs on a server or storage controller!

#Alias for your server's HBA. Replaces the x's with #the actual WWN, be careful to include the colons between segments.
fcalias ServerHBA wwn xx:xx:xx:xx:xx:xx:xx:xx

#Alias for your storage controller, you will likely have multiple
#controllers for the same storage, so create more aliases, and
#include them in the zone if required.
fcalias StorageController xx:xx:xx:xx:xx:xx:xx:xx

#Create a zone named FirstZone zone name FirstZone
#Add aliases of HBA, and Controller.
member fcalias ServerHBA member fcalias StorageController

#exit this context
exit

#Now create a zone set. We only have one zone, but
#if you have multiple related zones, you can group
#them logically.
zoneset name ActiveConfig member FirstZone exit

#Now make the zone set, and zone, active.
zoneset activate name ActiveConfig

You should then be able to rescan your HBAs and see the storage controllers on the server.

Easy, right? Feel free to send me any questions, or let me know if you hit any snags. I hope this is useful, and that it saves you some time. Thanks for reading.