thinking sysadmin

qstat -u aleonard -s z

NexentaStor in front of a NetApp FC LUN using MPxIO

leave a comment

  1. Create a Fibre Channel LUN on your NetApp and map it to your NexentaStor machine (I’m using version 3.0.2 in this example). For this example, I’ve created a 10GB LUN on a filer running ONTAP 7.2:
    netapp01> lun show /vol/nexenta01/lun01/lun
            /vol/nexenta01/lun01/lun      10g (10737418240)   (r/w, online, mapped)
    

    There are eight paths from our NetApp to our NexentaStor appliance, so the LUN appears eight times on the “qlc” adapter (lines 9-16 below):

    nmc@nexenta01:/$ lunsync
    Cleanup obsolete (dangling) device links?  Yes
    Re-enumerating LUNs... done.
    
    nmc@nexenta01:/$ show lun
    LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
    c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
    c1t0d0      sd128     cdrom        No Media              no      ata    -
    c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    syspo~/swap           zvol         1.0GB      syspool    no
    
  2. In NexentaStor “expert” mode, enable MPxIO for your Fibre Channel HBA (schedule this for a maintenance window, as it requires a reboot):
    root@nexenta01:/volumes# stmsboot -L
    stmsboot: MPXIO disabled
    root@nexenta01:/volumes# stmsboot -e -D fp
    WARNING: This operation will require a reboot.
    Do you want to continue ? [y/n] (default: y)
    updating //platform/i86pc/boot_archive
    updating //platform/i86pc/amd64/boot_archive
    The changes will come into effect after rebooting the system.
    Reboot the system now ? [y/n] (default: y)
    

    Note that this will not have any immediately noticable effect after rebooting:

    nmc@nexenta01:/$ lunsync
    Cleanup obsolete (dangling) device links?  Yes
    
    Re-enumerating LUNs... done.
    
    nmc@nexenta01:/$ show lun
    LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
    c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
    c1t0d0      sd128     cdrom        No Media              no      ata    -
    c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
    syspo~/swap           zvol         1.0GB      syspool    no             -
    

    However, in expert mode, you will now see the following:

    root@nexenta01:/volumes# stmsboot -L
    stmsboot: No STMS devices have been found
    
  3. Enable ALUA (Asymmetric Logical Unit Access) on the initiator group on the NetApp:
    netapp01> igroup show -v nexenta01
        nexenta01 (FCP):
            OS Type: solaris
            Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
            Member: 21:01:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
    netapp01> igroup set nexenta01 alua yes
    netapp01> igroup show -v nexenta01
        nexenta01 (FCP):
            OS Type: solaris
            Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
            Member: 21:01:00:aa:bb:cc:dd:ee(logged in on: 0b, 0d, vtic)
            ALUA: Yes
    
  4. Reconfigure and re-scan your NexentaStor HBA; note that the LUN is now attached to “mpxio” where it was previously attached to “qlc”:
    nmc@nexenta01:/$ lunsync -r
    Cleanup obsolete (dangling) device links?  Yes
    Re-scanning HBAs... done.
    Re-enumerating LUNs... done.
    
    nmc@nexenta01:/$ show lun
    LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
    c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
    c1t0d0      sd128     cdrom        No Media              no      ata    -
    c4t6*469d0  sd9       disk         10GB                  no      mpxio  60a98000486e542f5034577076716469
    syspo~/swap           zvol         1.0GB      syspool    no             -
    

    In NexentaStor expert mode, note that stmsboot now shows devices:

    root@nexenta01:/volumes# stmsboot -L
    non-STMS device name                    STMS device name
    ------------------------------------------------------------------
    /dev/rdsk/c3t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
    /dev/rdsk/c3t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
    /dev/rdsk/c3t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
    /dev/rdsk/c3t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
    /dev/rdsk/c2t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
    /dev/rdsk/c2t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
    /dev/rdsk/c2t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
    /dev/rdsk/c2t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
    

    You can now create a NexentaStor volume on your LUN.

Hat Tip to @complex on Twitter.

Reference: Is it possible to use I/O multipathing? How?

Written by Andy

May 28th, 2010 at 9:35 am

Leave a Reply