thinking sysadmin » opensolaris

NexentaStor in front of a NetApp FC LUN using MPxIO

Andy — Fri, 28 May 2010 17:35:13 +0000

Create a Fibre Channel LUN on your NetApp and map it to your NexentaStor machine (I’m using version 3.0.2 in this example). For this example, I’ve created a 10GB LUN on a filer running ONTAP 7.2:

netapp01> lun show /vol/nexenta01/lun01/lun
        /vol/nexenta01/lun01/lun      10g (10737418240)   (r/w, online, mapped)

There are eight paths from our NetApp to our NexentaStor appliance, so the LUN appears eight times on the “qlc” adapter (lines 9-16 below):

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no

In NexentaStor “expert” mode, enable MPxIO for your Fibre Channel HBA (schedule this for a maintenance window, as it requires a reboot):

root@nexenta01:/volumes# stmsboot -L
stmsboot: MPXIO disabled
root@nexenta01:/volumes# stmsboot -e -D fp
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
updating //platform/i86pc/boot_archive
updating //platform/i86pc/amd64/boot_archive
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)

Note that this will not have any immediately noticable effect after rebooting:

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes

Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

However, in expert mode, you will now see the following:

root@nexenta01:/volumes# stmsboot -L
stmsboot: No STMS devices have been found

Enable ALUA (Asymmetric Logical Unit Access) on the initiator group on the NetApp:

netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
netapp01> igroup set nexenta01 alua yes
netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee(logged in on: 0b, 0d, vtic)
        ALUA: Yes

Reconfigure and re-scan your NexentaStor HBA; note that the LUN is now attached to “mpxio” where it was previously attached to “qlc”:

nmc@nexenta01:/$ lunsync -r
Cleanup obsolete (dangling) device links?  Yes
Re-scanning HBAs... done.
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c4t6*469d0  sd9       disk         10GB                  no      mpxio  60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

In NexentaStor expert mode, note that stmsboot now shows devices:

root@nexenta01:/volumes# stmsboot -L
non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c3t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0

You can now create a NexentaStor volume on your LUN.

Hat Tip to @complex on Twitter.

Reference: Is it possible to use I/O multipathing? How?

OpenSolaris 2008.05 on EC2 – Why 32-bit only?

Andy — Mon, 18 Aug 2008 22:42:13 +0000

Since Sun and Amazon removed the limit on the number of OpenSolaris 2008.05 instances able to run on EC2, I’ve been curious – and a little bothered – by the fact that the 2008.05 AMI is 32-bit only. Curious because OpenSolaris shouldn’t have any issues running on a 64-bit EC2 instance (there are other 64-bit OpenSolaris AMIs available on EC2, after all), and a little bothered because there have been long-standing reports of trouble running Solaris on 32-bit architectures, which makes me hesitant to invest much effort in a 32-bit OpenSolaris EC2 environment.

Well, perhaps a 64-bit AMI is forthcoming – I think this is still a beta program – and perhaps Sun’s just trying to save us a buck or two, since the cheapest 64-bit EC2 instance is four times as expensive per hour as the cheapest 32-bit instance.

Links 8/18/2008: CacheFS

Andy — Mon, 18 Aug 2008 22:16:27 +0000

Less known Solaris Features: CacheFS – Joerg Moellenkamp at c0t0d0s0.org offers another installment of his excellent Less known Solaris Features series. Of note: “In the recent days there was some discussion about the declaration of the End-of-Feature status for CacheFS which will lead to the announcement of the removal of CacheFS. After a few days of discussion the ARC decided in favour of the removal.” While I’ve never personally used CacheFS – and see no use case for it on the horizon – I’m not thrilled to see it slated for removal as it does sound like it serves an important role. Perhaps ADM or SAM-QFS will become more general to support this style of HSM as well in the future.

Capacity Limit for OpenSolaris on EC2 no more

Andy — Thu, 14 Aug 2008 21:27:43 +0000

According to a blog post on blogs.sun.com, the capacity limit for OpenSolaris 2008.05 on EC2 has been removed.

The blog entry makes it sound like you no longer need to register with Sun to use OpenSolaris on EC2, but that doesn’t appear to be the case – I only see the AMI in my private instances, and the details on the image seem to confirm this.

I’m running an instance right now; it did seem to take an extra long time for the image to come up, even by EC2 standards, and ec2-describe-instances showed the instance as “running” for a long time before I could connect using SSH. But it’s up now, ZFS root and all:

-bash-3.2# uname -a SunOS domU-12-31-38-00-28-35 5.11 snv_91 i86pc i386 i86xpv -bash-3.2# zfs list NAME USED AVAIL REFER MOUNTPOINT mnt 106K 147G 18K /mnt rpool 2.74G 6.86G 59K /rpool rpool/ROOT 2.73G 6.86G 18K /rpool/ROOT rpool/ROOT/opensolaris-1 2.73G 6.86G 2.72G legacy rpool/ROOT/opensolaris-1/opt 12.9M 6.86G 12.9M /opt rpool/export 37K 6.86G 19K /export rpool/export/home 18K 6.86G 18K /export/home swap 450M 9.89M 18K /swap swap/swapfile 450M 460M 16K -

One curiosity:

-bash-3.2# pkg image-update pkg: "image-update" option currently not supported on Amazon EC2. Please check out http://blogs.sun.com/ec2 for more details.

I must have missed those details…

Update: Here’s the reason why you can’t run “pkg image-update” from the Getting Started Guide for Amazon EC2: “pkg image-update – This command is currently not supported on Amazon EC2 since it modifies the kernel and ramdisk files resulting in non-bootable AMI. As we know, in the EC2 environment modifying the kernel and ramdisk is not permitted. In certain cases, if the user wants to enable this command, then the user can edit the /usr/bin/pkg file appropriately.” (Which makes sense.)

Link Dump, 7/17/2008

Andy — Thu, 17 Jul 2008 19:50:44 +0000

Elektronkind: OpenSolaris 2008.11 – A Preview For The Storage Admin – A look at upcoming storage technologies in OpenSolaris 2008.11, including ZFS, iSCSI, NDMP, COMSTAR, AVS and SAM-QFS. These products really set OpenSolaris apart from Linux distributions, although I wonder how official this list is, and have some doubts about the status of some of the projects. For example, there doesn’t appear to be much activity on the SAM-QFS OpenSolaris project, although maybe I’m just looking in the wrong place. (Seen at c0t0d0s0.org.)
Ruling: SCO owes Novell $2.54 million from SCO-Sun SVRX deal – Interesting excerpt: “Judge Kimball also reviewed SCO’s agreement with Sun and found that some of the terms exceeded SCO’s licensing authority. Through the agreement, SCO lifted the confidentiality provisions of Sun’s 1994 SVRX deal with Novell even though SCO was not permitted to do so without Novell’s explicit consent. The judge concluded that lifting of the SVRX confidentiality provisions was not incidental to a UnixWare license and was consequently not permissible. This raises some intriguing legal questions about OpenSolaris, which includes SVRX code that we now know SCO clearly had no right to let Sun open.” I wonder if we’ll be hearing more about this in the coming months.
Interview: IT consumerization and the future of higher ed – Another interesting piece on Ars Technica from today, an interview with Oren Sreebny of the University of Washington, whose best bits obliquely refer to the challenges of miasma computing and information security. Quotes: “Lately we’ve been looking at Google and Microsoft offerings for commodity stuff, and one of the things we deal with in some of our research [departments] is government regulations about ‘exporting munitions.’ So one of the manifestations of those government regulations is that you cannot store your data outside the US if you’re working on some types of government-funded projects. Google has said, ‘We can’t guarantee that anybody’s stuff in particular won’t be in a datacenter that’s located outside the US, so don’t bring that stuff to us,’ which is exactly what I’d be saying if I was them. So we have to figure out, as we start to move in those directions, what we do about that.” Also: “[Separate identity principals for people who are working on sensitive data] is an interesting conversation because, in many ways we’ve spent the last decade trying to integrate people’s identity, and do single-sign-on, and not make them have lots of separate accounts in separate places. And in many ways it really goes against the grain to step back from that, but maybe it’s time to do that.”

Hotlinks, 7/1/2008

Andy — Tue, 01 Jul 2008 19:08:03 +0000

The Hitz report – Robin Harris at StorageMojo on the Sun-NetApp lawsuit:

NetApp’s biggest misperception is that WAFL is somehow central to the success they are enjoying today. That was true about 10 years ago. Guys, your average F500 CIO today could care less about WAFL.

NetApp is growing because they offer a compelling value proposition of quality products, relevant services and worldwide support. WAFL certainly supports that, but as NetApp execs note much of their recent success is due to the integration software that NetApp now offers.

WAFL is a small piece of the picture. Sun could copy it line for line and still not have a quarter of what NetApp offers.

NetApp faces challenges. Storage commoditization threatens all vendors traditional 60% gross margins. The GX integration is problematic and the bottom line benefit uncertain. EMC’s move into cloud file services is a clever flanking strategy.

An interesting opinion summed up nicely, I think.
Saving and Restoring ZFS Snapshots to and from Amazon S3 – A ZFS to S3 workaround for the lack of persistent storage on EC2.

Monday 6/30/2008 Links

Andy — Mon, 30 Jun 2008 23:05:37 +0000

Prediction: Citrix will drop the open source Xen hypervisor for Hyper-V. The rest of the open source world drops Xen for KVM. – Lengthy speculation about the future of Xen now that Hyper-V is out. If this turns out to be correct, I think it leaves Sun in a particularly awkward spot, given that the work they’ve done on integrating Xen with Solaris. (Seen at virtualization.info and vinternals.)

No Luck with a Quick-n-Dirty BFU of SXCE 79 on EC2

Andy — Wed, 25 Jun 2008 14:19:10 +0000

For grins, I tried a quick-and-dirty BFU of a SXCE 79 instance running on EC2 to the latest nightly build this morning. I roughly followed Ben Rockwood’s BFU instructions and didn’t do anything to resolve conflicts beyond running acr. On reboot, it looks like the system panicked – I presume the reason is probably somewhere in here. Console dump after the jump for the curious.

v3.0.3-rc5-8.1.14.e chgset 'unavailable ' WARNING: Found hypervisor version: v3.0.3-rc5-8.1.14.e but need at least version v3.0.4 SunOS Release 5.11 Version snv_79 32-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. NOTICE: xdf@0: failed to read feature-barrier Hostname: ip-10-250-11-159 Reading ZFS config: *done. ip-10-250-11-159 console login: syslogd: line 45: WARNING: loghost could not be resolved syncing file systems... done rebooting... v3.0.3-rc5-8.1.14.e chgset 'unavailable ' WARNING: Found hypervisor version: v3.0.3-rc5-8.1.14.e but need at least version v3.0.4 SunOS Release 5.11 Version snv_79 32-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. NOTICE: xdf@0: failed to read feature-barrier WARNING: file system 'ctfs' version mismatch WARNING: Cannot mount /system/contract /kernel/fs/procfs: undefined symbol 'schedctl_set_cidpri' WARNING: mod_load: cannot load module 'procfs' WARNING: Cannot mount /proc WARNING: file system 'mntfs' version mismatch WARNING: Cannot mount /etc/mnttab /kernel/fs/tmpfs: undefined symbol 'choose_addr' WARNING: mod_load: cannot load module 'tmpfs' WARNING: Cannot mount /etc/svc/volatile WARNING: file system 'objfs' version mismatch WARNING: Cannot mount /system/object WARNING: file system 'sharefs' version mismatch WARNING: Cannot mount /etc/dfs/sharetab /kernel/fs/procfs: undefined symbol 'schedctl_set_cidpri' WARNING: mod_load: cannot load module 'procfs' /kernel/drv/ip: undefined symbol 'uioamove' /kernel/drv/ip: undefined symbol 'dl_errstr' /kernel/drv/ip: undefined symbol 'dl_primstr' /kernel/drv/ip: undefined symbol 'dhcifname' WARNING: mod_load: cannot load module 'ip' /kernel/fs/sockfs: undefined symbol 'uioafini' /kernel/fs/sockfs: undefined symbol 'uioainit' /kernel/fs/sockfs: undefined symbol 'uioasync' /kernel/fs/sockfs: undefined symbol 'kssl_handle_mblk' /kernel/fs/sockfs: undefined symbol 'sctp_disconnect' /kernel/fs/sockfs: undefined symbol 'sctp_getsockname' /kernel/fs/sockfs: undefined symbol 'nd_free' /kernel/fs/sockfs: undefined symbol 'nd_load' /kernel/fs/sockfs: undefined symbol 'sctp_create' /kernel/fs/sockfs: undefined symbol 'sctp_close' /kernel/fs/sockfs: undefined symbol 'sctp_listen' /kernel/fs/sockfs: undefined symbol 'sctp_recvd' /kernel/fs/sockfs: undefined symbol 'mi_mpprintf' /kernel/fs/sockfs: undefined symbol 'sctp_alloc_hdr' /kernel/fs/sockfs: undefined symbol 'sctp_get_opt' /kernel/fs/sockfs: undefined symbol 'tcp_wput' /kernel/fs/sockfs: undefined symbol 'mi_sprintf' /kernel/fs/sockfs: undefined symbol 'sctp_bind' /kernel/fs/sockfs: undefined symbol 'udp_wput' /kernel/fs/sockfs: undefined symbol 'sctp_getpeername' /kernel/fs/sockfs: undefined symbol 'sctp_sendmsg' /kernel/fs/sockfs: undefined symbol 'sctp_set_opt' /kernel/fs/sockfs: undefined symbol 'sctp_connect' WARNING: mod_load: cannot load module 'sockfs' WARNING: sockfs: unable to resolve dependency, module 'drv/ip' not found /kernel/sys/c2audit: undefined symbol 'getsonode' WARNING: mod_load: cannot load module 'c2audit' WARNING: c2audit: unable to resolve dependency, module 'fs/sockfs' not found /kernel/misc/strplumb: undefined symbol 'dl_attach' /kernel/misc/strplumb: undefined symbol 'dl_phys_addr' /kernel/misc/strplumb: undefined symbol 'dhcacklen' /kernel/misc/strplumb: undefined symbol 'dhcifname' /kernel/misc/strplumb: undefined symbol 'dl_bind' WARNING: mod_load: cannot load module 'strplumb' panic[cpu0]/thread=f502c3a0: mod_hold_stub: Couldn't load stub module misc/strplumb f5056c84 genunix:mod_hold_stub+161 (f5006070, 1, f4d38f) f5056ca0 unix:stubs_common_code+9 () syncing file systems... done skipping system dump - no dump device configured rebooting...

EMC’s Flash Blind Spot

Andy — Fri, 20 Jun 2008 13:33:02 +0000

Chuck’s got another, uh, thought-provoking blog post up, More Examples Of Why Server Vendors Just Don’t Get Storage, surely intended to ruffle a few feathers. And he does raise some really good points: Most server vendors need more of an SSD strategy than just making a flash drive an option (it’s how you use it, not that you have it!). And as big a fan as I am of ZFS and Sun’s storage options in general, to win in the “enterprise” (and not just, say, HPC) Sun needs to pull everything together into Solaris (from OpenSolaris) and make it less of a DIY operation.

But here’s where I think EMC’s missing something – their flash blind spot, if you will: Price-performance. Sure, I recognize that EMC has chosen to enter the market from where they compete best – the high end. And I realize it’s inevitable that flash will move down-market in EMC’s arrays (really: flash in the CLARiiON line is a “when,” not an “if”). But EMC’s flash options are right now way too expensive for most shops, while, on the other hand, there’s a good probability I’ll have servers with SSDs in them by then end of this year. I’ll just need a strategy on how best to use them with my operating systems and applications to maximize their benefit. Unfortunately for HP and Dell, they’re not offering a strategy that I know of, they’re just offering the drives. Sun, on the other hand, has a good use case: ZIL and L2ARC on flash – now it just needs to get back-ported from OpenSolaris into Solaris.

Flash – A tale of three companies: EMC, NetApp and Sun

Andy — Fri, 13 Jun 2008 14:17:56 +0000

There’s been a lot of noise from the storage industry about flash recently – in particular, noise from EMC and Sun, both of whom recently announced storage products using flash, EMC in January and Sun earlier this month. Below are my thoughts on what EMC and Sun are doing, as well as what NetApp might do. Since I see a fair amount of visitors from all three companies here, if I’ve got something about your employer wrong, please correct me in the comments.

EMC’s approach seems to target the very high-end. I don’t have any EMC gear to play with, but if I understand correctly, their flash drives are pretty much plug and play with their DMX4 arrays, matching the physical form factor of their existing disk drives. Once the drives are in the array, you can do anything with them that you do with standard drives – these drives are just a lot faster. Sounds very much like a no-holds-barred-if-you-need-speed-this-is-how-we’ll-give-it-to-you – with a price to match the performance.

I’d love to have some of the EMC flash gear, but I doubt I’ll ever see this iteration. I work with NetApp storage arrays (at least for this budget cycle…), and from what I understand about the pricing of the EMC kit, it’s targeted way over the head of my employer. So will I see something similar from NetApp? Unfortunately, it doesn’t sound like they’ll have anything out in the near future. A couple years ago, Dave Hitz was almost dismissive of flash, answering “I don’t think so” and “I don’t care” to the question of whether flash would replace disk drives – and he has some good points, mainly that the storage and management layer is ultimately more important than the physical media one uses. More recently, Jay Kidd, NetApp’s chief marketing officer was quoted by ZDNet as saying that flash will become “an integral part” of enterprise storage – but there are no flash product announcements from NetApp that I’m aware of.

Given NetApp’s storage system architecture, this seems particularly unfortunate to me. SSDs seem like they could fit very nicely as a cache device between the controller and the conventional disk drives. I imagine that just a couple SSDs could give a substantial performance boost to an array for a very moderate price. Additionally, I don’t see a reason why NetApp couldn’t allow you to make an aggregate out of a shelf of flash drives for servers that need absolute top speed. Will they do either of these? I hope both, but we’ll have to wait and see.

(With their newly-announced Performance Acceleration Modules, NetApp seems to be acknowledging the value of an additional caching layer, although the PAM is for read performance only.)

Sun’s flash announcement is focused on the use of flash drives in their server hardware. Specifically, according to an Intel Developer Forum presentation, at least one use case Sun is targeting is ZFS, where direct-attached flash would be used in a cache role for specific ZFS reads and writes (in the second-level Adaptive Replacement Cache – L2ARC – and ZFS Intent Log – ZIL -pools – see page 11 of the above slides). The presentation suggests a possible configuration, comparing a hypothetical Sun X4450 with seven 10k RPM SAS drives and one with two SSDs and five 4200 RPM (did they mean 7200 RPM?) SATA drives. For almost the same price, the SSD/SATA system delivers substantially better Read IOPS and slightly better write IOPS.

Which is all good and nice – I do love a slide with reassuring bar graphs – but Sun won’t be shipping servers with flash drives until later this year. Additionally, the support for separating the L2ARC and ZIL is only in OpenSolaris, not Solaris – I haven’t seen word on when it will be backported. I also wonder about what effect single ZIL and L2ARC drives, as used in the scenario in the presentation, would have on a machine’s availability; Sun could presumably address this with some “best practices” documentation by the time they ship servers with SSDs. All that aside, I like what I see from Sun so far, particularly that they’ve found a use case for SSDs at a reasonable price point.