thinking sysadmin » solaris

NexentaStor in front of a NetApp FC LUN using MPxIO

Andy — Fri, 28 May 2010 17:35:13 +0000

Create a Fibre Channel LUN on your NetApp and map it to your NexentaStor machine (I’m using version 3.0.2 in this example). For this example, I’ve created a 10GB LUN on a filer running ONTAP 7.2:

netapp01> lun show /vol/nexenta01/lun01/lun
        /vol/nexenta01/lun01/lun      10g (10737418240)   (r/w, online, mapped)

There are eight paths from our NetApp to our NexentaStor appliance, so the LUN appears eight times on the “qlc” adapter (lines 9-16 below):

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no

In NexentaStor “expert” mode, enable MPxIO for your Fibre Channel HBA (schedule this for a maintenance window, as it requires a reboot):

root@nexenta01:/volumes# stmsboot -L
stmsboot: MPXIO disabled
root@nexenta01:/volumes# stmsboot -e -D fp
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
updating //platform/i86pc/boot_archive
updating //platform/i86pc/amd64/boot_archive
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)

Note that this will not have any immediately noticable effect after rebooting:

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes

Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

However, in expert mode, you will now see the following:

root@nexenta01:/volumes# stmsboot -L
stmsboot: No STMS devices have been found

Enable ALUA (Asymmetric Logical Unit Access) on the initiator group on the NetApp:

netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
netapp01> igroup set nexenta01 alua yes
netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee(logged in on: 0b, 0d, vtic)
        ALUA: Yes

Reconfigure and re-scan your NexentaStor HBA; note that the LUN is now attached to “mpxio” where it was previously attached to “qlc”:

nmc@nexenta01:/$ lunsync -r
Cleanup obsolete (dangling) device links?  Yes
Re-scanning HBAs... done.
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c4t6*469d0  sd9       disk         10GB                  no      mpxio  60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

In NexentaStor expert mode, note that stmsboot now shows devices:

root@nexenta01:/volumes# stmsboot -L
non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c3t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0

You can now create a NexentaStor volume on your LUN.

Hat Tip to @complex on Twitter.

Reference: Is it possible to use I/O multipathing? How?

OpenSolaris 2008.05 on EC2 – Why 32-bit only?

Andy — Mon, 18 Aug 2008 22:42:13 +0000

Since Sun and Amazon removed the limit on the number of OpenSolaris 2008.05 instances able to run on EC2, I’ve been curious – and a little bothered – by the fact that the 2008.05 AMI is 32-bit only. Curious because OpenSolaris shouldn’t have any issues running on a 64-bit EC2 instance (there are other 64-bit OpenSolaris AMIs available on EC2, after all), and a little bothered because there have been long-standing reports of trouble running Solaris on 32-bit architectures, which makes me hesitant to invest much effort in a 32-bit OpenSolaris EC2 environment.

Well, perhaps a 64-bit AMI is forthcoming – I think this is still a beta program – and perhaps Sun’s just trying to save us a buck or two, since the cheapest 64-bit EC2 instance is four times as expensive per hour as the cheapest 32-bit instance.

Links 8/18/2008: CacheFS

Andy — Mon, 18 Aug 2008 22:16:27 +0000

Less known Solaris Features: CacheFS – Joerg Moellenkamp at c0t0d0s0.org offers another installment of his excellent Less known Solaris Features series. Of note: “In the recent days there was some discussion about the declaration of the End-of-Feature status for CacheFS which will lead to the announcement of the removal of CacheFS. After a few days of discussion the ARC decided in favour of the removal.” While I’ve never personally used CacheFS – and see no use case for it on the horizon – I’m not thrilled to see it slated for removal as it does sound like it serves an important role. Perhaps ADM or SAM-QFS will become more general to support this style of HSM as well in the future.

Monday 6/30/2008 Links

Andy — Mon, 30 Jun 2008 23:05:37 +0000

Prediction: Citrix will drop the open source Xen hypervisor for Hyper-V. The rest of the open source world drops Xen for KVM. – Lengthy speculation about the future of Xen now that Hyper-V is out. If this turns out to be correct, I think it leaves Sun in a particularly awkward spot, given that the work they’ve done on integrating Xen with Solaris. (Seen at virtualization.info and vinternals.)

EMC’s Flash Blind Spot

Andy — Fri, 20 Jun 2008 13:33:02 +0000

Chuck’s got another, uh, thought-provoking blog post up, More Examples Of Why Server Vendors Just Don’t Get Storage, surely intended to ruffle a few feathers. And he does raise some really good points: Most server vendors need more of an SSD strategy than just making a flash drive an option (it’s how you use it, not that you have it!). And as big a fan as I am of ZFS and Sun’s storage options in general, to win in the “enterprise” (and not just, say, HPC) Sun needs to pull everything together into Solaris (from OpenSolaris) and make it less of a DIY operation.

But here’s where I think EMC’s missing something – their flash blind spot, if you will: Price-performance. Sure, I recognize that EMC has chosen to enter the market from where they compete best – the high end. And I realize it’s inevitable that flash will move down-market in EMC’s arrays (really: flash in the CLARiiON line is a “when,” not an “if”). But EMC’s flash options are right now way too expensive for most shops, while, on the other hand, there’s a good probability I’ll have servers with SSDs in them by then end of this year. I’ll just need a strategy on how best to use them with my operating systems and applications to maximize their benefit. Unfortunately for HP and Dell, they’re not offering a strategy that I know of, they’re just offering the drives. Sun, on the other hand, has a good use case: ZIL and L2ARC on flash – now it just needs to get back-ported from OpenSolaris into Solaris.

Flash – A tale of three companies: EMC, NetApp and Sun

Andy — Fri, 13 Jun 2008 14:17:56 +0000

There’s been a lot of noise from the storage industry about flash recently – in particular, noise from EMC and Sun, both of whom recently announced storage products using flash, EMC in January and Sun earlier this month. Below are my thoughts on what EMC and Sun are doing, as well as what NetApp might do. Since I see a fair amount of visitors from all three companies here, if I’ve got something about your employer wrong, please correct me in the comments.

EMC’s approach seems to target the very high-end. I don’t have any EMC gear to play with, but if I understand correctly, their flash drives are pretty much plug and play with their DMX4 arrays, matching the physical form factor of their existing disk drives. Once the drives are in the array, you can do anything with them that you do with standard drives – these drives are just a lot faster. Sounds very much like a no-holds-barred-if-you-need-speed-this-is-how-we’ll-give-it-to-you – with a price to match the performance.

I’d love to have some of the EMC flash gear, but I doubt I’ll ever see this iteration. I work with NetApp storage arrays (at least for this budget cycle…), and from what I understand about the pricing of the EMC kit, it’s targeted way over the head of my employer. So will I see something similar from NetApp? Unfortunately, it doesn’t sound like they’ll have anything out in the near future. A couple years ago, Dave Hitz was almost dismissive of flash, answering “I don’t think so” and “I don’t care” to the question of whether flash would replace disk drives – and he has some good points, mainly that the storage and management layer is ultimately more important than the physical media one uses. More recently, Jay Kidd, NetApp’s chief marketing officer was quoted by ZDNet as saying that flash will become “an integral part” of enterprise storage – but there are no flash product announcements from NetApp that I’m aware of.

Given NetApp’s storage system architecture, this seems particularly unfortunate to me. SSDs seem like they could fit very nicely as a cache device between the controller and the conventional disk drives. I imagine that just a couple SSDs could give a substantial performance boost to an array for a very moderate price. Additionally, I don’t see a reason why NetApp couldn’t allow you to make an aggregate out of a shelf of flash drives for servers that need absolute top speed. Will they do either of these? I hope both, but we’ll have to wait and see.

(With their newly-announced Performance Acceleration Modules, NetApp seems to be acknowledging the value of an additional caching layer, although the PAM is for read performance only.)

Sun’s flash announcement is focused on the use of flash drives in their server hardware. Specifically, according to an Intel Developer Forum presentation, at least one use case Sun is targeting is ZFS, where direct-attached flash would be used in a cache role for specific ZFS reads and writes (in the second-level Adaptive Replacement Cache – L2ARC – and ZFS Intent Log – ZIL -pools – see page 11 of the above slides). The presentation suggests a possible configuration, comparing a hypothetical Sun X4450 with seven 10k RPM SAS drives and one with two SSDs and five 4200 RPM (did they mean 7200 RPM?) SATA drives. For almost the same price, the SSD/SATA system delivers substantially better Read IOPS and slightly better write IOPS.

Which is all good and nice – I do love a slide with reassuring bar graphs – but Sun won’t be shipping servers with flash drives until later this year. Additionally, the support for separating the L2ARC and ZIL is only in OpenSolaris, not Solaris – I haven’t seen word on when it will be backported. I also wonder about what effect single ZIL and L2ARC drives, as used in the scenario in the presentation, would have on a machine’s availability; Sun could presumably address this with some “best practices” documentation by the time they ship servers with SSDs. All that aside, I like what I see from Sun so far, particularly that they’ve found a use case for SSDs at a reasonable price point.

6/11/2008 Links

Andy — Wed, 11 Jun 2008 14:04:43 +0000

Adam Leventhal’s Weblog: Flash, Hybrid Pools, and Future Storage – Excerpts from a forthcoming ACM article on hybrid (flash-disk) storage pools, including: “Flash should be viewed not as a replacement for existing storage, but rather as a means to enhance it. [...] By combining the use of flash as an intent-log to reduce write latency with flash as a cache to reduce read latency, we can create a system that performs far better and consumes less power than other system of similar cost.” Which perhaps could be thought of as a dig at EMC’s flash implementation, although I doubt that’s how the author intended it. (Seen at c0t0d0s0.)

Reading List, 6/10/2008, Afternoon Edition

Andy — Tue, 10 Jun 2008 23:30:55 +0000

Jonathan’s Blog: Anything But a Flash in the Pan – Jonathan Schwartz on Sun’s soon-to-be-released (late this year) flash drives. Quote: “ZFS will transparently incorporate Flash into the storage hierarchy of a running system, using the microprocessor cache for the most performance sensitive tasks, DRAM for the next, then Flash, then disk (then ultimately tape).” Speaking of tiered storage, I wonder how flash drives would work within a SAM-QFS implementation – and if SAM-QFS is destined to wither in the shadow of ZFS: Sure it’s open source now, but where is it going these days?

Links, 6/9/2008

Andy — Mon, 09 Jun 2008 23:24:28 +0000

Performance Isolation of a Misbehaving Virtual Machine with Xen, VMware and Solaris Containers – A paper comparing the ability of VMware (Workstation), Xen and Solaris Containers to isolate virtual machines from each other with regards to performance. Quick summary: VMware and Xen go one-two in protecting one virtual machine from another, and Solaris Zones lag far behind, a definite trade-off to consider if you’re inclined to use zones for their higher density. I would expect similar results to what was reported with Solaris if the authors had also looked at OpenVZ and FreeBSD Jails – in fact, (anecdotally) I’ve seen generally the same behavior with OpenVZ and Jails, although OpenVZ does give you a lot of knobs to adjust (if performance isolation is a concern for you with OpenVZ, you may want to look at all the options it gives you before making decisions about its suitability). It would have been nice to see more details about the Solaris configuration used in the paper, and to have had the work repeated with a recent OpenSolaris build – substantial work appears to be being done in this area (for example, Improved Resource Management and Zones Integration). Seen at blog.scottlowe.org.

OpenSolaris and EC2: Control Issues, Anyone?

Andy — Mon, 02 Jun 2008 03:58:19 +0000

Generally, I’m a fan of Sun Microsystems. For the most part, I like their hardware and their software – and their best products show real innovation and willingness to take risks. I’m also a fan of Amazon’s EC2 product, so the announcement that Sun would be officially bundling OpenSolaris for EC2 was great news. Unfortunately, it seems that after all the hullabaloo, Sun doesn’t really want to make it that easy for you to actually use OpenSolaris on EC2, by managing access to it like a control freak would.

I could throw in some snarkiness here about the titanic struggles that must be going on inside Sun between those that do and don’t “get it” on the OpenSolaris team, or pontificate that Sun just doesn’t get Open Source, period. Instead, I’ll leave you with a single thought: If one of EC2’s principal attractions is reducing friction in creating, prototyping, deploying and scaling, does Sun really expect to be successful by throwing up barriers to entry?

(Until I hear back from Sun – “our technical team will review your requirements and get back to you shortly” – guess I’ll just fire up a Fedora AMI. Oh well.)