thinking sysadmin

There is a long-standing bug in (Open)Solaris and derivatives (including NexentaStor) that breaks Active Directory interoperability:

Beginning with Windows Server 2003, Active Directory supports VLV searches. Every VLV search request must be accompanied by 2 request controls: the SSS control and the VLV control. However, Active Directory imposes some general criteria on the SSS control:

1. Cannot sort based on more than one sort keys/attributes.
2. Cannot sort based on the “distinguishedName” attribute (presumably Microsoft does not use the “DN” attribute).
3. Cannot sort based on a constructed attribute (presumably an attribute not stored on Active Directory).

Unfortunately, Solaris LDAP clients use 2 sort keys/attributes: “cn” and “uid” in the SSS control. Subsequently, when dumping a container or a naming database, Solaris LDAP clients would receive LDAP_UNAVAILABLE_CRITICAL_EXTENSION.

$ ldaplist passwd
ldaplist: Object not found (LDAP ERROR (12): Unavailable critical extension.)

This issue has been detailed elsewhere, including at utexas.edu. There appear to be at least four solutions:

1. Wait for the fix from Sun Oracle to reach the light of day: this bug was apparently fixed in SNV 144. (I expect the fix is out in Solaris 11 Express now, but have not tested this myself.)
2. Apply the hotfix in Microsoft’s KB886683 to your domain controllers, which will disable VLV.
3. Run separate ADAM instances with VLV disabled, and point your Solaris machines at them instead of directly at your domain controllers. From the blog post linked above, it sounds like the University of Texas chose this route.
4. Use OpenLDAP as a proxy in front of Active Directory; configure your Solaris machines to use the proxies instead of Active Directory servers. This is the solution detailed in this blog post.

Read the rest of this entry »

1. Create a Fibre Channel LUN on your NetApp and map it to your NexentaStor machine (I’m using version 3.0.2 in this example). For this example, I’ve created a 10GB LUN on a filer running ONTAP 7.2:

   netapp01> lun show /vol/nexenta01/lun01/lun
           /vol/nexenta01/lun01/lun      10g (10737418240)   (r/w, online, mapped)
   

   There are eight paths from our NetApp to our NexentaStor appliance, so the LUN appears eight times on the “qlc” adapter (lines 9-16 below):

   nmc@nexenta01:/$ lunsync
   Cleanup obsolete (dangling) device links?  Yes
   Re-enumerating LUNs... done.
   
   nmc@nexenta01:/$ show lun
   LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
   c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
   c1t0d0      sd128     cdrom        No Media              no      ata    -
   c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
   c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
   c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
   c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
   c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
   c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
   c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
   c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
   syspo~/swap           zvol         1.0GB      syspool    no
   

Read the rest of this entry »

Since Sun and Amazon removed the limit on the number of OpenSolaris 2008.05 instances able to run on EC2, I’ve been curious – and a little bothered – by the fact that the 2008.05 AMI is 32-bit only. Curious because OpenSolaris shouldn’t have any issues running on a 64-bit EC2 instance (there are other 64-bit OpenSolaris AMIs available on EC2, after all), and a little bothered because there have been long-standing reports of trouble running Solaris on 32-bit architectures, which makes me hesitant to invest much effort in a 32-bit OpenSolaris EC2 environment.

Well, perhaps a 64-bit AMI is forthcoming – I think this is still a beta program – and perhaps Sun’s just trying to save us a buck or two, since the cheapest 64-bit EC2 instance is four times as expensive per hour as the cheapest 32-bit instance.

• Less known Solaris Features: CacheFS – Joerg Moellenkamp at c0t0d0s0.org offers another installment of his excellent Less known Solaris Features series. Of note: “In the recent days there was some discussion about the declaration of the End-of-Feature status for CacheFS which will lead to the announcement of the removal of CacheFS. After a few days of discussion the ARC decided in favour of the removal.” While I’ve never personally used CacheFS – and see no use case for it on the horizon – I’m not thrilled to see it slated for removal as it does sound like it serves an important role. Perhaps ADM or SAM-QFS will become more general to support this style of HSM as well in the future.

According to a blog post on blogs.sun.com, the capacity limit for OpenSolaris 2008.05 on EC2 has been removed.

The blog entry makes it sound like you no longer need to register with Sun to use OpenSolaris on EC2, but that doesn’t appear to be the case – I only see the AMI in my private instances, and the details on the image seem to confirm this.
Read the rest of this entry »

• Elektronkind: OpenSolaris 2008.11 – A Preview For The Storage Admin – A look at upcoming storage technologies in OpenSolaris 2008.11, including ZFS, iSCSI, NDMP, COMSTAR, AVS and SAM-QFS. These products really set OpenSolaris apart from Linux distributions, although I wonder how official this list is, and have some doubts about the status of some of the projects. For example, there doesn’t appear to be much activity on the SAM-QFS OpenSolaris project, although maybe I’m just looking in the wrong place. (Seen at c0t0d0s0.org.)
• Ruling: SCO owes Novell $2.54 million from SCO-Sun SVRX deal – Interesting excerpt: “Judge Kimball also reviewed SCO’s agreement with Sun and found that some of the terms exceeded SCO’s licensing authority. Through the agreement, SCO lifted the confidentiality provisions of Sun’s 1994 SVRX deal with Novell even though SCO was not permitted to do so without Novell’s explicit consent. The judge concluded that lifting of the SVRX confidentiality provisions was not incidental to a UnixWare license and was consequently not permissible. This raises some intriguing legal questions about OpenSolaris, which includes SVRX code that we now know SCO clearly had no right to let Sun open.” I wonder if we’ll be hearing more about this in the coming months.
• Interview: IT consumerization and the future of higher ed – Another interesting piece on Ars Technica from today, an interview with Oren Sreebny of the University of Washington, whose best bits obliquely refer to the challenges of miasma computing and information security. Quotes: “Lately we’ve been looking at Google and Microsoft offerings for commodity stuff, and one of the things we deal with in some of our research [departments] is government regulations about ‘exporting munitions.’ So one of the manifestations of those government regulations is that you cannot store your data outside the US if you’re working on some types of government-funded projects. Google has said, ‘We can’t guarantee that anybody’s stuff in particular won’t be in a datacenter that’s located outside the US, so don’t bring that stuff to us,’ which is exactly what I’d be saying if I was them. So we have to figure out, as we start to move in those directions, what we do about that.” Also: “[Separate identity principals for people who are working on sensitive data] is an interesting conversation because, in many ways we’ve spent the last decade trying to integrate people’s identity, and do single-sign-on, and not make them have lots of separate accounts in separate places. And in many ways it really goes against the grain to step back from that, but maybe it’s time to do that.”

• The Hitz report – Robin Harris at StorageMojo on the Sun-NetApp lawsuit:
  

  NetApp’s biggest misperception is that WAFL is somehow central to the success they are enjoying today. That was true about 10 years ago. Guys, your average F500 CIO today could care less about WAFL.

  NetApp is growing because they offer a compelling value proposition of quality products, relevant services and worldwide support. WAFL certainly supports that, but as NetApp execs note much of their recent success is due to the integration software that NetApp now offers.

  WAFL is a small piece of the picture. Sun could copy it line for line and still not have a quarter of what NetApp offers.

  NetApp faces challenges. Storage commoditization threatens all vendors traditional 60% gross margins. The GX integration is problematic and the bottom line benefit uncertain. EMC’s move into cloud file services is a clever flanking strategy.

  An interesting opinion summed up nicely, I think.

• Saving and Restoring ZFS Snapshots to and from Amazon S3 – A ZFS to S3 workaround for the lack of persistent storage on EC2.

• Prediction: Citrix will drop the open source Xen hypervisor for Hyper-V. The rest of the open source world drops Xen for KVM. – Lengthy speculation about the future of Xen now that Hyper-V is out. If this turns out to be correct, I think it leaves Sun in a particularly awkward spot, given that the work they’ve done on integrating Xen with Solaris. (Seen at virtualization.info and vinternals.)

For grins, I tried a quick-and-dirty BFU of a SXCE 79 instance running on EC2 to the latest nightly build this morning. I roughly followed Ben Rockwood’s BFU instructions and didn’t do anything to resolve conflicts beyond running acr. On reboot, it looks like the system panicked – I presume the reason is probably somewhere in here. Console dump after the jump for the curious.
Read the rest of this entry »

Chuck’s got another, uh, thought-provoking blog post up, More Examples Of Why Server Vendors Just Don’t Get Storage, surely intended to ruffle a few feathers. And he does raise some really good points: Most server vendors need more of an SSD strategy than just making a flash drive an option (it’s how you use it, not that you have it!). And as big a fan as I am of ZFS and Sun’s storage options in general, to win in the “enterprise” (and not just, say, HPC) Sun needs to pull everything together into Solaris (from OpenSolaris) and make it less of a DIY operation.
Read the rest of this entry »