Makes me wonder how much engineering that I never thought about goes into designing disk shelves to keep drives insulated from vibrations. The Fishworks analytics interface is dazzling – wish I had that yesterday when I was looking at a possible Exchange I/O performance issue with perfmon…

As I noted in my “quick walk” post’s comments, I had difficulty getting LDAP working initially on my corporate Active Directory network. The crux for me turned out to be getting the LDAP Schema Definitions correct. Here are the settings that worked correctly for me, authenticating against an AD instance with the schema extended by Microsoft’s Services for Unix add-on (other LDAP schemata will, of course, need different mappings):

USERS
Search descriptor: Don’t leave this blank – according to the Fishworks documentation this “sets the LDAP search descriptor, attribute mappings and object class mappings for users and groups. By default, the search descriptor for users is ou=people,dc=example,dc=com, and for groups is ou=group,dc=example,dc=com” – so what you enter will be site-specific.

Attribute mappings:

• uid=msSFU30Name
• uidNumber=msSFU30UidNumber
• gidNumber=msSFU30GidNumber

Object class mappings:

• posixAccount=User

GROUPS
Search descriptor: Again, don’t leave this blank – enter the appropriate value for your site.

Attribute mappings:

• gidNumber=msSFU30GidNumber
• uniqueMember=msSFU30PosixMember

Object class mappings:

• posixGroup=group

How did I know that the schema definition mappings were the problem? The logs gave it away: Maintenance -> Logs -> System, where I saw messages similar to the following: “libsldap: Status: 0 Mesg: Unable to set value: schema map already existed for ‘User’.”

How did I know that I had the schema definitions working? Share settings that I had created using numeric UIDs and GIDs automatically became mapped to the correct user and group names.

I’ll update this post if I find additional configuration that may be necessary.

That virtualized Fishworks appliance got me thinking: What if you combined this with this? Yeah, managing Elastic Block Store devices would require some changes, but, if you needed a NAS for your EC2 instances…

That was easy...

Below is a quick walkthrough of my experience booting and installing the Fishworks VMware appliance; my thoughts follow.

Fishworks Boot Screen

Initial network configuration

Yes, Virginia, there is a command line - but not a conventional shell

Web interface login screen

Network configuration

Configure DNS - pretty basic stuff

NTP is a good thing

Name service configuration choices

Active Directory integration for CIFS shares is straightforward

Fishworks phone home...

Storage configuration made fast and easy

All done!

Thoughts: First off, as a VMware appliance and an introduction to Fishworks, this virtual machine is extremely well done; it gives users a low “activation energy” method to try out Fishworks. There’s a lot more effort involved getting a sixty-day loaner from Sun, as neat as that program is – even the logistics of racking and cabling the machine are relatively expensive in terms of time, after all. Of course, there are limitations inherent to a virtual machine; for example, I’d love to see how VMware ESX performs using a hybrid storage pool, but only real hardware can answer that. However, for most of my questions about Fishworks, the VM fits the bill.

Once you hit the VM’s virtual power button, it boots up quickly – faster than most OpenSolaris machines, subjectively – and the initial configuration is straightforward. Although I didn’t try it, it does appear that the machine can be completely configured from the command line, either at the console or over SSH. Once the system is configured, you have HTTPS or SSH remote administration access; however, SSH does not give you a standard Solaris shell, such as sh or bash. Instead, you get an interface very similar to Solaris’ zonecfg – a command line interface with multiple modes. (I suspect that a shell can be launched on the machine somewhere, somehow – it may even be documented – but I wouldn’t be stunned if it’s “unsupported.”)

The six-step configuration process via the web interface is smooth and consistent, and not particularly time-consuming – you get this sense that this is truly intended to be an appliance. My only complaints: The online docs were not immediately helpful (seriously – links to Wikipedia?), and when something does go wrong, you don’t get much more immediate help than this:

Uh-oh - how do I get to those log files?

By far, the best part of the configuration interface is step six – the storage configuration. You see a list of your configuration choices, with availability, performance and capacity rated. The left-hand column shows a pie-chart breakdown of your storage utilization, with the tiny “Reserved” slice surely being a jab at NetApp. (Well, maybe not – but as a NetApp user, it sure got my attention.) Simply pick your RAID type, click “commit” and you’re on your way. Genius.

All in all, very impressive. Makes me wish even more that I had a pair of 7410s to put behind ESX. And Exchange. And SQL Server. And my CIFS and NFS users… you get the idea.

• NFS v3 and v4
• CIFS
• iSCSI
• HTTP
• WebDAV
• FTP
• RAID-Z (RAID-5 and RAID-6), Mirrored, and Striped disk configurations
• Unlimited Read-only and Read-write Snapshots, with Snapshot Schedules
• Built-in Data Compression
• Remote Replication of data for Disaster Recovery
• Active-Active Clustering (in the Sun Storage 7410) for High Availability
• Thin Provisioning of iSCSI LUNs
• Virus Scanning and Quarantine
• NDMP Backup and Restore

A few comments: Looks like all of the usual ZFS features are there, with a few additions – in particular, I wasn’t aware that the virus scanning project existed, and I didn’t know that NDMP was far enough along to be included in a production release. Additionally, from looking at various Sun blogs, I believe that the remote replication feature is zfs send/recv, not AVS. Finally, from the nomenclature (“2008.11″), I’d guess that the software is based on the forthcoming release of OpenSolaris, not the recently released update to Solaris 10.

What’s missing? Off the top of my head:

• Fibre Channel – COMSTAR is coming, presumably.
• HSM – ADM is also presumably on its way in a future release.
• HCL entries for various products like VMware, but again, I have to believe that Sun is working hard on this as well.

My first impression from the launch materials: Neat, but the price seems high. Looking at list prices for the models, and doing some quick calculations for RAID-Z2 configurations with at least one hot spare, the price per usable TB ranges from $3999 and $3933 for a 7210 with 250GB and 1TB drives, respectively, to $11,209 for a single head 7410. Compare this to the hardware that the 7210/250GB is based on, the X4540, where you pay $2513.71 per usable TB. Now, as far as I know, Sun isn’t offering flash drives with their non-Fishworks hardware, so it makes direct comparisons of the price of the Fishworks Special Sauce impossible for most of the rest of the line, but that may change later this fall.

Other thoughts: Why not use the UltraSPARC T2 instead of Opterons? I’d expect better performance from the UltraSPARCs, especially when using 10GbE. Is it a cost issue?

One final note: Sun is making their simulator (a VMware image) available for download – nice touch.

The blog entry makes it sound like you no longer need to register with Sun to use OpenSolaris on EC2, but that doesn’t appear to be the case – I only see the AMI in my private instances, and the details on the image seem to confirm this.

I’m running an instance right now; it did seem to take an extra long time for the image to come up, even by EC2 standards, and ec2-describe-instances showed the instance as “running” for a long time before I could connect using SSH. But it’s up now, ZFS root and all:

-bash-3.2# uname -a
SunOS domU-12-31-38-00-28-35 5.11 snv_91 i86pc i386 i86xpv
-bash-3.2# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mnt 106K 147G 18K /mnt
rpool 2.74G 6.86G 59K /rpool
rpool/ROOT 2.73G 6.86G 18K /rpool/ROOT
rpool/ROOT/opensolaris-1 2.73G 6.86G 2.72G legacy
rpool/ROOT/opensolaris-1/opt 12.9M 6.86G 12.9M /opt
rpool/export 37K 6.86G 19K /export
rpool/export/home 18K 6.86G 18K /export/home
swap 450M 9.89M 18K /swap
swap/swapfile 450M 460M 16K -


One curiosity:

-bash-3.2# pkg image-update
pkg: "image-update" option currently not supported on Amazon EC2. Please check out http://blogs.sun.com/ec2 for more details.


I must have missed those details…

Update: Here’s the reason why you can’t run “pkg image-update” from the Getting Started Guide for Amazon EC2: “pkg image-update – This command is currently not supported on Amazon EC2 since it modifies the kernel and ramdisk files resulting in non-bootable AMI. As we know, in the EC2 environment modifying the kernel and ramdisk is not permitted. In certain cases, if the user wants to enable this command, then the user can edit the /usr/bin/pkg file appropriately.” (Which makes sense.)

• NetApp finds NAS could mean ‘never accessed storage’ – “According to a USENIX presentation, 90% of data on NetApp’s networked storage systems was untouched over a 3-month period, raising the issue of whether it would be better placed on cheaper storage.” I would find some irony in that cheaper storage being tape.
• zfs un-benchmarking – “Our rationale for testing was to finally get some numbers that we can provide to users/customers about real zfs performance. There is a huge amount of (largely uncontested) information (emanating mainly from Sun and its agents) that zfs is a very fast file system. We want to test this, on real, live hardware, and report. Well, we can’t do the latter due to Sun’s licensing, but we did do the former. Paraphrasing Mark Twain: ‘Rumors of zfs’s performance have been greatly exaggerated.’” When Joe Landman blogs about performance, I take what he has to say seriously, but given the stability problems he notes, I wonder if – as he suggests – that driver issues are a factor here, and we’re not seeing a generic ZFS issue. (Seen at InsideHPC.)
• Self-protecting archive for SharePoint – “A new archiving product from BridgeHead Software automatically moves older, infrequently-accessed SharePoint items to cheaper archive media and cuts down the SharePoint backup burden.” HSM for SharePoint, apparently? Sounds interesting.
• Since NFSv4 is Stateful It Must Be Less Robust, Right? – “The short answer is no.” Interesting summary of how locking works under NFSv4; although I haven’t used NFSv4, this sounds like a massive improvement over previous versions – can I get the time that I spent debugging locking problems on Linux NFS servers back now?
• Storage Opens Up – Sun releases new JBODs and upgrades Thumper’s hardware. Given that I heard rumors of the Thumper expansion shelves (J4500) maybe a year ago, I’m surprised it took them so long to come out. And is it just me, or do the J4200 and J4400 look a little like someone else’s boxes rebranded?