netapp01> lun show /vol/nexenta01/lun01/lun
        /vol/nexenta01/lun01/lun      10g (10737418240)   (r/w, online, mapped)

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no

root@nexenta01:/volumes# stmsboot -L
stmsboot: MPXIO disabled
root@nexenta01:/volumes# stmsboot -e -D fp
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
updating //platform/i86pc/boot_archive
updating //platform/i86pc/amd64/boot_archive
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)

nmc@nexenta01:/$ lunsync
Cleanup obsolete (dangling) device links?  Yes

Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c2t5*DDDd0  sd6       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd4       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd7       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c2t5*DDDd0  sd5       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd3       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd2       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd8       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
c3t5*DDDd0  sd1       disk         10GB                  no      qlc    60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

root@nexenta01:/volumes# stmsboot -L
stmsboot: No STMS devices have been found

netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
netapp01> igroup set nexenta01 alua yes
netapp01> igroup show -v nexenta01
    nexenta01 (FCP):
        OS Type: solaris
        Member: 21:00:00:aa:bb:cc:dd:ee (logged in on: 0b, 0d, vtic)
        Member: 21:01:00:aa:bb:cc:dd:ee(logged in on: 0b, 0d, vtic)
        ALUA: Yes

nmc@nexenta01:/$ lunsync -r
Cleanup obsolete (dangling) device links?  Yes
Re-scanning HBAs... done.
Re-enumerating LUNs... done.

nmc@nexenta01:/$ show lun
LUN ID      Device    Type         Size       Volume     Mounted Attach GUID
c0t0d0      sd0       disk         272.3GB    syspool    no      mega_sas 60024e805102c100118a3fa70ae8937a
c1t0d0      sd128     cdrom        No Media              no      ata    -
c4t6*469d0  sd9       disk         10GB                  no      mpxio  60a98000486e542f5034577076716469
syspo~/swap           zvol         1.0GB      syspool    no             -

root@nexenta01:/volumes# stmsboot -L
non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c3t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c3t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09869657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09889657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09888657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0
/dev/rdsk/c2t500A09868657ADDDd0 /dev/rdsk/c4t60A98000486E542F5034577076716469d0

Hat Tip to @complex on Twitter.

Reference: Is it possible to use I/O multipathing? How?

If you suspect that your system has almost used all of its free space, or if you use thin provisioning, you should check the amount of space in use by each aggregate. If any aggregate is 97 percent full or more, do not proceed with the upgrade until you have used the Upgrade Advisor or aggrSpaceCheck tools to determine your system capacity and plan your upgrade.

Upgrade Advisor is a great tool, and I heartily recommend you use it for your upgrade. However, it doesn’t give you a lot of visibility into what’s being checked for here. Lucky for us, NetApp offers an alternative tool: aggrSpaceCheck (NOW login required).

AggrSpaceCheck, as written, relies on you having rsh turned on for access to the filer – something that you probably locked down when you were still wearing acid-washed jeans. Assuming you don’t have rsh access, you’ll see an error like this when you attempt to run aggrSpaceCheck:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp 

Could not retrieve Aggregate free space. Could not get any aggregates in this filer

The fix is easy, however, if you’re using SSH: Edit the aggrSpaceCheck.pl file, replacing “rsh” with “ssh” (you only need to actually edit it in the line where “$remCmd” is defined, but changing rsh to ssh elsewhere won’t hurt). You will be prompted for root’s SSH password repeatedly – once for each command run remotely on the filer:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password:
root@toaster01's password: 

Aggregate aggr0 requires 54.18GB for volume metadata; 137.76GB is available.
Aggregate aggr0 has enough free space for you to upgrade to
Data ONTAP 7.3 or later.

There’s a solution for this too, of course: Enable SSH key pair authentication from your management host to your filer – no more password prompts:

> perl aggrSpaceCheck.pl -filer toaster01

aggrSpaceCheck V1.0.0 Copyright (c) 2008 NetApp 

Aggregate aggr0 requires 54.18GB for volume metadata; 137.76GB is available.
Aggregate aggr0 has enough free space for you to upgrade to
Data ONTAP 7.3 or later.

Beginning with Data ONTAP 7.3.1, FAS2020 systems support aggregates up to 16 TB raw capacity,
provided that the root volume is hosted in a dedicated aggregate (that is, one that contains only the root
volume and no user data).

The release notes go on to point out an alternative to the dedicated root aggregate – having two spare disks per controller.

It’s nice to see the FAS2020 finally getting a maximum aggregate size on par with the rest of NetApp’s product line. However, in an era where 2TB drives are available from Western Digital – and presumably other manufacturers before too long – ONTAP’s 16TB aggregate limit grows increasingly anachronistic.

We recently acquired SnapManager for Exchange (SME) at my place of employment. We have an existing NetApp deployment consisting of two primary filers in a SnapVault arrangement with a third filer. The SME install is part of an upgrade from Exchange 2003 (on DAS) to 2007 (on Fibre Channel storage).

What we missed prior to purchasing SME: If you want to use SnapVault with SME, you need two additional pieces of software: Protection Manager and NetApp Management Console (part of DataFabric Manager, apparently). Here’s what p. 408 of the SnapManager® 5.0 for Microsoft® Exchange Installation and Administration Guide (NOW login required) says:

The following are the software dependencies for integrating SnapManager with
data set and SnapVault:

◆ Protection Manager 3.7 and later
◆ NetApp Management Console 3.7 and later
◆ SnapDrive for Windows 6.0 and later
◆ Data ONTAP 7.3 or later

Wish I’d known that sooner.

(This is the point where some random NetApp fanboy pops down to the comments and fires off something about how NetApp is the greatest storage company ever, and if I’d done appropriate due diligence, I wouldn’t have missed this requirement. My advice: Spare us, smart guy. I’m writing this post to make it easier for other NetApp customers to do their “due diligence”.)

First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto and security/pinentry-curses ports.

If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:

2009-03-01_01:05:05: ... backing up filesystem
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 240,
in GPGWriteFile
bytes_to_go = data_size - get_current_size()
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 232,
in get_current_size
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory:
'/tmp/duplicity-gM4CN9-tempdir/mktemp-iZknw0-2'

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 232, in full_backup
sig_outfp = get_sig_fileobj("full-sig")
File "/usr/local/bin/duplicity", line 210, in get_sig_fileobj
fh = globals.backend.get_fileobj_write(sig_filename)
File "/usr/local/lib/python2.5/site-packages/duplicity/backend.py", line
354, in get_fileobj_write
fh = dup_temp.FileobjHooked(tdp.filtered_open("wb"))
File "/usr/local/lib/python2.5/site-packages/duplicity/path.py", line 716,
return gpg.GPGFile(1, self, gpg_profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 237,
in GPGWriteFile
file = GPGFile(True, path.Path(filename), profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

The solution to the above is simple – make sure the path includes /usr/local/bin, perhaps by including this at the start of the backup script:

export PATH=${PATH}:/usr/local/bin

Finally, when running an incremental backup, you may get this error:

Fatal Error: Neither remote nor local manifest is readable.

This can be solved by setting the HOME environment variable to /root assuming you’re running the backup as root (instead of the default /var/log for cron jobs):

export HOME=/root

Makes me wonder how much engineering that I never thought about goes into designing disk shelves to keep drives insulated from vibrations. The Fishworks analytics interface is dazzling – wish I had that yesterday when I was looking at a possible Exchange I/O performance issue with perfmon…

As I noted in my “quick walk” post’s comments, I had difficulty getting LDAP working initially on my corporate Active Directory network. The crux for me turned out to be getting the LDAP Schema Definitions correct. Here are the settings that worked correctly for me, authenticating against an AD instance with the schema extended by Microsoft’s Services for Unix add-on (other LDAP schemata will, of course, need different mappings):

USERS
Search descriptor: Don’t leave this blank – according to the Fishworks documentation this “sets the LDAP search descriptor, attribute mappings and object class mappings for users and groups. By default, the search descriptor for users is ou=people,dc=example,dc=com, and for groups is ou=group,dc=example,dc=com” – so what you enter will be site-specific.

Attribute mappings:

• uid=msSFU30Name
• uidNumber=msSFU30UidNumber
• gidNumber=msSFU30GidNumber

Object class mappings:

• posixAccount=User

GROUPS
Search descriptor: Again, don’t leave this blank – enter the appropriate value for your site.

Attribute mappings:

• gidNumber=msSFU30GidNumber
• uniqueMember=msSFU30PosixMember

Object class mappings:

• posixGroup=group

How did I know that the schema definition mappings were the problem? The logs gave it away: Maintenance -> Logs -> System, where I saw messages similar to the following: “libsldap: Status: 0 Mesg: Unable to set value: schema map already existed for ‘User’.”

How did I know that I had the schema definitions working? Share settings that I had created using numeric UIDs and GIDs automatically became mapped to the correct user and group names.

I’ll update this post if I find additional configuration that may be necessary.

That virtualized Fishworks appliance got me thinking: What if you combined this with this? Yeah, managing Elastic Block Store devices would require some changes, but, if you needed a NAS for your EC2 instances…

That was easy...

Below is a quick walkthrough of my experience booting and installing the Fishworks VMware appliance; my thoughts follow.

Fishworks Boot Screen

Initial network configuration

Yes, Virginia, there is a command line - but not a conventional shell

Web interface login screen

Network configuration

Configure DNS - pretty basic stuff

NTP is a good thing

Name service configuration choices

Active Directory integration for CIFS shares is straightforward

Fishworks phone home...

Storage configuration made fast and easy

All done!

Thoughts: First off, as a VMware appliance and an introduction to Fishworks, this virtual machine is extremely well done; it gives users a low “activation energy” method to try out Fishworks. There’s a lot more effort involved getting a sixty-day loaner from Sun, as neat as that program is – even the logistics of racking and cabling the machine are relatively expensive in terms of time, after all. Of course, there are limitations inherent to a virtual machine; for example, I’d love to see how VMware ESX performs using a hybrid storage pool, but only real hardware can answer that. However, for most of my questions about Fishworks, the VM fits the bill.

Once you hit the VM’s virtual power button, it boots up quickly – faster than most OpenSolaris machines, subjectively – and the initial configuration is straightforward. Although I didn’t try it, it does appear that the machine can be completely configured from the command line, either at the console or over SSH. Once the system is configured, you have HTTPS or SSH remote administration access; however, SSH does not give you a standard Solaris shell, such as sh or bash. Instead, you get an interface very similar to Solaris’ zonecfg – a command line interface with multiple modes. (I suspect that a shell can be launched on the machine somewhere, somehow – it may even be documented – but I wouldn’t be stunned if it’s “unsupported.”)

The six-step configuration process via the web interface is smooth and consistent, and not particularly time-consuming – you get this sense that this is truly intended to be an appliance. My only complaints: The online docs were not immediately helpful (seriously – links to Wikipedia?), and when something does go wrong, you don’t get much more immediate help than this:

Uh-oh - how do I get to those log files?

By far, the best part of the configuration interface is step six – the storage configuration. You see a list of your configuration choices, with availability, performance and capacity rated. The left-hand column shows a pie-chart breakdown of your storage utilization, with the tiny “Reserved” slice surely being a jab at NetApp. (Well, maybe not – but as a NetApp user, it sure got my attention.) Simply pick your RAID type, click “commit” and you’re on your way. Genius.

All in all, very impressive. Makes me wish even more that I had a pair of 7410s to put behind ESX. And Exchange. And SQL Server. And my CIFS and NFS users… you get the idea.