Beginning with Windows Server 2003, Active Directory supports VLV searches. Every VLV search request must be accompanied by 2 request controls: the SSS control and the VLV control. However, Active Directory imposes some general criteria on the SSS control:

1. Cannot sort based on more than one sort keys/attributes.
2. Cannot sort based on the “distinguishedName” attribute (presumably Microsoft does not use the “DN” attribute).
3. Cannot sort based on a constructed attribute (presumably an attribute not stored on Active Directory).

Unfortunately, Solaris LDAP clients use 2 sort keys/attributes: “cn” and “uid” in the SSS control. Subsequently, when dumping a container or a naming database, Solaris LDAP clients would receive LDAP_UNAVAILABLE_CRITICAL_EXTENSION.

$ ldaplist passwd
ldaplist: Object not found (LDAP ERROR (12): Unavailable critical extension.)

This issue has been detailed elsewhere, including at utexas.edu. There appear to be at least four solutions:

1. Wait for the fix from Sun Oracle to reach the light of day: this bug was apparently fixed in SNV 144. (I expect the fix is out in Solaris 11 Express now, but have not tested this myself.)
2. Apply the hotfix in Microsoft’s KB886683 to your domain controllers, which will disable VLV.
3. Run separate ADAM instances with VLV disabled, and point your Solaris machines at them instead of directly at your domain controllers. From the blog post linked above, it sounds like the University of Texas chose this route.
4. Use OpenLDAP as a proxy in front of Active Directory; configure your Solaris machines to use the proxies instead of Active Directory servers. This is the solution detailed in this blog post.

Method
I tested on Ubuntu 10.04 here, although adaptation for your chosen OS is probably straightforward.

First, build OpenLDAP; we need several options not included in the default .deb, so we’ll do this from source, with the following “configure” options:

./configure --prefix=/opt/openldap --enable-meta --enable-ldap --enable-rewrite \
--enable-rwm

(You will probably also want to take the extra step of building a package for your OS, to ease configuration management.)

After installing OpenLDAP, edit slapd.conf to proxy to your domain controllers. In this example, we’re pointing the proxies at some older Windows 2003 hosts using the Microsoft Services for Unix (SFU) extensions, so we’ll take this opportunity to make make what Solaris sees closer to RFC 2307-compliant and eliminate the need for attributeMap arguments to ldapclient on the Solaris host:

#
# See slapd.conf(5) for details on configuration options.
# This file should NOT be world readable.
#
include  /opt/openldap/etc/openldap/schema/core.schema
include  /opt/openldap/etc/openldap/schema/cosine.schema
include  /opt/openldap/etc/openldap/schema/inetorgperson.schema
include  /opt/openldap/etc/openldap/schema/nis.schema
include  /opt/openldap/etc/openldap/schema/mssfu30.schema

# Time out connections before the proxied bind drops:
idletimeout 60

pidfile  /var/run/slapd/slapd.pid
argsfile /var/run/slapd/slapd.args

access to dn.base="" by * read
access to dn.base="cn=Subschema" by * read
access to *
 by self write
 by users read
 by anonymous auth

loglevel   256

######################################################
# database definitions
######################################################

database ldap
suffix  "dc=example,dc=com"
uri  "ldap://dc1.example.com ldap://dc2.example.com"
acl-bind bindmethod=simple binddn="cn=ldapproxy,ou=Service Accounts,ou=Users,dc=example,dc=com" credentials=secret

# Do mapping in OpenLDAP, instead of on client, eliminating need for
# AD schema attribute mapping:
overlay rwm
rwm-map attribute       userpassword    msSFU30Password
rwm-map attribute       memberuid       msSFU30MemberUid
rwm-map attribute       gidnumber       msSFU30GidNumber
rwm-map attribute       gecos           name
rwm-map attribute       uid             msSFU30Name
rwm-map attribute       uidnumber       msSFU30UidNumber
rwm-map attribute       homedirectory   msSFU30HomeDirectory
rwm-map attribute       loginshell      msSFU30LoginShell
rwm-map objectclass     posixGroup      group
rwm-map objectclass     posixAccount    user

Walking through this config file line-by-line, not the following (keyed by line number):

3. Make the file owned by user “root”/group “openldap” without “other” read permissions (assuming you’ll be running OpenLDAP as user “openldap” in group “openldap” – adapt as necessary for your site), e.g.:

-r--r----- 1 root openldap 1569 2010-08-04 14:51 slapd.conf

9. We need to add some schema information for the older Microsoft Services for Unix extensions; they’re available in a GitHub Gist.

30-33. Configure the connection to the Active Directory servers. Note in particular that the database type is “ldap” meaning that another LDAP server (AD) is used as the data source, via a proxy. Line 33 is a user in Active Directory to which you bind to the back-end servers as.

37-47. Map the SFU attributes to their RFC 2307 equivalents. Note that you will need to populate msSFU30MemberUid manually – here’s one way to do it.

Other than that, perhaps an init script and a file for /etc/default/slapd (both lightly adapted from the stock Ubuntu “slapd” .deb), and you should be good to go.

References: eldapo: openldap as a pass-through proxy

See also: f5vpn-login.py, described here, and brought to my attention by sh4k3sph3r3. A CLI FirePass client is quite likely a better solution than separate browser instances, etc.

Preliminaries: Although the F5 FirePass SSL VPN product supports Linux, as best as I can tell, that support is somewhat limited: My understanding is that they officially claim support for 32-bit installs only, and they do not appear to track new distribution releases particularly aggressively. F5 has also been somewhat slow in supporting new browser versions: They announced support for Firefox 3 on October 6, 2008, nearly four months after its release and with only two months to go before Firefox 2 was end-of-lifed. For Firefox 3.6 support, a comment on the post linked above states that you need to request a special hot fix from F5 (which my site has not applied). There is no Google Chrome support that I am aware of.

Further, F5′s automated client installation tools have unfortunately never worked for me on Linux, even when the architecture and browser are in their support matrix. The manual download instruction links are also broken on the FirePass install I connect to.

Solution: Install a dedicated, 32-bit version of Firefox in a supported version; create a single-purpose Firefox profile for VPN use. Add the FirePass client to that browser and the operating system.

For the Firefox install, follow the “Manual Installation” instructions from the Ubuntu Community Documentation site. Install version 3.5 if your site does not have the hotfix mentioned above.

Be sure to create a new Firefox profile in your account for use with the FirePass; however, I recommend modifying the script in the Ubuntu documentation to automatically take you to your FirePass site (https://firepass.example.com/ for the purposes of this post):

#!/bin/bash
exec "\$HOME/firefox/firefox" -P mozilla-build https://firepass.example.com/

Next, download the client components from your F5 site; again, assuming firepass.example.com, retrieve and save:

https://firepass.example.com/vdesk/vpn/nogzip/downloads.php/linux/np_F5_SSL_VPN.so

and

https://firepass.example.com/vdesk/vpn/nogzip/downloads.php/linux/SSLVpn.tgz

Move np_F5_SSL_VPN.so to the plugins directory of the new Firefox installation – ~/firefox/plugins if following the Ubuntu documentation. Based on file layout, it appears that F5 intended for you to extract SSLVpn.tgz at the root of your file system. Instead of following this bad practice, in scratch space and as root, extract the SSLVpn.tgz tarball and manually move the files into place:

cp SSLVpn.tgz /tmp
cd /tmp
sudo tar -xvpzf SSLVpn.tgz
# inspect extracted files here...
cd /usr/local/lib
mkdir -p F5Networks/SSLVPN
cd /tmp/usr/local/lib/F5Networks/SSLVPN
cp -Rp etc svpn var /usr/local/lib/F5Networks/SSLVPN

Using the bash script above, you should now be able to launch your purpose-built FirePass browser installation and have it “just work” for Network Access. Good luck!

> rpm -q slocate
slocate-2.7-13.el4.8.i386

or

> rpm -q mlocate
mlocate-0.15-1.el5.2.x86_64

If so, multiple RHEL VMs plus mlocate or slocate may be adding up to an array-crushing 4:02am shared storage load and latency spike for you. Before being addressed, this spike was bad enough at my place of employment (when combined with a NetApp Sunday-morning disk scrub) to cause a Windows VM to crash with I/O errors. Ouch.

Details and ideas for resolution:

By default, a line in /etc/crontab runs the scripts within /etc/cron.daily at 4:02am each morning:

02 4 * * * root run-parts /etc/cron.daily

One of those scripts – mlocate.cron or slocate.cron, depending on your OS version – launches updatedb; as the man page says, “updatedb creates or updates a database used by locate(1).” (The “locate” binary is a filesystem search tool, see “man locate” for more information.) Updatedb refreshes its database by walking the filesystem, generating a fair amount of I/O on a single system. Imagine upwards of thirty of these running in parallel through VMDKs on one shared storage system carrying out internal maintenance at the same time, and you’re pretty much picturing the problem my employer had.

I see three options for addressing this issue:

1) Uninstall mlocate or slocate. If you don’t currently use “locate” and you’re not interested in learning to use a tool that will likely make you more effective at your job (again, see “man locate”), this is probably the best option. (Yeah, I know, people that fit this bill generally don’t read blogs more technical than this one, so I could probably have skipped it here. Consider it an option for completeness, or if you really need to strip down an install.)

2) Disable the scheduled job by removing mlocate.cron or slocate.cron from /etc/cron.daily. This keeps locate available for your use, but requires that you update locate’s database ad-hoc and interactively by running the following as root:

# updatedb

This will take a few minutes to return, depending on the size of your file systems.

I don’t recommend this option either; at least it doesn’t fit the way I work. I often find myself using locate in high-pressure situations in which I need to quickly get a file location on a system. Waiting minutes for updatedb to return is extra painful when every second counts.

3) Stagger when updatedb runs by inserting a random delay into the script.. This is my preferred alternative; locate’s database is kept current automatically, and your storage doesn’t have to bear a sudden spike in load. I implemented this by adding the lines in bold (lines 2-7 if your browser doesn’t display the bold text clearly):

#!/bin/sh
# sleep up to two hours before launching job:
value=$RANDOM
while [ $value -gt 7200 ] ; do
value=$RANDOM
done
sleep $value
nodevs=$(< /proc/filesystems awk '$1 == "nodev" { print $2 }')
renice +19 -p $$ >/dev/null 2>&1
/usr/bin/updatedb -f "$nodevs"


The added code inserts a pseudo-random sleep delay of up to two hours before updatedb runs, with the key being the built-in Bash function $RANDOM. In our environment, this removed a 2000 IOPS spike at 4:02am, and eliminated a corresponding jump in filer latency. Obviously, adjust the delay period as appropriate for your environment. Additionally, be sure to add this change to your configuration management or installation management tools so that all of your RHEL and RHEL-derived VMs get the updated script.

Using $RANDOM to avoid this variant of the thundering herd problem also works nicely for a range of similar problems; I believe I first saw it at Moundalexis.com.

(This problem may apply to other Linux distributions being run as VMs, and FreeBSD does something equivalent – weekly – with /etc/periodic/weekly/310.locate. A similar solution can be applied to these environments, if necessary.)

Well, perhaps a 64-bit AMI is forthcoming – I think this is still a beta program – and perhaps Sun’s just trying to save us a buck or two, since the cheapest 64-bit EC2 instance is four times as expensive per hour as the cheapest 32-bit instance.

The blog entry makes it sound like you no longer need to register with Sun to use OpenSolaris on EC2, but that doesn’t appear to be the case – I only see the AMI in my private instances, and the details on the image seem to confirm this.

I’m running an instance right now; it did seem to take an extra long time for the image to come up, even by EC2 standards, and ec2-describe-instances showed the instance as “running” for a long time before I could connect using SSH. But it’s up now, ZFS root and all:

-bash-3.2# uname -a
SunOS domU-12-31-38-00-28-35 5.11 snv_91 i86pc i386 i86xpv
-bash-3.2# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mnt 106K 147G 18K /mnt
rpool 2.74G 6.86G 59K /rpool
rpool/ROOT 2.73G 6.86G 18K /rpool/ROOT
rpool/ROOT/opensolaris-1 2.73G 6.86G 2.72G legacy
rpool/ROOT/opensolaris-1/opt 12.9M 6.86G 12.9M /opt
rpool/export 37K 6.86G 19K /export
rpool/export/home 18K 6.86G 18K /export/home
swap 450M 9.89M 18K /swap
swap/swapfile 450M 460M 16K -


One curiosity:

-bash-3.2# pkg image-update
pkg: "image-update" option currently not supported on Amazon EC2. Please check out http://blogs.sun.com/ec2 for more details.


I must have missed those details…

Update: Here’s the reason why you can’t run “pkg image-update” from the Getting Started Guide for Amazon EC2: “pkg image-update – This command is currently not supported on Amazon EC2 since it modifies the kernel and ramdisk files resulting in non-bootable AMI. As we know, in the EC2 environment modifying the kernel and ramdisk is not permitted. In certain cases, if the user wants to enable this command, then the user can edit the /usr/bin/pkg file appropriately.” (Which makes sense.)

I was expecting some hassle getting wireless going after the install. Instead, it worked out of the box. I did have to recreate my login keyring in seahorse for reasons that I didn’t bother to pursue; the default keyring didn’t want to unlock on login, but since I had nothing in it, I lost nothing in recreating it.

As far as post-install modifications, I made /tmp, /var/tmp and /var/log all tmpfs file systems to reduce writes to the Eee’s SSD (I also chose ext2 instead of a journaling file system during the install and mounted it noatime for the same reason). I had to hunt down and install a 2.x version of Firefox in parallel to Firefox 3.x since F5′s FirePass VPN client doesn’t work in 3.x. (Cheap shot: The reason why they call it clientless is that it doesn’t actually work on any clients out there. In seriousness, the advantage over traditional VPN clients just isn’t there.)

Other than the above and installing a couple stock Ubuntu packages (Thunderbird and libstdc++5 for Firefox), I haven’t had to make any tweaks to the vanilla install. If you’ve got an Eee, I highly recommend this upgrade.

(Until I hear back from Sun – “our technical team will review your requirements and get back to you shortly” – guess I’ll just fire up a Fedora AMI. Oh well.)