Comments for thinking sysadmin

Comment on Fishworks’ LDAP Schema Definition by Saint Aardvark

Saint Aardvark — Mon, 11 Jan 2010 17:10:09 +0000

Thanks for this — I hadn’t realized the importance of the Search Descriptor field.

Comment on Duplicity to Amazon S3 on FreeBSD: Building on the work of others by tash

tash — Sun, 03 Jan 2010 22:25:41 +0000

I’m getting that “Fatal Error: Neither remote nor local manifest is readable” message in my cron logs. Hopefully exporting the HOME variable does the trick. Thanks in advance if it does

Comment on Test Driving Google Public DNS (Updated with OpenDNS comparison) by Andy

Andy — Thu, 03 Dec 2009 23:09:38 +0000

@Raj – As query response times – reported by “dig” as “Query time” – fell into two fairly tight groups, it seemed a reasonable assumption to assume the the faster responses were served out of cache, while the slower response times were not. Noting that repeated queries of the same hostname were always faster and always tightly clustered bears this out. You could also look at the TTL of the returned record to see if it is cached or not – if it is below what the authoritative host has for it, you know you’re looking at a cache hit.

Comment on Test Driving Google Public DNS (Updated with OpenDNS comparison) by Raj

Raj — Thu, 03 Dec 2009 22:35:25 +0000

Hey I wnated to know how you compared public dns and opendns .I mean what tool did you use to see the cache hit and miss ratio.I want to test the cache hit and miss of some DNS related app need info and help on that

Comment on Interesting Linux VM Crash Pattern by Andy

Andy — Fri, 20 Nov 2009 23:37:48 +0000

I’ve posted the NFS settings above; unfortunately, they don’t appear to deviate from default/recommended settings.

Comment on Interesting Linux VM Crash Pattern by Andy

Andy — Fri, 20 Nov 2009 23:13:20 +0000

Thanks for the input and suggestions, Nate – much appreciated.

I failed to mention in the post that we are running another cluster of ESX 3.5 hosts off the same filer (and same aggregate) using a Fibre Channel LUN without issue, FWIW. Guests on that cluster include Linux VMs, but all without vSMP. That suggests it’s not the filer itself, but it could still be the storage protocol, or the host hardware, or vSMP.

(And now that I’ve written that, I’m curious about verifying the VMware NFS settings on the problem host. I’ll post an update on what I find there.)

Comment on Keeping your RHEL VMs from crushing your storage at 4:02am by Andy

Andy — Fri, 20 Nov 2009 23:03:05 +0000

Thanks for the comment, Nate. I hadn’t considered makewhatis when I wrote this article, but I believe that it is called with the “-u” flag which greatly reduces its I/O load; a couple quick experiments on my workstation seem to bear this out, but, obviously, test in your own environment.

Adding the random delay loop in a separate file in /etc/cron.daily – say “1randomdelay” so it runs after the “0*” files – may also be effective (and cleaner) and will stagger all the files after it, sorted by the glob in run-parts.

Comment on Interesting Linux VM Crash Pattern by nate

nate — Fri, 20 Nov 2009 22:54:32 +0000

just for reference, I run about 100 VMs with 2.6.18-128.1.10.el5 on CentOS 5.2 (5.3 kernel on 5.2 distro) with the SMP kernel though there is only 1 vCPU per VM. All of them are local storage on Dell R610. Also have about a dozen VMs on the same kernel running with 2 vCPUs.

I run about 40 physical servers on the same kernel on native R610 with 8 cores no panics there either.

I run the SMP kernel on everything even if it’s only 1 vCPU, because you never know if you may need to upgrade to 2 or more CPUs and if you do I don’t want to have to change kernels. Though now that I think about it I think RHEL/CentOS 5.x kernels are all SMP now vs on 4.x where they had SMP and UP kernels, but not 100% sure.

Never had a panic of any sort on any of them.

To me your problem looks related to storage, so I would look to NetApp or VMware’s NFS stuff. I would also consider hooking at least one host via Fiber or iSCSI and run some of the VMs off of that and see what you get.

My own infrastructure is split into two classes, we have our edge web servers which are at several physical locations, all of them run off of local storage. The other class is most of our back end stuff or QA or internal IT, of which it’s all fiber channel connected(most of it is boot from SAN, with the exception of some older ESXi 3.5 systems that don’t support that), connected to our 3PAR T400 storage system.

in case it helps you in your tracing of the issue..

Comment on Keeping your RHEL VMs from crushing your storage at 4:02am by nate

nate — Fri, 20 Nov 2009 22:45:12 +0000

also disable the makewhatis cron I found that to be expensive as well.

I just checked my configs and it turns out I don’t have updatedb disabled on my VMs, I thought I did, suppose I should go and disable it. I do have my cfengine installation push out a version of the makewhatis cron file that just has an exit 0 at the top of it.

Comment on ESX VM swap on NFS: If it crashes, try something else by Andy Daniel

Andy Daniel — Tue, 21 Jul 2009 21:04:35 +0000

There appears to be some internal debate over this. The KB has been pulled from public access.