thinking sysadmin » aws http://andyleonard.com qstat -u aleonard -s z Tue, 28 Feb 2012 04:47:09 +0000 en hourly 1 http://wordpress.org/?v=3.3.2 What t1.micro CPU Bursting Looks Like http://andyleonard.com/2011/12/09/what-t1-micro-cpu-bursting-looks-like/ http://andyleonard.com/2011/12/09/what-t1-micro-cpu-bursting-looks-like/#comments Sat, 10 Dec 2011 06:26:13 +0000 Andy http://andyleonard.com/?p=698 Amazon’s smallest and least expensive instance type, the t1.micro “provide[s] a small amount of consistent CPU resources and allow[s] you to burst CPU capacity when additional cycles are available. [It is] well suited for lower throughput applications and web sites that consume significant compute cycles periodically.” (source)

Running a cpu-bound workload (building Perl modules) on an Ubuntu 11.10 t1.micro instance in us-west-2 tonight, I noticed the following curious CPU usage pattern of approximately 15 seconds on, 60 seconds off:

> vmstat 5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0  38528  29524 370540    0    0    86   423   84  216 12  5 35  4
 1  0      0   6800  30288 388856    0    0  5356    26  660 1433 27 27  6 40
 5  0      0  21752  27624 378088    0    0    30   211  150  159 40 22  0  8
 6  0      0  21256  27636 378104    0    0     0    27    9    7  1  1  0  0
 7  0      0  21256  27644 378108    0    0     0    10    9    9  1  1  0  0
 7  0      0  21256  27652 378112    0    0     0     8    9    9  2  1  0  0
 7  0      0  20256  27652 378228    0    0     0     0    8   13  1  1  0  0
 8  0      0  20016  27660 378072    0    0     0   218   15   29  0  2  0  3
 6  0      0  37884  27672 378048    0    0     0    14    9   11  3  1  0  0
 4  0      0  30808  27684 378048    0    0     0    11    9   10  1  1  0  0
 4  0      0  23740  27692 378056    0    0     0    10    8    8  2  1  0  0
 4  0      0  30676  27692 378104    0    0     0     0   10   10  1  1  0  0
 5  0      0  26220  27700 378064    0    0     0     9    7   14  6  2  0  1
 5  0      0  21012  27712 378120    0    0     0    10    9   10  1  0  0  0
 5  0      0  27336  27720 378064    0    0     0    21   13   10  1  1  0  0
 1  0      0  29444  27732 378064    0    0     0    14  149   97 39 19  0  0
 1  0      0  33420  27744 378084    0    0     6    12  250  166 67 30  0  0
 2  0      0  41108  27756 378100    0    0     0    37  207  148 60 29  0  0
 6  0      0  33668  27768 378068    0    0     0    14    8    9  1  1  0  0
 5  0      0  37008  27780 378068    0    0     0    10   10   15  4  1  0  0
 4  0      0  30808  27788 378072    0    0     0    18   11    9  2  0  0  0
 5  0      0  24360  27796 378092    0    0     0     9    8    7  2  0  0  0
 2  0      0  19896  27796 378140    0    0     0     0    8    9  1  1  0  0
 6  0      0  27584  27804 378152    0    0     0     7    8   12  1  1  0  0
 6  0      0  22864  27812 378148    0    0     0     9   10   12  2  1  0  0
 7  0      0  19136  27820 378152    0    0     0    10    8    9  1  1  0  0
 6  0      0  26096  27828 378148    0    0     0    12   10    7  2  1  0  0
 6  0      0  20640  27828 378156    0    0     0    19   13    8  2  1  0  0
 6  0      0  27956  27836 378156    0    0     0    11    9   12  1  1  0  0
 6  0      0  22864  27844 378156    0    0     0     6    9   12  2  1  0  0
 6  0      0  19020  27844 378156    0    0     0     1    9    9  1  1  0  0
 2  0      0  46896  21504 368588    0    0   518    18  261  291 47 29  1  7
 1  0      0  35372  21692 368788    0    0     0    43  253  174 65 32  0  0
 1  0      0  43060  21796 368600    0    0     0    62  149  112 66 32  0  1
 5  0      0  38100  21808 368600    0    0     0    46   11   10  1  1  0  0
 5  0      0  45788  21816 368592    0    0     0     7    8   12  2  1  0  0
 7  0      0  38464  21816 368600    0    0     0     0    7    8  2  1  0  0
 7  0      0  45912  21824 368596    0    0     0    11    9    9  2  1  0  0
 7  0      0  39216  21832 368600    0    0     0     7    9    8  1  0  0  0
 4  0      0  35496  21840 368596    0    0     0    19   11    9  4  1  0  0
 5  0      0  43060  21848 368600    0    0     0    29   10   10  2  1  0  0
 5  0      0  37480  21856 368592    0    0     0    11    9   10  1  1  0  0
 5  0      0  45044  21864 368596    0    0     0     7    9   10  1  1  0  0
 5  0      0  38340  21872 368600    0    0     0     8    8    8  2  1  0  0
 4  0      0  46284  21880 368596    0    0     0    10   10   11  1  1  0  0
 6  0      0  38836  21888 368592    0    0     0     8    8    8  2  1  0  0
 1  0      0  38340  21888 368544    0    0     0    15   53   41 12  7  0  0
 1  0      0  40828  21900 368568    0    0     2    46  255  218 66 33  0  0
 1  0      0  39960  21912 368608    0    0     0    26  237  153 63 28  0  0
 3  0      0  50632  21924 368540    0    0     0    16   58   44 32 15  0  0
 4  0      0  46284  21932 368540    0    0     0     7    8   11  1  1  0  0
 4  0      0  45400  21940 368540    0    0     0     6    9   10  1  1  0  0
 5  0      0  45292  21948 368552    0    0     0    11    8   14  0  1  0  0
 6  0      0  37720  21948 368584    0    0     0    17   12    6  2  1  0  0

Apparently, the “small amount of consistent CPU resources” is about 3% of the CPU.

Moral of the story for me? Next time, pay the big bucks and launch an m1.small spot instance.

]]> http://andyleonard.com/2011/12/09/what-t1-micro-cpu-bursting-looks-like/feed/ 0 S3fs, or, 256TB of Storage on the Cheap http://andyleonard.com/2011/01/25/s3fs-or-256tb-of-storage-on-the-cheap/ http://andyleonard.com/2011/01/25/s3fs-or-256tb-of-storage-on-the-cheap/#comments Tue, 25 Jan 2011 14:59:14 +0000 Andy http://andyleonard.com/?p=624 There’s something pretty satisfying about seeing 256TB of storage available on a machine and knowing that you’re only paying pennies for what you’re using:

> df -h /cloud/hrc/src/
Filesystem            Size  Used Avail Use% Mounted on
s3fs-1.35             256T     0  256T   0% /cloud/hrc/src


In the words of its authors, “s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files).”

Now, make no mistake about it – since s3fs is backed by object storage in a remote data center, this is not for high- or even moderate-IOPS workloads. Routine tasks like expanding tarballs containing many small files or compiling code on an s3fs file system can be painful. But for “colder” storage applications – think online archives, or possibly some backup applications – it shines.

The installation procedure for s3fs is straightforward. I’ve also put a Puppet module for installing s3fs and managing its mounts on GitHub, although you may want to adapt it to distribute your own package of s3fs instead of building it locally on each machine.

S3fs is licensed under the GPL, as is my Puppet module.

]]> http://andyleonard.com/2011/01/25/s3fs-or-256tb-of-storage-on-the-cheap/feed/ 0 Amazon Route 53 DNS Service Examined http://andyleonard.com/2010/12/06/amazon-route-53-dns-service-examined/ http://andyleonard.com/2010/12/06/amazon-route-53-dns-service-examined/#comments Mon, 06 Dec 2010 21:19:00 +0000 Andy http://andyleonard.com/?p=555 Amazon has announced a new authoritative DNS service – Route 53.

Sign up is straightforward – click a few buttons on aws.amazon.com, and a few moments later, you’ll have an email confirming your access to the service. If you dig into the Getting Started Guide, you’ll note that “Part of the sign-up procedure involves receiving a phone call and entering a PIN using the phone keypad,” however, that wasn’t necessary for me. Perhaps it’s only for new AWS accounts?

There is no user interface in the AWS Console although there are indications one is on its way. The Route 53 developer tools are fairly bare-bones at this point – four Perl scripts. Those scripts require relatively uncommon Perl modules, not included in the default Ubuntu (Lucid) repositories, although they are available through CPAN.

However, the third-party Boto Python interface to Amazon Web Services already includes support, and presumably other tools are also rapidly adding support, if they don’t have it already.

Using the Perl tools, I created a zone for an example domain – gearlister.org – and was given four name servers:

ns-1945.awsdns-51.co.uk (205.251.199.153)
ns-39.awsdns-04.com (205.251.192.39)
ns-690.awsdns-22.net (205.251.194.178)
ns-1344.awsdns-40.org (205.251.197.64)

The cross-section of TLDs increase the likelihood that a glue record for one of the Route 53 name servers will be returned with a query to the TLD name servers, reducing latency for clients:

; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> @d0.org.afilias-nst.org gearlister.org
; (2 servers found)
;; global options:  printcmd
;; Got answer:
;; >>HEADER<<- opcode: QUERY, status: NOERROR, id: 51176
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; QUESTION SECTION:
;gearlister.org.                        IN      A

;; AUTHORITY SECTION:
gearlister.org.         86400   IN      NS      ns-39.awsdns-04.com.
gearlister.org.         86400   IN      NS      ns-1945.awsdns-51.co.uk.
gearlister.org.         86400   IN      NS      ns-1344.awsdns-40.org.
gearlister.org.         86400   IN      NS      ns-690.awsdns-22.net.

;; ADDITIONAL SECTION:
ns-1344.awsdns-40.org.  86400   IN      A       205.251.197.64

;; Query time: 161 msec
;; SERVER: 199.19.57.1#53(199.19.57.1)
;; WHEN: Mon Dec  6 12:13:30 2010
;; MSG SIZE  rcvd: 184

While all name servers are in 205.251.192.0/18, there appear to be separate routing table entries for each /23 containing a name server. Further, the servers appear to be anycast to different locations around the world:

traceroute to 205.251.199.153 (205.251.199.153), 30 hops max, 40 byte packets
 1  swiCS5-V108.switch.ch (130.59.108.5)  0.355 ms  0.429 ms  0.546 ms
 2  swiZH2-10GE-3-1.switch.ch (130.59.36.138)  0.437 ms  0.515 ms  0.610 ms
 3  swiIX1-10GE-1-3.switch.ch (130.59.36.129)  6.300 ms  6.391 ms  6.486 ms
 4  zch-b1-geth3-1.telia.net (213.248.79.189)  0.345 ms  0.348 ms  0.347 ms
 5  ffm-bb2-link.telia.net (80.91.249.115)  11.914 ms  11.963 ms  11.991 ms
 6  ffm-b10-link.telia.net (80.91.251.250)  11.846 ms  11.834 ms ffm-b10-link.telia.net (80.91.251.126)  11.834 ms
 7  xe-4-2-0.edge4.Frankfurt1.level3.net (4.68.63.121)  11.960 ms  11.972 ms  11.961 ms
 8  vlan99.csw4.Frankfurt1.Level3.net (4.68.23.254)  20.991 ms vlan89.csw3.Frankfurt1.Level3.net (4.68.23.190)  12.241 ms  12.220 ms
 9  ae-82-82.ebr2.Frankfurt1.Level3.net (4.69.140.25)  13.459 ms  12.333 ms ae-92-92.ebr2.Frankfurt1.Level3.net (4.69.140.29)  12.699 ms
10  ae-24-24.ebr2.London1.Level3.net (4.69.148.197)  24.591 ms ae-21-21.ebr2.London1.Level3.net (4.69.148.185)  26.136 ms ae-22-22.ebr2.London1.Level3.net (4.69.148.189)  25.632 ms
11  ae-22-52.car2.London1.Level3.net (4.69.139.99)  20.163 ms  20.064 ms  19.870 ms
12  AMAZONCOM.car2.London1.Level3.net (212.187.193.2)  19.840 ms  19.868 ms  20.107 ms
13  * * *

Tracing the route to 205.251.199.153

  1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 0 msec 0 msec 0 msec
  2 3.xe-1-3-0.uonet10-gw.uoregon.edu (128.223.3.10) [AS 3582] 0 msec 0 msec 0 msec
  3 vl-3.uonet9-gw.uoregon.edu (128.223.3.9) [AS 3582] 0 msec 0 msec 0 msec
  4 eugn-car1-gw.nero.net (207.98.68.181) [AS 3701] 4 msec 0 msec 0 msec
  5 eugn-core1-gw.nero.net (207.98.64.161) [AS 3701] 0 msec 0 msec 0 msec
  6 eugnor1wce1-gige7-0.wcg.net (64.200.134.197) [AS 3356] 16 msec 8 msec 8 msec
  7 ae-32-52.ebr2.Seattle1.Level3.net (4.68.105.62) [AS 3356] 20 msec 8 msec 16 msec
  8 ae-2-2.ebr2.Denver1.Level3.net (4.69.132.54) [AS 3356] 48 msec 40 msec 36 msec
  9 ae-1-100.ebr1.Denver1.Level3.net (4.69.132.37) [AS 3356] 44 msec 36 msec 36 msec
 10 ae-4-4.car1.StLouis1.Level3.net (4.69.132.181) [AS 3356] 56 msec 52 msec 56 msec
 11 ae-11-11.car2.StLouis1.Level3.net (4.69.132.186) [AS 3356] 52 msec 56 msec 52 msec
 12 AMAZONCOM.car2.StLouis1.Level3.net (4.53.162.66) [AS 3356] 56 msec 56 msec 56 msec
 13  *  *  *

Adapting the Getting Started Guide, I created two A records for “gearlister.org” and “www.gearlister.org”. For reasons I wasn’t able to track down – or reproduce – adding the “gearlister.org” A record failed the first time, although I was able to add it later.

Update 12/7/2010: I received an email from Amazon earlier today explaining the failed A record add:

Here’s what happened. In our “Getting Started Guide” we incorrectly provided an example ChangeResourceRecordSets request that showed a single element that included multiple elements. This was a mistake. In reality, only one element is permitted per element. Our API accepted this request as valid, but silently only processed one of the elements. We have now fixed both the documentation and the configuration API to enforce the proper semantics.

Record changes propagated quickly, although the zone serial number did not increment. Given the lack of support for connecting secondary servers to Route 53 and the API support for checking whether a change has propagated, this may matter little in practice, although it is certainly odd.

Although there is no direct support for secondary servers using Route 53 as primary DNS – or for using Route 53 as a secondary to a non-Amazon primary – the BIND conversion scripts hint that it should be straightforward to have a master script update Route 53 and non-Route 53 zone configuration simultaneously. Also, while Route 53 does support AAAA records (ironic, given that you cannot use IPv6 to address EC2 instances), it does not yet support DNSSEC.

At $1/domain/month and $0.50/millon queries, pricing is extremely low for anycast DNS. However, given the lack of integration with some of Amazon’s other products, such as Elastic Load Balancing – apparently forthcoming – and the limited tools for managing zones, uptake will probably be limited initially. Some heavy AWS users may be hesitant to put their DNS on the same service provider as the rest of their infrastructure – although ultimately, as Amazon adds features, the benefits of Route 53 may outweigh the risks.

]]> http://andyleonard.com/2010/12/06/amazon-route-53-dns-service-examined/feed/ 3 Adding Swap to an EC2 Micro Instance http://andyleonard.com/2010/12/03/adding-swap-to-an-ec2-micro-instance/ http://andyleonard.com/2010/12/03/adding-swap-to-an-ec2-micro-instance/#comments Fri, 03 Dec 2010 21:57:46 +0000 Andy http://andyleonard.com/?p=552 EC2 micro instances come with no swap by default – at least every micro instance that I’ve ever launched does, I’m not sure if it’s theoretically possible to launch an instance with swap. The lack of swap is probably a side-effect of the limited memory combined with EBS-only storage and concomitant risk of high EBS charges if you swap heavily.

However, if you’re willing to accept the risk of unexpected high EBS I/O costs, it’s straightforward to add swap:

# /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
# /sbin/mkswap /var/swap.1
# /sbin/swapon /var/swap.1

Or, if you prefer Puppet:

class swapfile {

  exec { "create swap file":
    command => "/bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024",
    creates => "/var/swap.1",
  }

  exec { "attach swap file":
    command => "/sbin/mkswap /var/swap.1 && /sbin/swapon /var/swap.1",
    require => Exec["create swap file"],
    unless => "/sbin/swapon -s | grep /var/swap.1",
  }

}
]]> http://andyleonard.com/2010/12/03/adding-swap-to-an-ec2-micro-instance/feed/ 7 Duplicity to Amazon S3 on FreeBSD: Building on the work of others http://andyleonard.com/2009/03/02/duplicity-to-amazon-s3-on-freebsd-building-on-the-work-of-others/ http://andyleonard.com/2009/03/02/duplicity-to-amazon-s3-on-freebsd-building-on-the-work-of-others/#comments Mon, 02 Mar 2009 19:47:53 +0000 Andy http://andyleonard.com/?p=226 (This post adds only a couple small details to work described at randys.org and cenolan.com – go there for background on this post and useful scripts for automated Duplicity backup to S3.)

First off, if you want to use Duplicity installed from FreeBSD Ports to backup to Amazon S3, be sure to also install the devel/py-boto and security/pinentry-curses ports.

If you attempt to run the backup script described at randys.org or cenolan.com from cron, you may run into an error similar to the following:

2009-03-01_01:05:05: ... backing up filesystem
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Cleanup of temporary directory /tmp/duplicity-gM4CN9-tempdir failed - this
is probably a bug.
Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 240,
in GPGWriteFile
bytes_to_go = data_size - get_current_size()
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 232,
in get_current_size
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory:
'/tmp/duplicity-gM4CN9-tempdir/mktemp-iZknw0-2'

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 232, in full_backup
sig_outfp = get_sig_fileobj("full-sig")
File "/usr/local/bin/duplicity", line 210, in get_sig_fileobj
fh = globals.backend.get_fileobj_write(sig_filename)
File "/usr/local/lib/python2.5/site-packages/duplicity/backend.py", line
354, in get_fileobj_write
fh = dup_temp.FileobjHooked(tdp.filtered_open("wb"))
File "/usr/local/lib/python2.5/site-packages/duplicity/path.py", line 716,
return gpg.GPGFile(1, self, gpg_profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

Traceback (most recent call last):
File "/usr/local/bin/duplicity", line 583, in <module>
with_tempdir(main)
File "/usr/local/bin/duplicity", line 577, in with_tempdir
fn()
File "/usr/local/bin/duplicity", line 558, in main
full_backup(col_stats)
File "/usr/local/bin/duplicity", line 234, in full_backup
bytes_written = write_multivol("full", tarblock_iter, globals.backend)
File "/usr/local/bin/duplicity", line 148, in write_multivol
globals.gpg_profile, globals.volsize)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 237,
in GPGWriteFile
file = GPGFile(True, path.Path(filename), profile)
File "/usr/local/lib/python2.5/site-packages/duplicity/gpg.py", line 112,
in __init__
'logger': self.logger_fp})
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 357,
in run
create_fhs, attach_fhs)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 401,
in _attach_fork_exec
if process.pid == 0: self._as_child(process, gnupg_commands, args)
File "/usr/local/lib/python2.5/site-packages/GnuPGInterface.py", line 442,
in _as_child
os.execvp( command[0], command )
File "/usr/local/lib/python2.5/os.py", line 354, in execvp
_execvpe(file, args)
File "/usr/local/lib/python2.5/os.py", line 390, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory

The solution to the above is simple – make sure the path includes /usr/local/bin, perhaps by including this at the start of the backup script:

export PATH=${PATH}:/usr/local/bin

Finally, when running an incremental backup, you may get this error:

Fatal Error: Neither remote nor local manifest is readable.

This can be solved by setting the HOME environment variable to /root assuming you’re running the backup as root (instead of the default /var/log for cron jobs):

export HOME=/root
]]> http://andyleonard.com/2009/03/02/duplicity-to-amazon-s3-on-freebsd-building-on-the-work-of-others/feed/ 4 ElasticFish? http://andyleonard.com/2008/11/12/elasticfish/ http://andyleonard.com/2008/11/12/elasticfish/#comments Wed, 12 Nov 2008 22:21:49 +0000 Andy http://andyleonard.com/?p=161 (In the spirit of Joerg Moellenkamp‘s thought experiments:)

That virtualized Fishworks appliance got me thinking: What if you combined this with this? Yeah, managing Elastic Block Store devices would require some changes, but, if you needed a NAS for your EC2 instances…

]]> http://andyleonard.com/2008/11/12/elasticfish/feed/ 0 Hotlinks, 7/1/2008 http://andyleonard.com/2008/07/01/hotlinks-712008/ http://andyleonard.com/2008/07/01/hotlinks-712008/#comments Tue, 01 Jul 2008 19:08:03 +0000 Andy http://andyleonard.com/?p=45
  • The Hitz report – Robin Harris at StorageMojo on the Sun-NetApp lawsuit:

    NetApp’s biggest misperception is that WAFL is somehow central to the success they are enjoying today. That was true about 10 years ago. Guys, your average F500 CIO today could care less about WAFL.

    NetApp is growing because they offer a compelling value proposition of quality products, relevant services and worldwide support. WAFL certainly supports that, but as NetApp execs note much of their recent success is due to the integration software that NetApp now offers.

    WAFL is a small piece of the picture. Sun could copy it line for line and still not have a quarter of what NetApp offers.

    NetApp faces challenges. Storage commoditization threatens all vendors traditional 60% gross margins. The GX integration is problematic and the bottom line benefit uncertain. EMC’s move into cloud file services is a clever flanking strategy.

    An interesting opinion summed up nicely, I think.

  • Saving and Restoring ZFS Snapshots to and from Amazon S3 – A ZFS to S3 workaround for the lack of persistent storage on EC2.
    • ]]>     http://andyleonard.com/2008/07/01/hotlinks-712008/feed/ 0     README, 6/25/2008 http://andyleonard.com/2008/06/25/readme-6252008/ http://andyleonard.com/2008/06/25/readme-6252008/#comments Wed, 25 Jun 2008 19:27:15 +0000 Andy http://andyleonard.com/?p=41
  • Daemonic Dispatches: Dissecting SimpleDB BoxUsage – Colin Percival takes a critical look at Amazon’s SimpleDB pricing scheme.
    • ]]>     http://andyleonard.com/2008/06/25/readme-6252008/feed/ 0     No Luck with a Quick-n-Dirty BFU of SXCE 79 on EC2 http://andyleonard.com/2008/06/25/no-luck-with-a-quick-n-dirty-bfu-of-sxce-79-on-ec2/ http://andyleonard.com/2008/06/25/no-luck-with-a-quick-n-dirty-bfu-of-sxce-79-on-ec2/#comments Wed, 25 Jun 2008 14:19:10 +0000 Andy http://andyleonard.com/?p=40 For grins, I tried a quick-and-dirty BFU of a SXCE 79 instance running on EC2 to the latest nightly build this morning. I roughly followed Ben Rockwood’s BFU instructions and didn’t do anything to resolve conflicts beyond running acr. On reboot, it looks like the system panicked – I presume the reason is probably somewhere in here. Console dump after the jump for the curious.

    v3.0.3-rc5-8.1.14.e chgset 'unavailable '
    WARNING: Found hypervisor version: v3.0.3-rc5-8.1.14.e but need at least version v3.0.4
    SunOS Release 5.11 Version snv_79 32-bit
    Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved.
    Use is subject to license terms.
    NOTICE: xdf@0: failed to read feature-barrier
    Hostname: ip-10-250-11-159
    Reading ZFS config: *done.
    ip-10-250-11-159 console login: syslogd: line 45: WARNING: loghost could not be resolved
    syncing file systems... done
    rebooting...
    v3.0.3-rc5-8.1.14.e chgset 'unavailable '
    WARNING: Found hypervisor version: v3.0.3-rc5-8.1.14.e but need at least version v3.0.4
    SunOS Release 5.11 Version snv_79 32-bit
    Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved.
    Use is subject to license terms.
    NOTICE: xdf@0: failed to read feature-barrier
    WARNING: file system 'ctfs' version mismatch
    WARNING: Cannot mount /system/contract
    /kernel/fs/procfs: undefined symbol 'schedctl_set_cidpri'
    WARNING: mod_load: cannot load module 'procfs'
    WARNING: Cannot mount /proc
    WARNING: file system 'mntfs' version mismatch
    WARNING: Cannot mount /etc/mnttab
    /kernel/fs/tmpfs: undefined symbol 'choose_addr'
    WARNING: mod_load: cannot load module 'tmpfs'
    WARNING: Cannot mount /etc/svc/volatile
    WARNING: file system 'objfs' version mismatch
    WARNING: Cannot mount /system/object
    WARNING: file system 'sharefs' version mismatch
    WARNING: Cannot mount /etc/dfs/sharetab
    /kernel/fs/procfs: undefined symbol 'schedctl_set_cidpri'
    WARNING: mod_load: cannot load module 'procfs'
    /kernel/drv/ip: undefined symbol 'uioamove'
    /kernel/drv/ip: undefined symbol 'dl_errstr'
    /kernel/drv/ip: undefined symbol 'dl_primstr'
    /kernel/drv/ip: undefined symbol 'dhcifname'
    WARNING: mod_load: cannot load module 'ip'
    /kernel/fs/sockfs: undefined symbol 'uioafini'
    /kernel/fs/sockfs: undefined symbol 'uioainit'
    /kernel/fs/sockfs: undefined symbol 'uioasync'
    /kernel/fs/sockfs: undefined symbol 'kssl_handle_mblk'
    /kernel/fs/sockfs: undefined symbol 'sctp_disconnect'
    /kernel/fs/sockfs: undefined symbol 'sctp_getsockname'
    /kernel/fs/sockfs: undefined symbol 'nd_free'
    /kernel/fs/sockfs: undefined symbol 'nd_load'
    /kernel/fs/sockfs: undefined symbol 'sctp_create'
    /kernel/fs/sockfs: undefined symbol 'sctp_close'
    /kernel/fs/sockfs: undefined symbol 'sctp_listen'
    /kernel/fs/sockfs: undefined symbol 'sctp_recvd'
    /kernel/fs/sockfs: undefined symbol 'mi_mpprintf'
    /kernel/fs/sockfs: undefined symbol 'sctp_alloc_hdr'
    /kernel/fs/sockfs: undefined symbol 'sctp_get_opt'
    /kernel/fs/sockfs: undefined symbol 'tcp_wput'
    /kernel/fs/sockfs: undefined symbol 'mi_sprintf'
    /kernel/fs/sockfs: undefined symbol 'sctp_bind'
    /kernel/fs/sockfs: undefined symbol 'udp_wput'
    /kernel/fs/sockfs: undefined symbol 'sctp_getpeername'
    /kernel/fs/sockfs: undefined symbol 'sctp_sendmsg'
    /kernel/fs/sockfs: undefined symbol 'sctp_set_opt'
    /kernel/fs/sockfs: undefined symbol 'sctp_connect'
    WARNING: mod_load: cannot load module 'sockfs'
    WARNING: sockfs: unable to resolve dependency, module 'drv/ip' not found
    /kernel/sys/c2audit: undefined symbol 'getsonode'
    WARNING: mod_load: cannot load module 'c2audit'
    WARNING: c2audit: unable to resolve dependency, module 'fs/sockfs' not found
    /kernel/misc/strplumb: undefined symbol 'dl_attach'
    /kernel/misc/strplumb: undefined symbol 'dl_phys_addr'
    /kernel/misc/strplumb: undefined symbol 'dhcacklen'
    /kernel/misc/strplumb: undefined symbol 'dhcifname'
    /kernel/misc/strplumb: undefined symbol 'dl_bind'
    WARNING: mod_load: cannot load module 'strplumb'
    panic[cpu0]/thread=f502c3a0: mod_hold_stub: Couldn't load stub module misc/strplumb
    f5056c84 genunix:mod_hold_stub+161 (f5006070, 1, f4d38f)
    f5056ca0 unix:stubs_common_code+9 ()
    syncing file systems... done
    skipping system dump - no dump device configured
    rebooting...

    ]]>     http://andyleonard.com/2008/06/25/no-luck-with-a-quick-n-dirty-bfu-of-sxce-79-on-ec2/feed/ 0     Linkage, 6/24/2008 http://andyleonard.com/2008/06/24/linkage-6242008/ http://andyleonard.com/2008/06/24/linkage-6242008/#comments Tue, 24 Jun 2008 23:19:37 +0000 Andy http://andyleonard.com/?p=39
  • S3 data corruption: “We’ve isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3′s total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted.” (Seen at Daemonic Dispatches.)
    • ]]>     http://andyleonard.com/2008/06/24/linkage-6242008/feed/ 0