That leads neatly to the second thing, which is from the iiNet blog, entitled The benefits of working from home. As with the above video, Yahoo gets a prominent mention for not allowing telework (although reports on that seem to be somewhat mixed). Anyway, benefits cited include lack of commute, lower overhead and fewer distractions, but the blog post does also make the very good point that you need to figure out whether or not telework is actually right for you. It also offers some tips I tend to agree with.

Now we get to the semi-related bit. Assuming telework is right for you, you want a good internet connection. As I’ve said before, “Yay NBN! Bring it on!“. As much disenchantment as I have with the major parties, this is something Labor is actually doing right. For an easy-to-appreciate comparison of what Labor and the LNP are proposing, check out How Fast is the NBN. Teleworkers, pay particular attention to the simulated Dropbox sync.

On each of my ceph nodes, a bit of upgrading and cleanup. Note the choice of ceph 0.60 was mostly arbitrary, I just wanted the latest thing I could find an RPM for in a hurry. Also some of the rm invocations won’t be necessary, depending on what state things are actually in:

# zypper ar -f http://download.opensuse.org/repositories/home:/dalgaaf:/ceph:/extra/openSUSE_12.3/home:dalgaaf:ceph:extra.repo
# zypper ar -f http://gitbuilder.ceph.com/ceph-rpm-opensuse12-x86_64-basic/ref/next/x86_64/ ceph.com-next_openSUSE_12_x86_64
# zypper in ceph-0.60
# kill $(pidof ceph-mon)
# rm /etc/ceph/*
# rm /var/run/ceph/*
# rm -r /var/lib/ceph/*/*

That last gets rid of any half-created mon directories.

I also edited the Ceph environment to only have one mon (one of my colleagues rightly pointed out that you need an odd number of mons, and I had declared two previously, for no good reason). That’s knife environment edit Ceph on my desktop, and set "mon_initial_members": "ceph-0" instead of "ceph-0,ceph-1".

I also had to edit each of the nodes, to add an osd_devices array to each node, and remove the mon role from ceph-1. That’s knife node edit ceph-0.example.com then insert:

  "normal": {
    ...
    "ceph": {
      "osd_devices": [  ]
    }
  ...

Without the osd_devices array defined, the osd recipe fails (“undefined method `each_with_index’ for nil:NilClass”). I was kind of hoping an empty osd_devices array would allow ceph to use the root partition. No such luck, the cookbook really does expect you to be doing a sensible deployment with actual separate devices for your OSDs. Oh, well. I’ll try that another time. For now at least I’ve demonstrated that ceph-0.60 does give you what appears to be a clean mon setup when using the upstream cookbooks on openSUSE 12.3:

knife ssh name:ceph-0.example.com -x root chef-client
[2013-04-15T06:32:13+00:00] INFO: *** Chef 10.24.0 ***
[2013-04-15T06:32:13+00:00] INFO: Run List is [role[ceph-mon], role[ceph-osd], role[ceph-mds]]
[2013-04-15T06:32:13+00:00] INFO: Run List expands to [ceph::mon, ceph::osd, ceph::mds]
[2013-04-15T06:32:13+00:00] INFO: HTTP Request Returned 404 Not Found: No routes match the request: /reports/nodes/ceph-0.example.com/runs
[2013-04-15T06:32:13+00:00] INFO: Starting Chef Run for ceph-0.example.com
[2013-04-15T06:32:13+00:00] INFO: Running start handlers
[2013-04-15T06:32:13+00:00] INFO: Start handlers complete.
[2013-04-15T06:32:13+00:00] INFO: Loading cookbooks [apache2, apt, ceph]
[2013-04-15T06:32:13+00:00] INFO: Processing template[/etc/ceph/ceph.conf] action create (ceph::conf line 6)
[2013-04-15T06:32:13+00:00] INFO: template[/etc/ceph/ceph.conf] updated content
[2013-04-15T06:32:13+00:00] INFO: template[/etc/ceph/ceph.conf] mode changed to 644
[2013-04-15T06:32:13+00:00] INFO: Processing service[ceph_mon] action nothing (ceph::mon line 23)
[2013-04-15T06:32:13+00:00] INFO: Processing execute[ceph-mon mkfs] action run (ceph::mon line 40)
creating /var/lib/ceph/tmp/ceph-ceph-0.mon.keyring
added entity mon. auth auth(auid = 18446744073709551615 key=AQC8umZRaDlKKBAAqD8li3u2JObepmzFzDPM3g== with 0 caps)
ceph-mon: mon.noname-a 192.168.4.118:6789/0 is local, renaming to mon.ceph-0
ceph-mon: set fsid to f80aba97-26c5-4aa3-971e-09c5a3afa32f
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-0 for mon.ceph-0
[2013-04-15T06:32:14+00:00] INFO: execute[ceph-mon mkfs] ran successfully
[2013-04-15T06:32:14+00:00] INFO: execute[ceph-mon mkfs] sending start action to service[ceph_mon] (immediate)
[2013-04-15T06:32:14+00:00] INFO: Processing service[ceph_mon] action start (ceph::mon line 23)
[2013-04-15T06:32:15+00:00] INFO: service[ceph_mon] started
[2013-04-15T06:32:15+00:00] INFO: Processing ruby_block[tell ceph-mon about its peers] action create (ceph::mon line 64)
mon already active; ignoring bootstrap hint

[2013-04-15T06:32:16+00:00] INFO: ruby_block[tell ceph-mon about its peers] called
[2013-04-15T06:32:16+00:00] INFO: Processing ruby_block[get osd-bootstrap keyring] action create (ceph::mon line 79)
2013-04-15 06:32:16.872040 7fca8e297780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
2013-04-15 06:32:16.872042 7fca8e297780 -1 unable to authenticate as client.admin
2013-04-15 06:32:16.872400 7fca8e297780 -1 ceph_tool_common_init failed.
[2013-04-15T06:32:18+00:00] INFO: ruby_block[get osd-bootstrap keyring] called
[2013-04-15T06:32:18+00:00] INFO: Processing package[gdisk] action upgrade (ceph::osd line 37)
[2013-04-15T06:32:27+00:00] INFO: package[gdisk] upgraded from uninstalled to
[2013-04-15T06:32:27+00:00] INFO: Processing service[ceph_osd] action nothing (ceph::osd line 48)
[2013-04-15T06:32:27+00:00] INFO: Processing directory[/var/lib/ceph/bootstrap-osd] action create (ceph::osd line 67)
[2013-04-15T06:32:27+00:00] INFO: Processing file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] action create (ceph::osd line 76)
[2013-04-15T06:32:27+00:00] INFO: entered create
[2013-04-15T06:32:27+00:00] INFO: file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] owner changed to 0m
[2013-04-15T06:32:27+00:00] INFO: file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] group changed to 0
[2013-04-15T06:32:27+00:00] INFO: file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] mode changed to 440
[2013-04-15T06:32:27+00:00] INFO: file[/var/lib/ceph/bootstrap-osd/ceph.keyring.raw] created file /var/lib/ceph/bootstrap-osd/ceph.keyring.raw
[2013-04-15T06:32:27+00:00] INFO: Processing execute[format as keyring] action run (ceph::osd line 83)
creating /var/lib/ceph/bootstrap-osd/ceph.keyring
added entity client.bootstrap-osd auth auth(auid = 18446744073709551615 key=AQAOl2tR0M4bMRAAatSlUh2KP9hGBBAP6u5AUA== with 0 caps)
[2013-04-15T06:32:27+00:00] INFO: execute[format as keyring] ran successfully
[2013-04-15T06:32:28+00:00] INFO: Chef Run complete in 14.479108446 seconds
[2013-04-15T06:32:28+00:00] INFO: Running report handlers
[2013-04-15T06:32:28+00:00] INFO: Report handlers complete

Witness:

ceph-0:~ # rcceph status
=== mon.ceph-0 ===
mon.ceph-0: running {"version":"0.60-468-g98de67d"}

On the note of building an easy-to-deploy Ceph appliance, assuming you’re not using Chef and just want something to play with, I reckon the way to go is use config pretty similar to what would be deployed by this Chef cookbook, i.e. an absolute minimal /etc/ceph/ceph.conf, specifying nothing other than initial mons, then use the various Ceph CLI tools to create mons and osds on each node and just rely on the init script in Ceph >= 0.58 to do the right thing with what it finds (having to explicitly specify each mon, osd and mds in the Ceph config by name always bugged me). Bonus points for using csync2 to propagate /etc/ceph/ceph.conf across the cluster.

First things first, need a Chef server, so I installed openSUSE 12.3 on a VM, then installed Chef 10 on that, roughly following the manual installation instructions. Note for those following along at home – sometimes the blocks I’ve copied here are just commands, sometimes they include command output as well. You’ll figure it out

# zypper ar -f http://download.opensuse.org/repositories/systemsmanagement:/chef:/10/openSUSE_12.3/systemsmanagement:chef:10.repo
# zypper in rubygem-chef-server
# chkconfig couchdb on
# rccouchdb start
# chkconfig rabbitmq-server on
# rcrabbitmq-server start
# rabbitmqctl add_vhost /chef
# rabbitmqctl add_user chef testing
# rabbitmqctl set_permissions -p /chef chef ".*" ".*" ".*"
# for service in solr expander server server-webui; do
      chkconfig chef-$service on
      rcchef-$service start
  done

I didn’t bother editing /etc/chef/server.rb, the config as shipped works fine (not that the AMQP password is very secure, mind). The only catch is the web UI didn’t start. IIRC this is due to /etc/chef/webui.pem not existing yet (chef-server creates it, but this doesn’t finish until later).

Then configured knife:

# knife configure -i
WARNING: No knife configuration file found
Where should I put the config file? [/root/.chef/knife.rb]
Please enter the chef server URL: [http://os-chef.example.com:4000]
Please enter a clientname for the new client: [root]
Please enter the existing admin clientname: [chef-webui]
Please enter the location of the existing admin client's private key: [/etc/chef/webui.pem]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem]
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Created client[root]
Configuration file written to /root/.chef/knife.rb

And make a client for me:

# knife client create tserong -d -a -f /tmp/tserong.pem
Created client[tserong]

Then set up my desktop as a Chef workstation (roughly following these docs, and again pulling Chef from systemsmanagement:chef:10 on OBS):

# sudo zypper in rubygem-chef
# cd ~
# git clone git://github.com/opscode/chef-repo.git
# cd chef-repo
# mkdir -p ~/.chef
# scp root@os-chef:/etc/chef/validation.pem ~/.chef/
# scp root@os-chef:/tmp/tserong.pem ~/.chef/
# knife configure
WARNING: No knife configuration file found
Where should I put the config file? [/home/tserong/.chef/knife.rb]
Please enter the chef server URL: [http://desktop.example.com:4000] http://os-chef.example.com:4000
Please enter an existing username or clientname for the API: [tserong]
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef/validation.pem] /home/tserong/.chef/validation.pem
Please enter the path to a chef repository (or leave blank): /home/tserong/chef-repo
[...]
Configuration file written to /home/tserong/.chef/knife.rb

Make sure it works:

# knife client list
chef-validator
chef-webui
root
tserong

Grab the cookbooks and upload them to the Chef server. The Ceph cookbook claims to depend on apache and apt, although presumably the former is only necessary for RADOSGW, and the latter for Debian-based systems. Anyway:

# cd ~/chef-repo
# git submodule add git@github.com:opscode-cookbooks/apache2.git cookbooks/apache2
# git submodule add git@github.com:opscode-cookbooks/apt.git cookbooks/apt
# git submodule add git@github.com:ceph/ceph-cookbooks.git cookbooks/ceph
# knife cookbook upload apache2
# knife cookbook upload apt
# knife cookbook upload ceph

Boot up a couple more VMs to be Ceph nodes, using the appliance image from last time. These need chef-client installed, and need to be registered with the chef server. knife bootstrap will install chef-client and dependencies for you, but after looking at the source, if /usr/bin/chef doesn’t exist, it actually uses wget or curl to pull http://opscode.com/chef/install.sh and runs that. How this is considered a good idea is completely baffling to me, so again I installed our chef build from OBS on each of my Ceph nodes (note to self: should add this to appliance image on Studio):

# zypper ar -f http://download.opensuse.org/repositories/systemsmanagement:/chef:/10/openSUSE_12.3/systemsmanagement:chef:10.repo
# zypper in rubygem-chef

And ran the now-arguably-safe knife bootstrap from my desktop:

# knife bootstrap ceph-0.example.com
Bootstrapping Chef on ceph-0.example.com
[...]
# knife bootstrap ceph-1.example.com
Bootstrapping Chef on ceph-1.example.com
[...]

Then, roughly following the Ceph Deploying with Chef document.

Generate a UUID and monitor secret (had to do the latter on one of my Ceph VMs, as ceph-authtool is conveniently already installed):

# uuidgen -r
f80aba97-26c5-4aa3-971e-09c5a3afa32f
# ceph-authtool /dev/stdout --name=mon. --gen-key
[mon.]
key = AQC8umZRaDlKKBAAqD8li3u2JObepmzFzDPM3g==

Then on my desktop:

knife environment create Ceph

This I filled in with:

{
  "name": "Ceph",
  "description": "",
  "cookbook_versions": {
  },
  "json_class": "Chef::Environment",
  "chef_type": "environment",
  "default_attributes": {
    "ceph": {
      "monitor-secret": "AQC8umZRaDlKKBAAqD8li3u2JObepmzFzDPM3g==",
      "config": {
        "fsid": "f80aba97-26c5-4aa3-971e-09c5a3afa32f",
        "mon_initial_members": "ceph-0,ceph-1",
        "global": {
        },
        "osd": {
          "osd journal size": "1000",
          "filestore xattr use omap": "true"
        }
      }
    }
  },
  "override_attributes": {
  }
}

Uploaded roles:

# knife role from file cookbooks/ceph/roles/ceph-mds.rb
# knife role from file cookbooks/ceph/roles/ceph-mon.rb
# knife role from file cookbooks/ceph/roles/ceph-osd.rb
# knife role from file cookbooks/ceph/roles/ceph-radosgw.rb

Assigned roles to nodes:

# knife node run_list add ceph-0.example.com 'role[ceph-mon],role[ceph-osd],role[ceph-mds]'
# knife node run_list add ceph-1.example.com 'role[ceph-mon],role[ceph-osd],role[ceph-mds]'

I didn’t bother with recipe[ceph::repo] as I don’t care about installation right now (Ceph is already installed in my VM images).

Had to set "chef_environment": "Ceph" for each node by running:

# knife node edit ceph-0.example.com
# knife node edit ceph-1.example.com

Didn’t set Ceph osd_devices per node – I’m just playing, so can sit on top of the root partition.

Now let’s see if it works:

# knife ssh name:ceph-0.example.com -x root chef-client
[2013-04-11T13:44:47+00:00] INFO: *** Chef 10.24.0 ***
[2013-04-11T13:44:48+00:00] INFO: Run List is [role[ceph-mon], role[ceph-osd], role[ceph-mds]]
[2013-04-11T13:44:48+00:00] INFO: Run List expands to [ceph::mon, ceph::osd, ceph::mds]
[2013-04-11T13:44:48+00:00] INFO: HTTP Request Returned 404 Not Found: No routes match the request: /reports/nodes/ceph-0.example.com/runs
[2013-04-11T13:44:48+00:00] INFO: Starting Chef Run for ceph-0.example.com
[2013-04-11T13:44:48+00:00] INFO: Running start handlers
[2013-04-11T13:44:48+00:00] INFO: Start handlers complete.
[2013-04-11T13:44:48+00:00] INFO: Loading cookbooks [apache2, apt, ceph]
No ceph-mon found.

[2013-04-11T13:44:48+00:00] INFO: Processing template[/etc/ceph/ceph.conf] action create (ceph::conf line 6)
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] backed up to /var/chef/backup/etc/ceph/ceph.conf.chef-20130411134448
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] updated content
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] owner changed to 0
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] group changed to 0
[2013-04-11T13:44:48+00:00] INFO: template[/etc/ceph/ceph.conf] mode changed to 644
[2013-04-11T13:44:48+00:00] INFO: Processing service[ceph_mon] action nothing (ceph::mon line 23)
[2013-04-11T13:44:48+00:00] INFO: Processing execute[ceph-mon mkfs] action run (ceph::mon line 40)
creating /var/lib/ceph/tmp/ceph-ceph-0.mon.keyring
added entity mon. auth auth(auid = 18446744073709551615 key=AQC8umZRaDlKKBAAqD8li3u2JObepmzFzDPM3g== with 0 caps)
ceph-mon: mon.noname-a 192.168.4.118:6789/0 is local, renaming to mon.ceph-0
ceph-mon: set fsid to f80aba97-26c5-4aa3-971e-09c5a3afa32f
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ceph-0 for mon.ceph-0
[2013-04-11T13:44:49+00:00] INFO: execute[ceph-mon mkfs] ran successfully
[2013-04-11T13:44:49+00:00] INFO: execute[ceph-mon mkfs] sending start action to service[ceph_mon] (immediate)
[2013-04-11T13:44:49+00:00] INFO: Processing service[ceph_mon] action start (ceph::mon line 23)
[2013-04-11T13:44:49+00:00] INFO: service[ceph_mon] started
[2013-04-11T13:44:49+00:00] INFO: Processing ruby_block[tell ceph-mon about its peers] action create (ceph::mon line 64)
connect to
/var/run/ceph/ceph-mon.ceph-0.asok
failed with
(2) No such file or directory

connect to
/var/run/ceph/ceph-mon.ceph-0.asok
failed with
(2) No such file or directory

[2013-04-11T13:44:49+00:00] INFO: ruby_block[tell ceph-mon about its peers] called
[2013-04-11T13:44:49+00:00] INFO: Processing ruby_block[get osd-bootstrap keyring] action create (ceph::mon line 79)
2013-04-11 13:44:49.928800 7f58e9677700 0
-- :/23863 >> 192.168.4.117:6789/0 pipe(0x18f0d30 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault

2013-04-11 13:44:52.928739 7f58efc1c700 0 -- :/23863 >> 192.168.4.118:6789/0 pipe(0x7f58e0000c00 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 13:44:55.929375 7f58e9677700 0 -- :/23863 >> 192.168.4.117:6789/0 pipe(0x7f58e0003010 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 13:44:58.929211 7f58efc1c700 0 -- :/23863 >> 192.168.4.118:6789/0 pipe(0x7f58e00039f0 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 13:45:01.929787 7f58e9677700 0 -- :/23863 >> 192.168.4.117:6789/0 pipe(0x7f58e00023b0 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
[...]

And it’s stuck there, trying and failing to talk to something.

See those “no such file or directory” errors after “service[ceph_mon] started”? Yeah? Well, the mon isn’t started, hence the missing sockets in /var/run/ceph.

Why isn’t the mon started? Turns out the ceph init script won’t start any mon (or osd or mds for that matter) if you don’t have entries in the config file with some suffix, e.g. [mon.a]. And all I’ve got is:

[global]
  fsid =  f80aba97-26c5-4aa3-971e-09c5a3afa32f
  mon initial members = ceph-0,ceph-1
  mon host = 192.168.4.118:6789, 192.168.4.117:6789

[osd]
    osd journal size = 1000
    filestore xattr use omap = true

But given the mon recipe triggers ceph-mon-all-starter if using upstart (which it would be, on the “Tested as working: Ubuntu Precise”), and ceph-mon-all-starter seems to just ultimately run something like ceph-mon --cluster=ceph -i ceph-0 regardless of what’s in the config file… Maybe I can cheat.

Directly starting ceph-mon from a shell on ceph-0 before the chef-client run turned out to be a bad idea (bit of a chicken and egg problem figuring out what to inject into the “mon host” line of the config file). So I put a bit of evil into the mon recipe:

diff --git a/recipes/mon.rb b/recipes/mon.rb
index 5cd76de..a518830 100644
--- a/recipes/mon.rb
+++ b/recipes/mon.rb
@@ -61,6 +61,10 @@ EOH
   notifies :start, "service[ceph_mon]", :immediately
 end

+execute 'hack to force mon start' do
+  command "ceph-mon --cluster=ceph -i #{node['hostname']}"
+end
+
 ruby_block "tell ceph-mon about its peers" do
   block do
     mon_addresses = get_mon_addresses()

Try again:

# knife ssh name:ceph-0.example.com -x root chef-client
[2013-04-11T15:10:43+00:00] INFO: *** Chef 10.24.0 ***
[2013-04-11T15:10:44+00:00] INFO: Run List is [role[ceph-mon], role[ceph-osd], role[ceph-mds]]
[2013-04-11T15:10:44+00:00] INFO: Run List expands to [ceph::mon, ceph::osd, ceph::mds]
[2013-04-11T15:10:44+00:00] INFO: HTTP Request Returned 404 Not Found: No routes match the request: /reports/nodes/ceph-0.example.com/runs
[2013-04-11T15:10:44+00:00] INFO: Starting Chef Run for ceph-0.example.com
[2013-04-11T15:10:44+00:00] INFO: Running start handlers
[2013-04-11T15:10:44+00:00] INFO: Start handlers complete.
[2013-04-11T15:10:44+00:00] INFO: Loading cookbooks [apache2, apt, ceph]
[2013-04-11T15:10:44+00:00] INFO: Storing updated cookbooks/ceph/recipes/mon.rb in the cache.
No ceph-mon found.

[2013-04-11T15:10:44+00:00] INFO: Processing template[/etc/ceph/ceph.conf] action create (ceph::conf line 6)
[2013-04-11T15:10:44+00:00] INFO: Processing service[ceph_mon] action nothing (ceph::mon line 23)
[2013-04-11T15:10:44+00:00] INFO: Processing execute[ceph-mon mkfs] action run (ceph::mon line 40)
[2013-04-11T15:10:44+00:00] INFO: Processing execute[hack to force mon start] action run (ceph::mon line 65)
starting mon.ceph-0 rank 1 at 192.168.4.118:6789/0 mon_data /var/lib/ceph/mon/ceph-ceph-0 fsid f80aba97-26c5-4aa3-971e-09c5a3afa32f
[2013-04-11T15:10:44+00:00] INFO: execute[hack to force mon start] ran successfully
[2013-04-11T15:10:44+00:00] INFO: Processing ruby_block[tell ceph-mon about its peers] action create (ceph::mon line 69)
adding peer 192.168.4.118:6789/0 to list: 192.168.4.117:6789/0,192.168.4.118:6789/0

adding peer 192.168.4.117:6789/0 to list: 192.168.4.117:6789/0,192.168.4.118:6789/0

[2013-04-11T15:10:44+00:00] INFO: ruby_block[tell ceph-mon about its peers] called
[2013-04-11T15:10:44+00:00] INFO: Processing ruby_block[get osd-bootstrap keyring] action create (ceph::mon line 84)
2013-04-11 15:10:44.432266 7f8f9f8c0700  0
-- :/25965 >> 192.168.4.117:6789/0 pipe(0x16d9d30 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault

2013-04-11 15:10:50.433053 7f8f9f7bf700  0 -- 192.168.4.118:0/25965 >> 192.168.4.117:6789/0 pipe(0x7f8f94001d30 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 15:10:56.433268 7f8fa5e65700  0 -- 192.168.4.118:0/25965 >> 192.168.4.117:6789/0 pipe(0x7f8f94001d30 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 15:11:02.433987 7f8f9f8c0700  0 -- 192.168.4.118:0/25965 >> 192.168.4.117:6789/0 pipe(0x7f8f94002db0 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault
2013-04-11 15:11:08.434358 7f8f9f7bf700  0 -- 192.168.4.118:0/25965 >> 192.168.4.117:6789/0 pipe(0x7f8f94004fb0 sd=3 :0 s=1 pgs=0 cs=0 l=1).fault

At this point it’s stalled presumably waiting to talk to the other mon, so in another terminal window had to kick off a chef-client run on ceph-1 to get it into the same state as ceph-0 (knife ssh name:ceph-1.example.com -x root chef-client). This allowed both nodes to progress to the next problem:

2013-04-11 15:11:28.563438 7f8fa5e67780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
2013-04-11 15:11:28.563443 7f8fa5e67780 -1 unable to authenticate as client.admin
2013-04-11 15:11:28.563814 7f8fa5e67780 -1 ceph_tool_common_init failed.
2013-04-11 15:11:29.572208 7f2369130780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
2013-04-11 15:11:29.572210 7f2369130780 -1 unable to authenticate as client.admin
2013-04-11 15:11:29.572527 7f2369130780 -1 ceph_tool_common_init failed.
2013-04-11 15:11:31.380073 7f1907d18780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
2013-04-11 15:11:31.380078 7f1907d18780 -1 unable to authenticate as client.admin
2013-04-11 15:11:31.380720 7f1907d18780 -1 ceph_tool_common_init failed.
2013-04-11 15:11:32.392345 7fc2bc462780 -1 monclient(hunting): authenticate NOTE: no keyring found; disabled cephx authentication
[...]

And we’re spinning again.

But that’s enough for one day.

From the web server logs, the JSEC tutorial (and, for that matter, the KeyGlove project, which happened at a similar time), are still pretty high on the list of search terms and entry pages that land random visitors on wirejunkie.com. And I still think the JSEC tutorial is a decent, well-structured tutorial, insofar as it leads one through a certain development process, but I did realise at some point that even though the code all still works in modern web browsers, the tutorial no longer necessarily instructs best practice. For example, it presents a makeEmptyArray() function and associated scaffolding necessary for running on Netscape 2.0.

You all remember Netscape 2.0, right?

But even though the content is outdated, I could never bring myself to take the JSEC tutorial down, because cool URIs don’t change. This is a positive restatement of the old saw ”I hope you know that this will go down on your permanent record”. Consequently I added a note at the top of the table of contents to say:

Please note that this tutorial was first published in the year 2000, and is based on work conducted between 1997 and 2000. While we trust that it still constitutes useful tutorial material, it should not necessarily be construed as to impart best practice in the year $THIS_YEAR.

That’s suitably cautionary, but I really want to take the opportunity to state clearly that if I were building an online shop today, I would probably not do so as outlined in the JSEC tutorial. There are several reasons for this, the two most obvious being:

1. The framework presented uses <frameset> heavily. This will almost certainly provide a poor experience for mobile web users, which is a huge number of people these days. I suspect it’s also problematic for people using screen readers.
2. At the time the tutorial was written, server-side (PHP/Python/Ruby etc.) ecommerce frameworks were either rare, immature, expensive, nonexistent, or perhaps all of the above. I rather hope think things have changed in the meantime, which makes a pure JavaScript solution less interesting than it used to be.

In summary, the JSEC tutorial will remain online for all to enjoy, but do please be aware of the historical context in which it was created.

(I also find it entertaining that I can believably use the term “historical context” after only a decade or so. Aaah, technology….)

I tried making a SLES 11 SP2 appliance with SUSE Studio. I had to add the filesystems and devel:libraries:c_c++ repos from OBS to get reasonably up-to-date Ceph 0.56 and libboost_thread.so.1.49.0, but on boot when the appliance tried to expand its root filesystem, it died claiming it couldn’t load libe2p.so.2. Studio claims to be pulling in e2fsprogs from both the SP2 Updates and filesystems repo, so maybe that’s the problem. It seems impossible to choose one or the other, as they are the same version. (Update: it was just pointed out to me that you can click the little box next to the version number to choose which one is installed – must try again.)

So I left that alone and tried an openSUSE 12.3 appliance. The filesystems/ceph build for 12.3 is disabled, so I branched it and kicked off a build which failed with an exciting OOM error:

[ 3831s] [ 3803.167109] Out of memory: Kill process 16364 (cc1plus) score 254 or sacrifice child
[ 3831s] [ 3803.167959] Killed process 16364 (cc1plus) total-vm:825128kB, anon-rss:168760kB, file-rss:4kB
[ 3831s] g++: internal compiler error: Killed (program cc1plus)
[ 3831s] Please submit a full bug report,
[ 3831s] with preprocessed source if appropriate.
[ 3831s] See  for instructions.

Guess I should do what it says and file a bug. But I really did want something to play with immediately, so I added http://ceph.com/rpm/opensuse12/x86_64/ as a repo, and pulled in the upstream Ceph 0.56 RPMs. This seems to have worked and given me an openSUSE 12.3 image I can use to run through the Ceph 5-Minute Quick Start, Block Device Quick Start and CephFS Quick Start. So, here’s my extremely terse openSUSEified version of those quick start documents:

5-Minute Quick Start

Deploy the Appliance Image

I’m doing this with a couple of VMs, so in my case I make a couple of copies of the image:

# cp ~/openSUSE_12.3_Ceph_0.56.x86_64-0.0.3.qcow2 \
    /var/lib/libvirt/images/ceph-quickstart-server.qcow2
# cp ~/openSUSE_12.3_Ceph_0.56.x86_64-0.0.3.qcow2 \
    /var/lib/libvirt/images/ceph-quickstart-client.qcow2

Then I use virt-manager to create two VMs, backed by those images. Boot ‘em up, log in (root password is “linux”), run yast network and set sensible hostnames (“ceph-client” and “ceph-server” instead of “linux-kjqd”, although admittedly those names wouldn’t be very sensible in a real deployment with more than one node).

Edit the Configuration File

The appliance image includes the /etc/ceph/ceph.conf file from the original 5-minute quick start, so log in to ceph-server, edit that file and replace {hostname} and {ip-address} with their real values, then copy the configuration file to ceph-client:

# scp /etc/ceph/ceph.conf ceph-client:/etc/ceph/

Deploy the Configuration

On ceph-server, create directories for each daemon:

# mkdir -p /var/lib/ceph/osd/ceph-0
# mkdir -p /var/lib/ceph/osd/ceph-1
# mkdir -p /var/lib/ceph/mon/ceph-a
# mkdir -p /var/lib/ceph/mds/ceph-a

Still on ceph-server, run the following:

# cd /etc/ceph
# mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring

Start Ceph

On ceph-server:

# chkconfig ceph on
# rcceph start
# ceph health

This will initially show something like:

HEALTH_ERR 576 pgs stuck inactive; 576 pgs stuck unclean; no osds

Eventually it will say HEALTH_OK and you’re good to go.

Copy the Keyring to the Client

This is necessary for authentication:

# scp /etc/ceph/ceph.keyring ceph-client:/etc/ceph/

Block Device Quick Start

On ceph-client:

# rbd create foo --size 4096
# modprobe rbd
# rbd map foo --pool rbd --name client.admin
# mkfs.ext4 -m0 /dev/rbd1
# mkdir /mnt/myrbd
# mount /dev/rbd1 /mnt/myrbd

(Why is this /dev/rbd1, not /dev/rbd/rbd/foo as in the original quick start?)

CephFS Quick Start

On ceph-client (kernel driver, not FUSE):

# mkdir /mnt/mycephfs
# mount -t ceph -o name=admin,secret=$(ceph-authtool \
    --name client.admin /etc/ceph/ceph.keyring --print-key) \
    ceph-server:/ /mnt/mycephfs

Interestingly, this gives “mount: error writing /etc/mtab: Invalid argument”, but still seems to actually mount the filesystem.

Also note that it appears I have 32GB of space for Ceph to use, even though ceph-server only has a 16GB root partition. I rather think that’s because there’s two OSDs, but both are just running off the root filesystem, they’re not separate disks/filesystems. I assume this is one of those Don’t Try This At Home things.

My new Lenovo T430 arrived last week. After delighting in that satisfying new laptop smell, I made recovery DVDs I will presumably never need, then blew away Windows 7 and installed openSUSE 12.3 (full disclosure: I work for SUSE, so my choice of distro may not be entirely unbiased).

Some niceties:

• The textured touchpad is lovely. Much better feel than a pure flat surface.
• As I’d expect, the keyboard is excellent (even if PGUP/PGDN aren’t where I’m used to).
• The openSUSE installer is quick and easy. I’m pretty sure there’s less steps than last time I did a regular openSUSE install from scratch a couple of years ago.
• No problem setting up encrypted LVM, although on my ~500GB drive it defaults to a 20GB root and 25GB /home, with a whole lotta free space left over in the encrypted partition, so that might want some tweaking.
• Entering the passphrase on boot happens on a pretty graphical screen, you don’t get thrown back to a terminal window where random junk is appearing over the passphrase entry prompt.
• Moving my mail over from my old laptop was pretty much just an rsync of the Thunderbird profile directory (and maybe a tweak to ~/.thunderbird/profiles.ini)

Some oddities:

• The Novell GroupWise 8.0.2 client had a couple of problems:
• It claims to need libXm.so.3 (listed in RPM Requires), but works fine without it. This is fortunate, because openSUSE 12.3 doesn’t ship openmotif22-libs-32bit anymore.
• Unless you’ve installed libpangox-1_0-0-32bit, the GroupWise client will segfault somewhere in libwebrenderer.so. This is less than obvious.
• The YaST disk partitioner seems slightly confused adding new LVs inside my encrypted VG later on (it either locked up or crashed). I haven’t had time to investigate this properly, so I’ve ignored it for the moment and used lvcreate and mkfs in a terminal instead.
• You do need to reboot at least once after initial install for NetworkManager to work properly (this is mentioned in the release notes).
• I’m running GNOME 3.6, and I tried using the tweak tool to have it just blank the screen – not suspend – when closing the laptop lid. Turns out systemd is being too clever for me, so I had to fiddle with that a bit (set HandleLidSwitch=ignore in /etc/systemd/logind.conf, then run sudo systemctl restart systemd-logind).

Very little else to report so far. Aside from the oddities above everything else seems to Just Work^TM. OTOH, all I’ve really done is web browsing, email and assorted fiddling around in terminals. Maybe listened to a bit of music (the inbuilt speakers are well and truly loud enough, but a bit tinnier than real speakers – can’t say I’m terribly surprised by that though).

Two things in particular stuck with me, and I thought it worth noting them here.

1) Professor Singer is in a field known as Applied Ethics. At some time in the past there was only Ethics. My inference is that the latter group are talking about – and thinking about – ethics, but not actually behaving any differently as a result of their cogitations. I find this notion simultaneously hilarious and horrifying.

2) At one point while speaking about living ethically, Professor Singer said that if you look back at the end of each day and say to yourself “well, I didn’t lie, cheat or steal, and I didn’t maim anybody”, you’re setting the bar too low. It would be better, he suggested, to look back and say “what did I do to improve the world today, or to help someone else in some way?” This seems like a pretty good approach to me.

A flock of chickens (whether or not any roosters are present) will establish a pecking order. This shouldn’t be much of a surprise. They will, quite literally, peck each other to establish and maintain dominance or order.

Some groups of chickens (say, two or three) might seem to enjoy each others’ company more than the rest of the flock and will tend to hang out together.

If a chicken becomes ill, or too weak to defend itself, it’s quite likely that other members of the flock will peck it to death, unless you segregate the sick chook until it gets better.

If there are roosters, they will tend to fight each other. Less so if they were raised together and have known each other their whole lives, but even then there’ll still be some scrapping. Roosters will also look after the hens. They will go forth and find food, then make the *dook* *dook* *dook* there’s-food-over-here noise to call their ladies over. And you really do get the sense that a rooster believes they are his ladies. If another rooster tries to mate with one of the hens and the boss rooster notices, he’ll attack the up and comer. He’ll even attack you, the human, trying to give the chickens food, if he thinks you’re up to no good. But by and large, these are actually all good qualities in roosters.

All this observation got me to thinking… Maybe the bad behaviour we humans exhibit – the sexism, the racism, the put-downs, the fighting, all that, maybe that’s because we’ve forgotten (or, worse yet, never learned) that we’re not chickens anymore. How many human males behave just like roosters, with little or no regard for the fact that women don’t actually need men to find food for them? And, when I mentioned my thesis to a female friend, she pointed out – how many women still behave like hens? How many of us peck at people we think are inferior to, or less knowledgeable than, ourselves?

The life of a chicken seems to consist mostly of eating, fucking, fighting and making little baby chickens. Humans do all these things too – and should be enjoying most of them – but I’d like to think that in the 313-330 million years since the common ancestor of humans and birds split, we’d have learned a thing or two about how to treat each other with decency and respect.

The Talk

[introduction, audience applauds politely]

Thank you.

This is a gratuitous photo from the top of Hartz Peak in Southern Tasmania. If I were to light the Beacon Fire of Amon Din at my house, you could possibly see it from here.

I’m here to tell a story about Staying Sane and Productive While Hacking Free and Open Source Software on a Farm.

I work for SUSE, which is an enterprise Linux company that I hope most of everybody here has heard of. Mostly I work on High Availability and Cloud Software; Linux-HA, Pacemaker, OpenStack, Crowbar.

I’ve been working from home exclusively for just under four years, most of that time at SUSE, and for the last two and a half years I’ve been doing this from a small farm in southern Tasmania.

This talk is a little bit about farming, and it’s a bit about teleworking. It’s about work/life balance, but I hate that term, because it’s like saying your work is not part of your life… But we spend so much time working – when you meet a new person they say “what do you do” and you tell them what your job is.

So I would prefer a term like “life balance”, or “harmony”, or something like that. But there is actually a good word for this, which is “eudaimonia”, a concept which appears in the works of Aristotle. At the time that probably meant “right living”, except that doesn’t really mean anything unless you already know what it means. More modernly it might be described as “flourishing”. If you look at the wikipedia page for eudaimonia, someone in modern psychology has described it as six things; autonomy, personal growth, self acceptance, purpose in life, environmental mastery and positive relations with others. All of which sound really good to me.

A bit of backstory – my wife and I have this crazy idea that sustainability and self-sufficiency are worthy things to try to achieve, and my work with high availability makes me a little bit professionally paranoid about the power going out and that sort of thing, and that mindset kind of fits well there too. Growing and raising your own food can get you a reasonable distance down that path, I think. I also think that it’s better for the environment, it’s better for your food, and it’s better for you. And it’s a lot easier to spend time doing this if you work from home.

We’ve been gardening pretty much forever – fruit, vegetables, that sort of thing, and for most of forever we’ve also kept chickens, which give you the best free range eggs and meat in the world. It’s awesome. They’ll eat kitchen scraps, they’ll do the weeding for you, they’ll churn up your garden as well. It’s really cool. And you have mice… But we have cats!

[interjection from audience member with friend who keeps chickens - they don't have a mouse problem, because the chooks get them]

Suburban gardens have a maximum chicken population though – and you can’t typically keep roosters, they annoy the neighbours.

So we always wanted some more space, and rural areas aren’t limited in this regard.

The final bit of backstory is that I spent years working mostly in offices in Melbourne and occasionally working from home. But since I got my job with SUSE (which has always been a remote working thing), and my wife is doing a PhD (which is mostly a work from home gig too), that meant that we could be anywhere we wanted to, provided we had internet access.

Incredibly it turns out that Tasmania does, in fact, have internet access.

It was where the first NBN rollout was (but only in three places, and none of those three places are where I am).

It has rural areas. It has loads of other lovely things.

There’s nice scenery, there’s good food, there’s art, there’s people. MONA museum is possibly the reason that Hobart ended up on the Lonely Planet list of places to visit in 2013, and that’s really awesome if you can get down there and check that out.

One thing I will say about moving to Tasmania – don’t get the ferry there in winter, it’s a really bad idea. We went over, and coming into Devonport in the morning there were ten metre swells and we woke up in bed… airborne. Which is a little bit alarming.

But now a couple of years later we have two pigs…

…there are several sheep (that one’s called “Bob the Rasta Sheep”)…

…something like 40 chickens…

…two cats…

…that’s the tough farm cat now…

…and we’re teleworking.

Pia Waugh tweeted about this the other week – saying that she thinks teleworking has become really overhyped and that it’s actually not hard to do, and the main problem is cultural change within an organisation – which i’m inclined to agree with.

We at OSDC presumably are already aware that teleworking works just fine, because most or all of the software that we use – or we work with – has been built by distributed teams anyway, so we’re good with that. And continuous supervision and in-person contact is not actually necessary to ensure productivity. If you are in management and you think that it is, then I’m really sorry, but you’re wrong. If you treat your staff like adults, they’ll behave like adults.

There’s a book by Abraham Maslow – which is the Hierarchy of Needs guy –  from 1965, called Eupsychian Management, wherein he talks about this sort of thing – about treating people with respect and getting the best out of them, and there’s psychology and other stuff in there. He was ridiculed at the time, because in 1965 that wasn’t how management worked, but if you go and do a management training course with the Australian Institute of Management they will tell you “no, really, treat people like people and you’ll get good work out of them”.

And I think it can actually be more productive to work at home. I have had days like this working in an office, where you get in and there’s some people in the kitchen, so you have a coffee. For an hour and a half. And then you get to your desk, and you check your email, and then it’s lunchtime. And then you get back to your desk, and you finish checking your email. And you have another coffee, and then finally you manage to do an hour’s work before going home.

That’s… Well, the coffee’s probably good, or the coffee’s hopefully good if you’re spending that much time with it, but it’s not necessarily the most productive.

At home there might be different distractions. You might have to do the washing. If you have children – which I don’t – but that might be a distraction too. People who do have children have told me that it is a distraction, but I’ve tended to find that I can organize things so that I get the long unbroken slabs of time that you need to do good software development.

You probably have an office with a door that closes too, if you do have children, although that probably won’t stop them.

It turns out also that the government likes telework. The cumbersomely named Department of Broadband, Communications and the Digital Economy declared November 12-16 to be National Telework Week. There’s some details of this on their web site; they ran a whole lot of events, they encouraged people to work from home a couple of days that week.

I didn’t pay any attention at the time because I was on leave – I was at home but I wasn’t teleworking.

The web site reads a bit like a big advertisement for the National Broadband Network, because, well, it’s the same guys, and let me say: “Yay NBN!  Bring it on!  Especially if I don’t get stuck on satellite”. But they do seem to be a little bit fixated on video conferencing as their killer app, and Pia had something to say about that too, that teleworking is not actually videoconferencing, and that collaboration tools are far more important.

And I thought about this and I realised that I actually haven’t done any video conferencing at all in my three or four years of doing this.

I have used – and use every day – IRC. Sometimes I’ve used Skype, although I try not to do that now because it’s creepy. Instant messenger programs, telephones (obviously), we have regular conference calls from time to time. There’s some interesting screen sharing and whiteboarding tools like Elluminate, where someone can demo something to somebody else, and you can all sort of talk to each other and chat at the same time. Things like etherpads. We also have company all-hands things over this internal tool which lets somebody show slides and everybody can hear the audio and ask questions via a chat. So those sorts of things…

I also think that as Dave Hall pointed out, video conferencing is not necessarily always ideal.

I tend to think, actually, that video conferencing is much more of a social thing rather than a productivity tool. One of my colleagues outside of SUSE has a company where everybody works remote and what they do is pick a time and set up a Google+ hangout, and then they have a coffee break together. So they’re actually using this as a social thing. I’ve used Google+ hangouts, but only really for remotely attending the Tasmanian Linux User Group meetings when I couldn’t actually get there in person.

So if video conferencing isn’t really necessary, what things are?

I’m assuming that everybody here is probably familiar with most or all of these things because you probably already use them, and that goes back to Pia’s tweet about this technology already being here.

Email, some sort of group text chat, the ability to speak somehow, to collaboratively edit documents.

Trello – if you haven’t looked at it, trello.com is really kind of cool for task management. It’s free. Joel Spolsky’s company made it, they’re trying to get a million users and then figure out how to make money off it later. It’s a list of lists – sort of like Kanban boards if you’ve ever done any of that stuff – you can have little tasks that you assign people to, and they can have checklists, and you can move them from the “doing” column to the “done” column to the “oh god oh god” column.

Bug tracking, revision control, obviously. And I think it’s also useful to have a means of venting when things aren’t working properly, and apparently that’s what Twitter is for.

That’s some of mine.

I don’t use Tracker! I don’t care! I don’t need desktop search! I know where I put everything already. I mean, y’know, that’s what “git grep” is for.

Um. Not that I’m bitter.

But speaking of revision control. Working remotely – I’m assuming loads of people are already using git or mercurial or whatever – you really do need a distributed version control system, because your net connection might be down. You might be travelling. It’s just easier to be able to work offline. And even if you’re not remote, again, to quote Joel Spolsky, ”if you are using Subversion, stop it. Just stop. Subversion = Leeches. Mercurial and Git = Antibiotics. We have better technology now.”

Speaking of broken things – being stuck on satellite is probably really bad. I use a VPN for various things – to access systems – and unless they’re building satellites a lot closer to Earth now, the latency is still going to be a real problem for that sort of traffic.

Power outages tend to be more common in rural areas, so having a decent UPS is really important. According to my logs I’ve had about six outages in the last year for long enough that I actually wanted to shut everything down. It might have only been half an hour or an hour or two, but it’s, y’know… And plenty more where there’ve been little brownouts or drops or whatever.

It’s just a good idea. My UPS is a 1000VA APC thing that I got years ago actually, and it’s got two desktop systems, one LCD, one old server, two Mac Minis, an ancient Pentium (which is a firewall), two ethernet switches, a DSL modem, a wireless access point and my laptop hanging off it (which is a bit silly because my laptop has a battery). It purports to be about 57% loaded and it will run all of that gear for about nine minutes, which is completely fine for any little odd glitches and it’s certainly long enough to cleanly shut everything down if the power goes.

The other interesting thing, and why that last point is on the slide, is we’re on tank water. Our tank water feeds into our house via a mains powered electric pump – we don’t have running water if the power goes out. So it is useful to keep beer handy so you’ve got something to drink in case there’s a power outage.

Side point – the toilets and the garden hose are from a gravity-fed dam, so they’re OK, and in theory if there’s a fire we can still use the garden hose. Unless the fire gets the poly pipe that comes from the dam, so… Anyway.

My list of gear that I’ve got is kind of large. I’ve had lots of this stuff for ages. I run my own mail server and do some other things which are independent of the work I do anyway, so minimally I think to work remotely all the time I would really want to have a laptop – because “laptop!” – I have something I can take with me when I travel… For email, whatever. A desktop system as well that has lots more CPUs and lots more RAM and lots more disks, so that I can spin up virtual machines and do builds and everything. External hard disk for backups, for mirroring software. I’ve got about 36GB of SUSE Linux mirrored locally because this makes it a hell of a lot quicker if I’m doing lots of test VM deployments and testing bare-metal deployment. (Is it bare metal if you’re doing it to a VM?  No. Virtual-metal?)  So, it took a while to get that 36GB of mirror in the first place. I am actually on ADSL – it’s about full speed ADSL1 – but there’s a nightly rsync run that keeps that up to date now. And the other thing, obviously, is the UPS.

Interacting with teams I’ve found tends to vary based on the team and the work. The teams that I work with are all used to working in a distributed fashion though – we have offices in a few places around the world where there’s large concentrations of engineers, but we also have a bunch of people who work remotely, and so we’re used to using all of these tools and having con-calls and whatnot.

My work on the HA team tends to be a bit more solo. It’s mature software, the responsibilities tend to be split kind of along package lines, and so we might spend some time with feature tracking tools, and calls and email to sort out what we’re all going to do next, but then we can all kind of go away and hack on stuff for as long as it takes, then all come back together and do integration testing or whatever.

Work on SUSE Cloud was a much more agile kind of thing, there were two week sprints, it was a new product, teams with people from a few places, we had a couple of status calls once or twice a week for half an hour, we kept very active with each other on IRC, and that was actually a lot of fun. A very sort of fluid, interactive experience.

Time zones haven’t tended to be too much of a problem because I’m within about ten or twelve hours of everybody, so if I work from the afternoon into the evening that overlaps with the European morning, so there’s scope for having calls at that time. It’s a real pain if you have to wait 24 hours for an email where you just say “where’s the source for X?”

Another thing that we do is we have these little weekly reports that get sent round to a slightly broader team. It’s not particularly formal, it’s just a list of what you’ve been doing to kind of keep everybody up to speed with what’s going on. We split this into red, amber and green sections. Red is “oh god oh god we’re all gonna die”, amber is “oh god oh god we’re all gonna die soon unless somebody fixes this thing” and green is just “I fixed this bug, I was working on this thing, I’m waiting for this, what have you”, and hopefully all of your status reports are green all the time. That’s good.

And quite apart from keeping other people informed of what you’re doing, I think it would be useful to do this sort of thing personally, actually, from week to week anyway, because it can help to keep you on track. If you realise that you’ve sort of written “still working on this”, “still working on this”, “still working on this”, you know, once a week, you might want to take a step back and go “Why am I still working on this? What’s not going right here?” So that’s good.

But my week is Monday to Friday. I don’t work nine to five, but I do a forty hour week, or that’s what I’m meant to do. Sometimes I do more, possibly too often I do more, but that’s the idea. Sometimes I do less.

My optimal day would be something like this; get up, have a shower, meditation – I’ve found if you can regularly sit down for fifteen minutes and completely empty your mind (which is harder than it sounds) it can help with sanity and productivity and focus. Then we feed the chickens, and pigs. Breakfast, coffee. These things my wife and I mix up between us. Anything else that needs doing – if you need to go to the shop or whatever – and then hopefully you can sit down and do just a nice solid eight hours of work, with a break for lunch in the late afternoon, probably, and more animal feeding. Then later you stop, slack off, do whatever it is you want to do with the rest of your life, go to bed.

I don’t always have optimal days. But that’s what I strive for.

Some days I’ve had days where I couldn’t focus and I just had to stop and give up and go away, and other days I have done monstrous twelve or fourteen hour hacking sessions where I somehow got three days worth of work done all at once.

But it’s OK. If you keep track of time it’ll all balance out, right?

It helps to have separate space to help with keeping your focus when you’re at work, and to help you not thinking about work things when you don’t want to be. My father had a colleague who worked from home years ago, and he would get up in the morning and put on a suit and tie and then he’d go into his home office and he would sit there and he’d do his work, and when he finished he’d take his suit and tie off and that’s how he knew that he was home again. Even though he never left the building. And this worked for him.

I’ve managed to create… Two very badly resized photos

I’ve got two offices – one on either side of the hall – and one of them’s my “work home office” and one of them’s my “home home office”, and on the weekends the curtains are shut and the lights are off in the work office, which means if I want to do something on the computer I’m not accidentally checking my work email. It would actually require volition to go into that other room and turn things on, so that’s good.

But I do spend lots of time in front of the computer, and I think it’s important to have balance there too, because too much time in front of the computer is no good for your heath. So what better thing to do than stop for a sunset or something, or go hike up the back, or actually sit down with your animals for half an hour and talk to them. Sit down at their level and you get a really different sense of a creature when you experience the world from the height that it looks at it from. It’s interesting.

Dig in the garden, or learn new skills. This is a quote which I’m fond of from a Robert Heinlein book.

And I like this quote because it gives you something to aim for. I’m assuming most people here actually have several of these things. I imagine that computer programming is going to be fairly high up there on the list. I’ve got about twelve, and some of those I wouldn’t have if we weren’t doing this farming thing. Design a building for instance – we’ve built chicken houses and pig houses that are quite sturdy and don’t fall down. I suspect that our pig house will outlast me. Pitch manure is fairly obvious. I’ve also worked with several types of hay, and this has led me to the conclusion that whomever coined the term “go for a roll in the hay” was completely insane.

I do not yet know how to butcher a hog, but ask me again in a week or two. I do know how to do chickens.

It occurred to me that farming is kind of like software hacking, if you tilt your head sideways and squint at it just right, or more to the point, some of the skills translate.

If you’ve ever worked on any long term project that’s been dependent on somebody else to deliver something at a certain time, well there’s putting crops in at the right time. Figuring out how much water you’ve got, or need, or problem solving abilities. Why are my corn cobs really tiny?  Because it was too windy where they were planted and we won’t put them there next time.

And the tolerances aren’t as fine. If you’re developing software – if you’re developing quality software, you want things to be exact, right? You want them to be absolutely right. Because you can have boolean true and false in software development and math.

In real life, things don’t actually need to be perfectly square in order for them to function properly, for a long time.

It is important to have more than one pair of boots… I had to buy new boots for this conference because my previous dress boots (which are also steelcap workboots) are covered in mud and full of water at the moment, so, more boots.

It’s really a bit more like hardware hacking though, because you’re actually building physically manifest objects and interacting with the real world – housing, fences, what have you.

Sometimes these projects involve electricity. Sometimes lots of electricity – like electric fences which run at somewhere between 3,000 and 6,000 volts. Not a lot of current, but you’ll know about it. I mentioned I’ve had more power outages since going rural. Perhaps this balances out – I’ve also been electrocuted more times, by the fence.

And it’s also a good excuse to buy more power tools, because power tools are great, and I’m sure that you can integrate Arduinos into this somewhere. We have an unfinished project to make a little Arduino-based light and temperature sensor which would record the light level off an LDR over time and then if we could put these around the garden in different places, and then look at that, you could figure out which spots are warmer or get more light, and figure out optimal planting perhaps.

I would suggest not doing this solo. People have asked me from time to time what’s it like working from home all the time and not seeing many people very often, and I said that if I lived by myself I would probably be crazy by now (or crazier).

If my wife wasn’t at home looking after animals, I wouldn’t have been able to be here for this conference. The sheep would be OK, because they only need grass (provided there’s lots of it). The pigs and chickens would run out of food after a day or two. We can put enough food out for the chickens in their house for two or three days if we wanted to go away for a weekend – unless we wanted to go away for a weekend to Melbourne on the week that the volcano in Chile erupted and then we got trapped in Melbourne and had to get our friend to go over to our house to help with that. [audience: just hypothetically?] Just hypothetically speaking.

The work’s easier and more fun if there’s more than one of you, and food or produce that you grow yourself – like most of the rest of life – it’s always better when you’ve got someone to share it with.

I don’t think that there’s any reason that it has to be a couple doing this. I think a rural geek sharehouse / hackerspace kind of thing would be totally awesome. If a whole lot of tech people wanted to get together and do something that’d be a lot of fun.

Human contact… Weirdly, since working remote, I’ve probably met more different people now that I work further away from everybody, because I’ve somehow become much more involved in local user groups, or coming to conferences or these sorts of things, and there’s other interesting community stuff as well. We spent one evening a couple of months ago hanging out with our local volunteer fire brigade while they did a controlled burn of somebody’s property, which was interesting, educational and completely terrifying.

Because I have never seen a bushfire up close, and this was a controlled burn… They’ve got this area here with two trucks with water, they’ve got fire sticks that go along and you get a straight line with one of these – it’s burning liquid of some description – if you get a straight line with this, the fire’s going to go one way, or it’s going to go the other – it’s not going to go crazy, it’ll go that way and eat up everything behind it and it dies out surprisingly quickly. But it goes up that high [gestures wildly at ceiling], like… um… Ever seen one of those blasted-out areas in World of Warcraft? Where it’s like, flaming and black, right? It’s like a real one of them! And yeah, we put out the edges and everything, we were out there for a couple of hours, but yeah, that was an experience.

So, I’ll summarize over a few slides. The benefits if you’re thinking of going rural – the benefits of doing something like this; clean air, scenery, the best free range produce in the world. And you know that there’s no pesticides, because you didn’t use them. And you know that it’s free range and the animals had a good life, because you let them out to roam. You learn interesting and useful skills. You get bonus time through lack of commuting.

My parents came and stayed and we showed them our animal feeding regimen. If you do the whole thing – feed three lots of chickens and replace all their water, and you feed the pigs, and you check all of the electric fence because the grass is growing up to it and shorting it out so it doesn’t work as well – that takes about an hour, and it was pointed out that that’s a fair amount of time. But I could be spending an hour commuting, and I’d much rather be spending an hour with my animals. So you can kind of try to engineer your life so that it has the shape that you want.

Things to consider though before doing something like that – if you’ve never worked remotely before, do you actually like it? I think it would be valuable to work remote from home in your regular home for a while to see if it’s going to work for you before you up stumps and go somewhere else. Is your work easily remotable? Large bits of software development might be, but if you’re particularly client facing they might not be and you might suddenly need to do a lot of travel, so how much of that do you want to do?

I think it’s really important to figure out the comms problem before you go, because we’ve been looking around at other bits of land and looking at adsl2exchanges.com.au to find out where things are. They’re 7km away and if you ring up Telstra they’ll go “Oh, yeah we can get, yeah, yeah, there’s ports on that exchange, no worries”. It’s not going to work. At all. You’ll probably be on pair gain or something horrible. You could probably get a free (or heavily discounted) satellite thing on the NBN interim satellite program, but the latency’s going to suck if you’re using a VPN. That will work over 3G though, so it might be interesting if you had a satellite and a 3G dongle to route your web traffic (or whatever the satellite can fake into working quickly, or appearing to work quickly) over the satellite, and routing your VPN or interactive SSH over the 3G thing. And I do think that wireless technology is only going to get better over time, but it’s definitely something you need to figure out. Because if we didn’t have our internet connection, I couldn’t do my job. Well, I could do it for a while, but eventually I’ve done enough with git and I actually have to access the bug tracker, so you need comms at some point.

[audience: what kind of line of sight options do you have?]

Where I am, I can look down towards the Huon Valley, so I’ve got a big open space in front of me, and I can see a cell tower up there too. There’s a company called TasmaNet - they’re actually building a data centre for something completely unrelated, for putting big servers and things in Hobart somewhere – but they do that sort of wireless tech for people in that area, and if they’ve got a tower up and you can get line of sight to it, you can get for a hundred and something dollars a month, a couple of megabit synchronous, with a cap of half a dozen gig maybe. I’m not sure what happened if you hit the cap. They’re billing that as business grade internet for a rural area and they have a lot of people – tradies or whatever – who will remotely access their MYOB system in their home area somehow through this thing as well. So there’s stuff around, but you might have to go and look for it. I did see a case study on the NBN web site from somebody who used to work for Google, who then moved to somewhere out in the sticks, who’d just got the NBN wireless up and the receiver was quite a distance away behind some trees that might need pruning in the future, but right now he’s getting 14 megabit and he’s very happy about it. Obviously that’s a good story, because it would be, being a case study on the NBN web site, but this tells me there’s hope.

The last point there… I used to work with a guy, somebody asked him how he felt about gardening, or what he’d like in a garden, and his answer was “does it prune itself?” This is probably not the life for him.

Solutions to problems.

If there aren’t enough people in your life, get involved with some. There’s loads of them around. Hi guys! [waves at audience]

Having good kit, a UPS, local mirrors of software is important. Good collaboration tools that everybody’s happy using, and knows how to use. Some sort of backup internet connection – I’ve got a little wireless thingy in case the DSL goes down, or I want to go somewhere else. A 3G or 4G thing.

It’s nice to know where you can find an internet cafe nearby if you want to go and do that for a while, or if something breaks.

Other things you can do – consider making extra personal space like I managed to do with my offices. There are coworking spaces around where you can go and be in another place with other likeminded people, even if you’re not working on the same stuff. Get out and about – make sure you go out and have a meal or something once in a while or you get “I’ve been trapped in the house for three weeks”.

And I’ve found that it’s kind of important to find somebody who doesn’t telework to remind you when the public holidays are, because I would never know. And I would never take those days off if somebody didn’t tell me.

But, even if you never work remote I would encourage everybody to go and get chickens.

Because, if you’ve got a garden (it’s no good if you’re in a flat – it’s not free range) they’re great fun, they’re lovely, they’ll get you out of the house, they’ll make wonderful eggs for you, they’ll dig in your garden. It’s like I said, we’ve kept chickens for ten years, it’s cool, it’s good.

And that’s about it actually, but I’ll finish on a slightly less terrifying slide.

These are our newest additions. There’s sixteen little chicks there. These are from fertile eggs we got sent over from Western Australia, so that next year we can breed this lot with our lot, so that we increase genetic diversity within the area. Because Barnevelder chickens – which is what we’ve got – in Tassie, they all came from one guy or… Y’know… [interjection from audience - one chicken?].

I’ll finish on this one actually, ’cause they’re going into their house.

Thank you!

Questions

There were several questions from the audience, which I’ll paraphrase here, again for the sake of clarity.

How do you market yourself?  Do you have to go see lots of people?

I’m not exactly certain what is meant by “market yourself”, but I am in a position where most of my work doesn’t actually involve being physically present with other people most of the time. If you are working remote, consider how much time you might need to spend travelling if you do need to spend a lot of time face to face. If I had to travel every week to go and meet people, I probably wouldn’t want to be doing this because it would be too much strain for me personally.

What kind of online learning resources have you found, and are you blogging about this?

I haven’t done any blogging about it which is probably rather remiss of me. I guess this transcript is a start. I didn’t have any links handy when the question was asked, but the three I thought of later are:

• ChookNet – forums, classifieds, whatnot.
• Appropedia - sustainability, appropriate technology, poverty reduction (I should really try to contribute to this myself in my Copious Free Time).
• The Encyclopedia of Country Living - actually a book, so not online, but this book is how I learned to butcher chickens. It’s awesome. Everyone should get a copy. There’s also the author’s web site, although sadly she passed away in 2005.

Are you worried someone in, say, the Ukraine might be willing to do the same remote job, but for less money?

My initial facetious answer was that I’m not personally worried about that, because I live in a happy land (although someone later pointed out that this is actually not a bad philosophy).

Comments from the audience included the point that everyone has this problem (not just me working remote), and that we’re software developers, and we develop a reputation for doing good work, so will tend to become desirable.

My more considered answer, using high availability as an example, is that there’s a relatively high barrier to entry to doing the work I do, because there is a lot to learn. Someone pointed out a couple of years ago that it’s difficult to find people who know the Linux HA software stack, because everyone with those skills is already employed by somebody.

On learning new skills – you didn’t mention making cheese!

Cheese is on my personal list – I should have mentioned that – but I haven’t done it yet. Someone pointed out that Arjen Lentz makes cheese, so he’d be worth talking to about that.

How does working for an overseas company work (tax, workcover, etc.)?

I work for SUSE, which is headquartered in Germany, but I’m legally an employee of Novell Australia, at least for salary and taxation purposes, so that’s easy in my case. It’s effectively no different than working a salaried job for any other Australian company.

If I were being paid directly by an overseas entity with no Australian presence, the least worst thing to do would probably be to have my own company set up, which then invoiced the overseas entity, and in turn paid me a salary and took care of tax and whatnot.

Addenda

As this is a transcript, I naturally have the liberty of expanding on a few things:

• When I mentioned my optimal day, I said “later you stop, slack off, do whatever it is you want to do with the rest of your life”, and something about “keeping your focus when you’re at work”. This should not be misconstrued to mean I want my life split into work plus everything else – that would contradict my earlier statement regarding my loathing of the term “work/life balance”. Rather this should be interpreted to mean that your brain needs downtime after hacking – like in The Matrix, when Choi said to Neo ”Hey, it just sounds to me like you need to unplug, man. You know, get some R&R?”.  I tend to find that I sleep better if I have at least an hour or two away from the computer before bed, for example.
• Regarding using Google+ hangouts for coffee breaks, I have it on good authority that my aforementioned colleague actually uses G+ hangouts for rather more than just coffee, given the various collaboration tools Google provides. But really I think this just supports my argument that collaboration tools are key, as opposed to the notion that video conferencing is what makes telework work.
• On TasmaNet, their pricing and web site seems to have changed a fair bit since the figures I mentioned in my talk entered my brain. AFAICT the price seems to be a bit higher than I remember.
• The guy I mentioned who has NBN wireless is Scott Weston, whose story can be found on the NBN blog. Apparently he’s actually getting 12Mbps, not 14Mbps, but that still sounds pretty good to me, although upload is only 0.5Mbps, which is a bit rough.
• In case it’s not abundantly clear, everything here is my personal opinion and/or experience, and should not be taken as any form of statement on behalf of SUSE.

This miniconf is about building reliable infrastructure, from two-node HA failover pairs to multi-thousand-core cloud systems. You might like to think of it as a sequel to the LCA 2012 High Availability and Distributed Storage miniconf (videos here).

Do any of the following describe you?

• You’re building cloud infrastructure for others to use (openstack, cloudstack, eucalyptus, …)
• Your data needs to be reliably available everywhere (ceph, glusterfs, drbd, …)
• Your system absolutely must be up all the time (pacemaker, corosync, …)

If so, this is the miniconf for you! Please consider submitting a presentation at http://tinyurl.com/cidsha-lca2013

We’re expecting most talk slots to be 25 minutes (including questions and changeover), but there will be openings for shorter lightning talks and maybe a couple of longer talks. CFP closes on Sunday November 4, 2012. Notifications of acceptance will be emailed out after this date.

Note that there is also an OpenStack-specific miniconf running on Tuesday 29 January. We’re hoping this will give us a pretty awesome two-day LCA 2013 CloudFest. As a rough rule of thumb, more generic or infrastructure-related talks should go to Cloud, Distributed Storage & HA, while deeper OpenStack-specific talks should probably go to the OpenStack miniconf. If in doubt, or if you have any other questions, please contact me directly.

Thanks!