How to Really Clean a Roomba

The official iRobot Roomba instructional videos show a Roomba doing its thing in an immaculately clean house. When it comes time to clean the Roomba itself, an immaculately manicured woman empties a sprinkling of dirt from the Roomba’s hopper into a bin, flicks no dust at all off the rotor brush and then delicately grooms the main brush, before putting the Roomba back on to charge.

It turns out the cleaning procedure is a bit more involved for two long-haired adults and three cats living on a farm. Note that the terminology used in the instructions below was made up by me just now, and may or may not match what’s in the Roomba manual. Also, our Roomba is named Neville.

Continue reading

My Personal Travel Ban

I plan to avoid any and all travel to the USA for the foreseeable future due to the complete mess unfolding there with Trump’s executive orders banning immigration from some Muslim-majority countries, related protests, illegal detainment, etc. etc. (the list goes on, and I expect it to get longer).

It’s not that I’m from one of the blacklist countries, and I’m not a Muslim. I’m even white. But I no longer consider travel to the USA safe (especially bearing in mind my ridiculous beard and long hair), and even if I did, I’d want to stand in solidarity with the people who are currently being screwed. The notion of banning entire groups of people based on a single shared trait (in this case, probable adherence to a particular religion) is abhorrent; it demonizes our fellow humans, divides us and builds walls – whether metaphorical or physical – between our various communities. The fact that this immigration ban will impact refugees and asylum seekers just makes matters worse. I am deeply ashamed by Australia’s record on that front too, and concerned that our government will not do much better.

So I won’t be putting in any talks for Cephalocon - which is a damn shame, as I’m working on Ceph – or for any other US-based tech conference unless and until the situation over there changes.

I realise this post may not make much difference in the grander scheme of things, but one more voice is one more voice.

Random Test Subject

Tim Serong Almost every time I fly, it seems like I get pulled aside for the random explosives trace detection test. I always assumed it was because I usually look like a crazy mountain man (see photo). But, if you google around for “airport random explosives test”, you’ll find forum posts from security staff assuring everyone they’re not doing profiling, and even a helpful FAQ from Newcastle Airport (“Why are you always chosen for ‘explosive testing’?“) which says the process is “as the officer finishes screening one person, they are required to ask the next person walking through screening to undertake the ETD test”.

So maybe it’s just bad luck. Except possibly for that time at Hobart airport last week, where I was seeing off a colleague after 2017. As far as I could tell, we were the only two people approaching security, and my colleague was in front. He was waved through to the regular security screening, and I was asked over for an explosives test, to which I replied “you’re most welcome to test me if you like, but I’m not actually going through security into departures”. The poor guy looked a bit nonplussed at this, then moved on to the next traveler who’d since appeared in line behind us.

What to do about this in future? Obviously, I need a new t-shirt, with text something like one of these:

Random Test SubjectPick Me for Random Testing Randomly Chosen Every Time

If anyone else would like a t-shirt along these lines, the images above conveniently link to my Redbubble store. Or, if you’d rather DIY, there’s PNGs here, here and here (CC-BY-SA as usual, and no, they’re not broken, it’s white text on a transparent background).

Finally, for some real randomness, check out Keith Packard’s ChaosKey To Production presentation. I’m not actually affiliated with Keith, but the ChaosKey sure looks nifty.

Hello Salty Goodness

Anyone who’s ever deployed Ceph presumably knows about ceph-deploy. It’s right there in the Deployment chapter of the upstream docs, and it’s pretty easy to use to get a toy test cluster up and running. For any decent sized cluster though, ceph-deploy rapidly becomes cumbersome… As just one example, do you really want to have to `ceph-deploy osd prepare` every disk? For larger production clusters it’s almost certainly better to use a fully-fledged configuration management tool, such as Salt, which is what this post is about.

Continue reading

Thunderbird Uses OpenGL – Who Knew?

I have a laptop and a desktop system (as well as a bunch of other crap, but let’s ignore that for a moment). Both laptop and desktop are running openSUSE Tumbleweed. I’m usually in front of my desktop, with dual screens, a nice keyboard and trackball, and the laptop is sitting with the lid closed tucked away under the desk. Importantly, the laptop is where my mail client lives. When I’m at my desk, I ssh from desktop to laptop with X forwarding turned on, then fire up Thunderbird, and it appears on my desktop screen. When I go travelling, I take the laptop with me, and I’ve still got my same email client, same settings, same local folders. Easy. Those of you considering heckling me for not using $any_other_mail_client and/or $any_other_environment, please save it for later.

Yesterday I had an odd problem. A new desktop system arrived, so I installed Tumbleweed, eventually ssh’d to my Laptop, started Thunderbird, and…

# thunderbird

…nothing happened. There’s usually a little bit of junk on the console at that point, and the Thunderbird window should have appeared on my desktop screen. But it didn’t. strace showed it stuck in a loop, waiting for something:

wait4(22167, 0x7ffdfc669be4, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGVTALRM {si_signo=SIGVTALRM, si_code=SI_TKILL, si_pid=22164, si_uid=1000} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(22167, 0x7ffdfc669be4, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGVTALRM {si_signo=SIGVTALRM, si_code=SI_TKILL, si_pid=22164, si_uid=1000} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(22167, 0x7ffdfc669be4, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGVTALRM {si_signo=SIGVTALRM, si_code=SI_TKILL, si_pid=22164, si_uid=1000} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)

After an assortment of random dead ends (ancient and useless bug reports about Thunderbird and Firefox failing to run over remote X sessions), I figured I may as well attach a debugger to see if I could get any more information:

# gdb -p 22167
GNU gdb (GDB; openSUSE Tumbleweed) 7.11
Attaching to process 22167
Reading symbols from /usr/lib64/thunderbird/thunderbird-bin...
0x00007f2e95331a1d in poll () from /lib64/
(gdb) break
Breakpoint 1 at 0x7f2e95331a1d
(gdb) bt
#0 0x00007f2e95331a1d in poll () from /lib64/
#1 0x00007f2e8730b410 in ?? () from /usr/lib64/
#2 0x00007f2e8730cecf in ?? () from /usr/lib64/
#3 0x00007f2e8730cfe2 in xcb_wait_for_reply () from /usr/lib64/
#4 0x00007f2e86ecc845 in ?? () from /usr/lib64/
#5 0x00007f2e86ec74b8 in ?? () from /usr/lib64/
#6 0x00007f2e86e9a2a9 in ?? () from /usr/lib64/
#7 0x00007f2e86e9654b in ?? () from /usr/lib64/
#8 0x00007f2e86e966b3 in glXChooseVisual () from /usr/lib64/
#9 0x00007f2e90fa0d6f in glxtest () at /usr/src/debug/thunderbird/mozilla/toolkit/xre/glxtest.cpp:230
#10 0x00007f2e90fa1003 in fire_glxtest_process () at /usr/src/debug/thunderbird/mozilla/toolkit/xre/glxtest.cpp:333
#11 0x00007f2e90f9b4cd in XREMain::XRE_mainInit (this=this@entry=0x7ffdfc66c448, aExitFlag=aExitFlag@entry=0x7ffdfc66c3ef) at /usr/src/debug/thunderbird/mozilla/toolkit/xre/nsAppRunner.cpp:3134
#12 0x00007f2e90f9ee27 in XREMain::XRE_main (this=this@entry=0x7ffdfc66c448, argc=argc@entry=1, argv=argv@entry=0x7ffdfc66d958, aAppData=aAppData@entry=0x7ffdfc66c648)
at /usr/src/debug/thunderbird/mozilla/toolkit/xre/nsAppRunner.cpp:4362
#13 0x00007f2e90f9f0f2 in XRE_main (argc=1, argv=0x7ffdfc66d958, aAppData=0x7ffdfc66c648, aFlags=) at /usr/src/debug/thunderbird/mozilla/toolkit/xre/nsAppRunner.cpp:4484
#14 0x00000000004054c8 in do_main (argc=argc@entry=1, argv=argv@entry=0x7ffdfc66d958, xreDirectory=0x7f2e9504a9c0) at /usr/src/debug/thunderbird/mail/app/nsMailApp.cpp:195
#15 0x0000000000404c4a in main (argc=1, argv=0x7ffdfc66d958) at /usr/src/debug/thunderbird/mail/app/nsMailApp.cpp:332
(gdb) continue
[Inferior 1 (process 22167) exited with code 01]

OK, so it’s libGL that’s waiting for something. Why is my mail client trying to do stuff with OpenGL?

Hang on! When I told gdb to continue, suddenly Thunderbird appeared, running properly, on my desktop display. WTF?

As far as I can tell, the problem is that my new desktop system has an NVIDIA GPU (nouveau drivers, BTW), and my laptop and previous desktop system both have Intel GPUs. Something about ssh’ing from the desktop with the NVIDIA GPU to the laptop with the Intel GPU, causes Thunderbird (and, indeed, any GL app — I also tried glxinfo and glxgears) to just wedge up completely. Whereas if I do the reverse (ssh from Intel GPU laptop to NVIDIA GPU desktop) and run GL apps, it works fine.

After some more Googling, I discovered I can make Thunderbird work properly over remote X like this:


That will apparently cause glXCreateContext to return BadValue, which is enough to kick Thunderbird along. LIBGL_ALWAYS_SOFTWARE=1 works equally well to enable Thunderbird to function, while presumably still allowing it to use OpenGL if it really needs to for something (proof: LIBGL_ALWAYS_INDIRECT=1 glxgears fails, LIBGL_ALWAY_SOFTWARE=1 glxgears gives me spinning gears).

I checked Firefox too, and it of course has the same remote X problem, and the same solution.

It’s OK to be Wrong in Public

I’ve spent a reasonably long time with computers. I’ve been doing something with either software or hardware (mostly software) for pretty close to three quarters of my current lifespan. I started when I was about 10, but (perhaps unsurprisingly) nobody was paying me for my work yet then. Flash forwards a few decades, and I have a gig I rather enjoy with SUSE, working on storage stuff.

OK, “yay Tim”. Enough of the backstory, what’s the point?

The point (if I can ball up my years of experience, and the experience of the world at large), is that, in aggregate, we write better software if we do it in the open. There’s a whole Free Software vs. Open Source thing, and the nuances of that discussion are interesting and definitely important, but to my mind this is all rather less interesting than the mechanics of how F/OSS projects actually work in practice. In particular, given that projects are essentially communities, and communities are made up of individuals, how does an individual join an existing project, and become accepted and confident in that space?

Continue reading

The Politics of Resentment

I’ve been reading The Archdruid Report regularly for a long time now, because unlike me, John Michael Greer posts every week and always writes something interesting. Given that we’ve got a federal election coming up in Australia and that I’ve mentioned one of JMG’s notions on the current state of politics to several people over the last few months, I though I’d provide a TL;DR here:

If you want, you can split people in the US into four classes, based on how they get most of their income:

  1. The investment class (income derived from returns on investment)
  2. The salary class (who receive a monthly salary)
  3. The wage class (who receive an hourly wage)
  4. The welfare class (who receive welfare payments)

According to JMG, over the last fifty years or so, three of these classes of people have remained roughly where they are; the investment class still receives returns on investment (modulo a recession or two), the salary class still draws a reasonable salary, and life still sucks for people on welfare. But the wage class, to be blunt, has been systematically fucked over this time period. There’s a lot of people there, and it’s this disenfranchised group who sees someone outside the political establishment status quo (Trump) as someone they can get behind. Whether or not Trump is elected in the US, there’s still going to be a whole lot of people out there pissed off with the current state of things, and it’s going to be really interesting to see how this plays out.

You should probably go read the full post, because I doubt I’ve done it justice here, but I don’t think it’s unreasonable to imagine the same (or a similar) thesis might be valid for Australia, so my question is: what, if anything, does this mean for our 2016 federal election?

Oops, That Didn’t Work

As an experiment, we’ve written a play. It might be viable to perform in an actual theatre, provided there’s a large screen with a projector that a few between-scene montage things can be played on. Failing that, we’ll just have to run with a short film version.

(a Tragedy in Three Acts, based on a true story)

by Tim Serong and Morgan Leigh


Cast of Characters

TIM, cranky software developer
MORGAN, lover of robots and tech promised by golden age science fiction
SUPPORT, various LIFX support people, QA people and possibly Android developers*

* (These are several different people in real life, but it should be easier to perform as one character)
Continue reading

Some Photos from Nuremberg

The Basslink cable was cut this morning in order to repair it. Concerns that this could cause Tasmanian internet speeds to go to shit appear to have been well founded (see #basslink on Twitter). In my case, my NBN wireless has gone from 21Mbps/5Mbps to 1Mbps/4Mbps, i.e. my download speed is a twentieth of what it was before, but my upload is about the same. So I may as well post some photos, right?

Continue reading

A Gentle Introduction to Ceph

A Gentle Introduction to CephI told a little story about Ceph today in the sysadmin miniconf at 2016. Simon and Ewen have already put a copy of my slides up on the miniconf web site, but for bonus posterity you can get them from here also. For some more detail, check out Lars’ SUSE Enterprise Storage – Design and Performance for Ceph slides from SUSECon 2015 and Software-defined Storage: Sizing and Performance.

Update (2016-02-07): The video of the talk can be found here.