NBN Fixed Wireless – Four Years On

It’s getting close to the fourth anniversary of our NBN fixed wireless connection. Over that time, speaking as someone who works from home, it’s been generally quite good. 22-24 Mbps down and 4-4.5 Mbps up is very nice. That said, there have been a few problems along the way, and more recently evenings have become significantly irritating.

There were some initial teething problems, and at least three or four occasions where someone was performing “upgrades” during business hours over the course of several consecutive days. These upgrade periods wouldn’t have affected people who are away at work or school or whatever during the day, as by the time they got home, the connection would have been back up. But for me, I had to either tether my mobile phone to my laptop, or go down to a cafe or friend’s place to get connectivity.

There’s also the icing problem, which occurs a couple of times a year when snow falls below 200-300 metres for a few days. No internet, and also no mobile phone.

These are all relatively isolated incidents though. What’s been happening more recently is our connection speed in the evenings has gone to hell. I don’t tend to do streaming video, and my syncing several GB of software mirrors happens automatically in the wee hours while I’m asleep, so my subjective impression for some time has just been that “things were kinda slower during the evenings” (web browsing, pushing/pulling from already cloned git repos, etc.). I vented about this on Twitter in mid-June but didn’t take any further action at the time.

Several weeks later, on the evening of July 28, I needed to update and rebuild a Ceph package for openSUSE and SLES. The specifics aren’t terribly relevant to this post, but the process (which is reasonably automated) involves running something like `git clone git@github.com:SUSE/ceph.git && cd ceph && git submodule update --init --recursive`, which in turn downloads a few GB of data. I’ve done this several times in the past, and it usually takes an hour, or maybe a bit more. So you start it up, then go make a meal, come back and you’re done.

Not so on that Friday evening. It took six hours.

I ran a couple of speed tests:

I looked at my smokeping graphs:

smokeping-2017-07-28

That’s awfully close to 20% packet loss in the evenings. It happens every night:

smokeping-last-10-days

And it’s been happening for a long time:

smokeping-last-400-days

Right now, as I’m writing this, the last three hours show an average of 15.57% packet loss:

smokeping-last-three-hours

So I’ve finally opened a support ticket with iiNet. We’ll see what they say. It seems unlikely that this is a problem with my equipment, as my neighbour on the same wireless tower has also had noticeable speed problems for at least the last couple of months. I’m guessing it’s either not enough backhaul, or the local NBN wireless tower is underprovisioned (or oversubscribed). I’m leaning towards the latter, as in recent times the signal strength indicators on the NTD flick between two amber and three green lights in the evenings, whereas during the day it’s three green lights all the time.

How to Really Clean a Roomba

The official iRobot Roomba instructional videos show a Roomba doing its thing in an immaculately clean house. When it comes time to clean the Roomba itself, an immaculately manicured woman empties a sprinkling of dirt from the Roomba’s hopper into a bin, flicks no dust at all off the rotor brush and then delicately grooms the main brush, before putting the Roomba back on to charge.

It turns out the cleaning procedure is a bit more involved for two long-haired adults and three cats living on a farm. Note that the terminology used in the instructions below was made up by me just now, and may or may not match what’s in the Roomba manual. Also, our Roomba is named Neville.

Continue reading

My Personal Travel Ban

I plan to avoid any and all travel to the USA for the foreseeable future due to the complete mess unfolding there with Trump’s executive orders banning immigration from some Muslim-majority countries, related protests, illegal detainment, etc. etc. (the list goes on, and I expect it to get longer).

It’s not that I’m from one of the blacklist countries, and I’m not a Muslim. I’m even white. But I no longer consider travel to the USA safe (especially bearing in mind my ridiculous beard and long hair), and even if I did, I’d want to stand in solidarity with the people who are currently being screwed. The notion of banning entire groups of people based on a single shared trait (in this case, probable adherence to a particular religion) is abhorrent; it demonizes our fellow humans, divides us and builds walls – whether metaphorical or physical – between our various communities. The fact that this immigration ban will impact refugees and asylum seekers just makes matters worse. I am deeply ashamed by Australia’s record on that front too, and concerned that our government will not do much better.

So I won’t be putting in any talks for Cephalocon – which is a damn shame, as I’m working on Ceph – or for any other US-based tech conference unless and until the situation over there changes.

I realise this post may not make much difference in the grander scheme of things, but one more voice is one more voice.

Random Test Subject

Tim Serong Almost every time I fly, it seems like I get pulled aside for the random explosives trace detection test. I always assumed it was because I usually look like a crazy mountain man (see photo). But, if you google around for “airport random explosives test”, you’ll find forum posts from security staff assuring everyone they’re not doing profiling, and even a helpful FAQ from Newcastle Airport (“Why are you always chosen for ‘explosive testing’?“) which says the process is “as the officer finishes screening one person, they are required to ask the next person walking through screening to undertake the ETD test”.

So maybe it’s just bad luck. Except possibly for that time at Hobart airport last week, where I was seeing off a colleague after linux.conf.au 2017. As far as I could tell, we were the only two people approaching security, and my colleague was in front. He was waved through to the regular security screening, and I was asked over for an explosives test, to which I replied “you’re most welcome to test me if you like, but I’m not actually going through security into departures”. The poor guy looked a bit nonplussed at this, then moved on to the next traveler who’d since appeared in line behind us.

What to do about this in future? Obviously, I need a new t-shirt, with text something like one of these:

Random Test SubjectPick Me for Random Testing Randomly Chosen Every Time

If anyone else would like a t-shirt along these lines, the images above conveniently link to my Redbubble store. Or, if you’d rather DIY, there’s PNGs here, here and here (CC-BY-SA as usual, and no, they’re not broken, it’s white text on a transparent background).

Finally, for some real randomness, check out Keith Packard’s ChaosKey To Production presentation. I’m not actually affiliated with Keith, but the ChaosKey sure looks nifty.

Hello Salty Goodness

Anyone who’s ever deployed Ceph presumably knows about ceph-deploy. It’s right there in the Deployment chapter of the upstream docs, and it’s pretty easy to use to get a toy test cluster up and running. For any decent sized cluster though, ceph-deploy rapidly becomes cumbersome… As just one example, do you really want to have to `ceph-deploy osd prepare` every disk? For larger production clusters it’s almost certainly better to use a fully-fledged configuration management tool, such as Salt, which is what this post is about.

Continue reading

Thunderbird Uses OpenGL – Who Knew?

I have a laptop and a desktop system (as well as a bunch of other crap, but let’s ignore that for a moment). Both laptop and desktop are running openSUSE Tumbleweed. I’m usually in front of my desktop, with dual screens, a nice keyboard and trackball, and the laptop is sitting with the lid closed tucked away under the desk. Importantly, the laptop is where my mail client lives. When I’m at my desk, I ssh from desktop to laptop with X forwarding turned on, then fire up Thunderbird, and it appears on my desktop screen. When I go travelling, I take the laptop with me, and I’ve still got my same email client, same settings, same local folders. Easy. Those of you considering heckling me for not using $any_other_mail_client and/or $any_other_environment, please save it for later.

Yesterday I had an odd problem. A new desktop system arrived, so I installed Tumbleweed, eventually ssh’d to my Laptop, started Thunderbird, and…

# thunderbird

…nothing happened. There’s usually a little bit of junk on the console at that point, and the Thunderbird window should have appeared on my desktop screen. But it didn’t. strace showed it stuck in a loop, waiting for something:

wait4(22167, 0x7ffdfc669be4, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGVTALRM {si_signo=SIGVTALRM, si_code=SI_TKILL, si_pid=22164, si_uid=1000} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(22167, 0x7ffdfc669be4, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGVTALRM {si_signo=SIGVTALRM, si_code=SI_TKILL, si_pid=22164, si_uid=1000} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
wait4(22167, 0x7ffdfc669be4, 0, NULL)   = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGVTALRM {si_signo=SIGVTALRM, si_code=SI_TKILL, si_pid=22164, si_uid=1000} ---
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)

After an assortment of random dead ends (ancient and useless bug reports about Thunderbird and Firefox failing to run over remote X sessions), I figured I may as well attach a debugger to see if I could get any more information:

# gdb -p 22167
GNU gdb (GDB; openSUSE Tumbleweed) 7.11
[...]
Attaching to process 22167
Reading symbols from /usr/lib64/thunderbird/thunderbird-bin...
[...]
0x00007f2e95331a1d in poll () from /lib64/libc.so.6
(gdb) break
Breakpoint 1 at 0x7f2e95331a1d
(gdb) bt
#0 0x00007f2e95331a1d in poll () from /lib64/libc.so.6
#1 0x00007f2e8730b410 in ?? () from /usr/lib64/libxcb.so.1
#2 0x00007f2e8730cecf in ?? () from /usr/lib64/libxcb.so.1
#3 0x00007f2e8730cfe2 in xcb_wait_for_reply () from /usr/lib64/libxcb.so.1
#4 0x00007f2e86ecc845 in ?? () from /usr/lib64/libGL.so.1
#5 0x00007f2e86ec74b8 in ?? () from /usr/lib64/libGL.so.1
#6 0x00007f2e86e9a2a9 in ?? () from /usr/lib64/libGL.so.1
#7 0x00007f2e86e9654b in ?? () from /usr/lib64/libGL.so.1
#8 0x00007f2e86e966b3 in glXChooseVisual () from /usr/lib64/libGL.so.1
#9 0x00007f2e90fa0d6f in glxtest () at /usr/src/debug/thunderbird/mozilla/toolkit/xre/glxtest.cpp:230
#10 0x00007f2e90fa1003 in fire_glxtest_process () at /usr/src/debug/thunderbird/mozilla/toolkit/xre/glxtest.cpp:333
#11 0x00007f2e90f9b4cd in XREMain::XRE_mainInit (this=this@entry=0x7ffdfc66c448, aExitFlag=aExitFlag@entry=0x7ffdfc66c3ef) at /usr/src/debug/thunderbird/mozilla/toolkit/xre/nsAppRunner.cpp:3134
#12 0x00007f2e90f9ee27 in XREMain::XRE_main (this=this@entry=0x7ffdfc66c448, argc=argc@entry=1, argv=argv@entry=0x7ffdfc66d958, aAppData=aAppData@entry=0x7ffdfc66c648)
at /usr/src/debug/thunderbird/mozilla/toolkit/xre/nsAppRunner.cpp:4362
#13 0x00007f2e90f9f0f2 in XRE_main (argc=1, argv=0x7ffdfc66d958, aAppData=0x7ffdfc66c648, aFlags=) at /usr/src/debug/thunderbird/mozilla/toolkit/xre/nsAppRunner.cpp:4484
#14 0x00000000004054c8 in do_main (argc=argc@entry=1, argv=argv@entry=0x7ffdfc66d958, xreDirectory=0x7f2e9504a9c0) at /usr/src/debug/thunderbird/mail/app/nsMailApp.cpp:195
#15 0x0000000000404c4a in main (argc=1, argv=0x7ffdfc66d958) at /usr/src/debug/thunderbird/mail/app/nsMailApp.cpp:332
(gdb) continue
[Inferior 1 (process 22167) exited with code 01]

OK, so it’s libGL that’s waiting for something. Why is my mail client trying to do stuff with OpenGL?

Hang on! When I told gdb to continue, suddenly Thunderbird appeared, running properly, on my desktop display. WTF?

As far as I can tell, the problem is that my new desktop system has an NVIDIA GPU (nouveau drivers, BTW), and my laptop and previous desktop system both have Intel GPUs. Something about ssh’ing from the desktop with the NVIDIA GPU to the laptop with the Intel GPU, causes Thunderbird (and, indeed, any GL app — I also tried glxinfo and glxgears) to just wedge up completely. Whereas if I do the reverse (ssh from Intel GPU laptop to NVIDIA GPU desktop) and run GL apps, it works fine.

After some more Googling, I discovered I can make Thunderbird work properly over remote X like this:

# LIBGL_ALWAYS_INDIRECT=1 thunderbird

That will apparently cause glXCreateContext to return BadValue, which is enough to kick Thunderbird along. LIBGL_ALWAYS_SOFTWARE=1 works equally well to enable Thunderbird to function, while presumably still allowing it to use OpenGL if it really needs to for something (proof: LIBGL_ALWAYS_INDIRECT=1 glxgears fails, LIBGL_ALWAY_SOFTWARE=1 glxgears gives me spinning gears).

I checked Firefox too, and it of course has the same remote X problem, and the same solution.

It’s OK to be Wrong in Public

I’ve spent a reasonably long time with computers. I’ve been doing something with either software or hardware (mostly software) for pretty close to three quarters of my current lifespan. I started when I was about 10, but (perhaps unsurprisingly) nobody was paying me for my work yet then. Flash forwards a few decades, and I have a gig I rather enjoy with SUSE, working on storage stuff.

OK, “yay Tim”. Enough of the backstory, what’s the point?

The point (if I can ball up my years of experience, and the experience of the world at large), is that, in aggregate, we write better software if we do it in the open. There’s a whole Free Software vs. Open Source thing, and the nuances of that discussion are interesting and definitely important, but to my mind this is all rather less interesting than the mechanics of how F/OSS projects actually work in practice. In particular, given that projects are essentially communities, and communities are made up of individuals, how does an individual join an existing project, and become accepted and confident in that space?

Continue reading

The Politics of Resentment

I’ve been reading The Archdruid Report regularly for a long time now, because unlike me, John Michael Greer posts every week and always writes something interesting. Given that we’ve got a federal election coming up in Australia and that I’ve mentioned one of JMG’s notions on the current state of politics to several people over the last few months, I though I’d provide a TL;DR here:

If you want, you can split people in the US into four classes, based on how they get most of their income:

  1. The investment class (income derived from returns on investment)
  2. The salary class (who receive a monthly salary)
  3. The wage class (who receive an hourly wage)
  4. The welfare class (who receive welfare payments)

According to JMG, over the last fifty years or so, three of these classes of people have remained roughly where they are; the investment class still receives returns on investment (modulo a recession or two), the salary class still draws a reasonable salary, and life still sucks for people on welfare. But the wage class, to be blunt, has been systematically fucked over this time period. There’s a lot of people there, and it’s this disenfranchised group who sees someone outside the political establishment status quo (Trump) as someone they can get behind. Whether or not Trump is elected in the US, there’s still going to be a whole lot of people out there pissed off with the current state of things, and it’s going to be really interesting to see how this plays out.

You should probably go read the full post, because I doubt I’ve done it justice here, but I don’t think it’s unreasonable to imagine the same (or a similar) thesis might be valid for Australia, so my question is: what, if anything, does this mean for our 2016 federal election?

Oops, That Didn’t Work

As an experiment, we’ve written a play. It might be viable to perform in an actual theatre, provided there’s a large screen with a projector that a few between-scene montage things can be played on. Failing that, we’ll just have to run with a short film version.

OOPS, THAT DIDN’T WORK
(a Tragedy in Three Acts, based on a true story)

by Tim Serong and Morgan Leigh

 

Cast of Characters

TIM, cranky software developer
MORGAN, lover of robots and tech promised by golden age science fiction
SUPPORT, various LIFX support people, QA people and possibly Android developers*

* (These are several different people in real life, but it should be easier to perform as one character)
Continue reading

Some Photos from Nuremberg

The Basslink cable was cut this morning in order to repair it. Concerns that this could cause Tasmanian internet speeds to go to shit appear to have been well founded (see #basslink on Twitter). In my case, my NBN wireless has gone from 21Mbps/5Mbps to 1Mbps/4Mbps, i.e. my download speed is a twentieth of what it was before, but my upload is about the same. So I may as well post some photos, right?

Continue reading