Network Maintenance

To my intense amazement, it seems that NBN Co have finally done sufficient capacity expansion on our local fixed wireless tower to actually resolve the evening congestion issues we’ve been having for the past couple of years. Where previously we’d been getting 22-23Mbps during the day and more like 2-3Mbps (or worse) during the evenings, we’re now back to 22-23Mbps all the time, and the status lights on the NTD remain a pleasing green, rather than alternating between green and amber. This is how things were way back at the start, six years ago.

Continue reading

Herringback

It occurs to me that I never wrote up the end result of the support ticket I opened with iiNet after discovering significant evening packet loss on our fixed wireless NBN connection in August 2017.

The whole saga took about a month. I was asked to run a battery of tests (ping, traceroute, file download and speedtest, from a laptop plugged directly into the NTD) three times a day for three days, then send all the results in so that a fault could be lodged. I did this, but somehow there was a delay in the results being communicated, so that by the time someone actually looked at them, they were considered stale, and I had to run the whole set of tests all over again. It’s a good thing I work from home, because otherwise there’s no way it would be possible to spend half an hour three times a day running tests like this. Having finally demonstrated significant evening slowdowns, a fault was lodged, and eventually NBN Co admitted that there was congestion in the evenings.

Continue reading

NBN Fixed Wireless – Four Years On

It’s getting close to the fourth anniversary of our NBN fixed wireless connection. Over that time, speaking as someone who works from home, it’s been generally quite good. 22-24 Mbps down and 4-4.5 Mbps up is very nice. That said, there have been a few problems along the way, and more recently evenings have become significantly irritating.

There were some initial teething problems, and at least three or four occasions where someone was performing “upgrades” during business hours over the course of several consecutive days. These upgrade periods wouldn’t have affected people who are away at work or school or whatever during the day, as by the time they got home, the connection would have been back up. But for me, I had to either tether my mobile phone to my laptop, or go down to a cafe or friend’s place to get connectivity.

There’s also the icing problem, which occurs a couple of times a year when snow falls below 200-300 metres for a few days. No internet, and also no mobile phone.

These are all relatively isolated incidents though. What’s been happening more recently is our connection speed in the evenings has gone to hell. I don’t tend to do streaming video, and my syncing several GB of software mirrors happens automatically in the wee hours while I’m asleep, so my subjective impression for some time has just been that “things were kinda slower during the evenings” (web browsing, pushing/pulling from already cloned git repos, etc.). I vented about this on Twitter in mid-June but didn’t take any further action at the time.

Several weeks later, on the evening of July 28, I needed to update and rebuild a Ceph package for openSUSE and SLES. The specifics aren’t terribly relevant to this post, but the process (which is reasonably automated) involves running something like `git clone git@github.com:SUSE/ceph.git && cd ceph && git submodule update --init --recursive`, which in turn downloads a few GB of data. I’ve done this several times in the past, and it usually takes an hour, or maybe a bit more. So you start it up, then go make a meal, come back and you’re done.

Not so on that Friday evening. It took six hours.

I ran a couple of speed tests:

I looked at my smokeping graphs:

smokeping-2017-07-28

That’s awfully close to 20% packet loss in the evenings. It happens every night:

smokeping-last-10-days

And it’s been happening for a long time:

smokeping-last-400-days

Right now, as I’m writing this, the last three hours show an average of 15.57% packet loss:

smokeping-last-three-hours

So I’ve finally opened a support ticket with iiNet. We’ll see what they say. It seems unlikely that this is a problem with my equipment, as my neighbour on the same wireless tower has also had noticeable speed problems for at least the last couple of months. I’m guessing it’s either not enough backhaul, or the local NBN wireless tower is underprovisioned (or oversubscribed). I’m leaning towards the latter, as in recent times the signal strength indicators on the NTD flick between two amber and three green lights in the evenings, whereas during the day it’s three green lights all the time.

Snow in the Forest

“How’d you solve the icing problem?”
“Icing problem?”
“Might want to look into it.”
Iron Man (2008)

On a particularly cold Saturday morning a couple of years ago, my mobile phone couldn’t get any signal for a few hours. But I didn’t really care, because I had breakfast to eat, animals to feed, and nobody I urgently needed to talk to at the time. Also it came good again shortly after midday.

The following week the same thing happened, but for rather longer, i.e. well into the evening. This was enough to prompt me to use my landline to call Optus (our mobile phone provider) and report a fault. The usual dance ensued:

“Have you tried turning it off and on again?”
“Yes.”
“Have you tried a different handset?”
“Yes.”
“A different SIM?”
“Yes.”
“Holding the phone in your other hand?”
“Yes.”
“Sacrificing a lamb to the god Mercury?”
“Yes.”

I might be misremembering the details of the conversation, but you get the idea. Long story short, I got a fault lodged.

Later I received a call – on my mobile – asking if my mobile was working again. “Indeed it is, and you wouldn’t have been able to make this call if it wasn’t”, I replied. Then I asked what the problem had been. “Let me check”, said the support person. “Uhm… It says here there was… 100mm of ice on the local tower.”

Flash forwards to a couple of days ago, when snow fell down to sea level for the first time since 2005, and my mobile phone was dead again. I can only assume they haven’t solved the icing problem, and that maybe the local NBN fixed wireless tower suffers from the same affliction, as that was dead too for something like 24 hours.

It was very pretty though.

Snow in the Forest

Coda

Keep your eyes on the road, your hands upon the wheel
— Roadhouse, The Doors

It’s been a bit over two and a half months since I rolled my car on an icy corner on the way to the last day of PyCon Au 2013. My body works properly again modulo some occasional faint stiffness in my right side and I’ve been discharged by my physiotherapist, so I thought publishing some retrospective thoughts might be appropriate.

Continue reading

I Want My… I Want My… NBN

I thought I’d start this post with some classic Dire Straits, largely for the extreme tech culture shock value, but also because “MTV” has the same number of syllables as “NBN”. I’ll wait while you watch it.

Now that that’s out of the way, I thought it might be interesting to share my recent NBN experience. Almost a year ago I said “Yay NBN! Bring it on! Especially if I don’t get stuck on satellite”. Thankfully it turns out I didn’t get stuck on satellite, instead I got NBN fixed wireless. For the uninitiated, this involves an antenna on your roof, pointed at a nearby tower:

An NTD (Network Terminating Device) is affixed to a wall inside your house:

The box on the left is the NTD, the box on the right is the line from the antenna on the roof. Not pictured are the four ports on the bottom of the NTD, one of which is now plugged into a BoB Lite from iiNet. According to the labels on the NTD, the unit remains the property of NBN Co, and you’re not allowed to tamper with it:

Getting online is easy. Whatever you plug into the NTD just gets an IP address via DHCP – there’s none of that screwing around with PPPoE usernames and passwords. The BoB Lite lets you configure one of its ethernet ports as a “WAN source” instead of using ADSL, which is what I’m doing. Alternately you could plug in a random Linux box and use that as a router, or even just plug your laptop straight in, which is what I did later when trying to diagnose a fault.

The wireless itself is LTE/4G, but unlike 4G on your mobile phone (which gets swamped to a greater or lesser degree when lots of people are in the same place), each NBN fixed wireless tower apparently serves a set number of premises (see the fact sheet PDF), so speed should remain relatively consistent. Here’s the obligatory speed tests, first from my ADSL connection:

And here’s what I get via NBN wireless:

Boo-yah!

Speaking as someone who works from home and who has to regularly download and upload potentially large amounts of data, this is a huge benefit. Subjectively, random web browsing doesn’t seem wildly different than before, but suddenly being able to download openSUSE ISOs, update test VMs, and pull build dependencies at ~2 megabytes per second has markedly decreased the amount of time I spend sitting around waiting. And let’s not leave uploads out of the picture here – I push code up to github, I publish my blog, I upload tarballs, I contribute to wikis. I’ve seen too much discussion of FTTP vs. FTTN focus on download speed only, which seems to assume that we’re all passive consumers of MTV videos. OK, fine, I’m on wireless and FTTP is never going to be an option where I live, but I don’t want anyone to lose sight of the fact that being able to produce and upload content is a vital part of participating in our shiny new digital future. A reliable connection with decent download and upload speed is vital.

Now that I’ve covered the happy part of my NBN experience, I’ll also share the kinks and glitches for completeness. I rang up to get connected on August 1st. At the time, the next available installation appointment was August 21st. On that day I got a call saying the technician wouldn’t be able to make it because his laptop was broken. I offered to let them use my laptop instead, but apparently this isn’t possible, so the installation was rescheduled for September 6th. All attempts at escalating this (i.e. getting the subsequent 2.5 week delay reduced, because after all it was their fault they had a broken laptop) failed. By the time the right person at iiNet was able to rattle enough sabres in the direction of NBN Co, the new installation date was close enough that it didn’t make a difference anymore. To be clear, as far as I can tell, iiNet is not at fault; the problem seems to be one of bureaucracy combined with probable understaffing/overdemand of NBN Co (apparently this newfangled interwebs thingy is popular). As an aside, I mentioned the “broken laptop” problem to a friend, who said a friend of his had also had installation rescheduled with the same excuse. I’m not quite sure what to make of that, but I will state for the record that I seriously doubt our new fearless leaders would have been able to make matters any better had they been in power at the time.

Anyway, installation finally went ahead and all was sweetnees and light for just under three weeks, until one of those wacky spring days where it’s sun, then howling wind, then sideways rain, then sun again, then two orange signal strength lights and a red ODU light:

ODU stands for Outdoor Unit, which is apparently the antenna. For at least an hour, the lights cycled between red and green ODU, and two orange / three green signal strength. Half an hour on the phone to iiNet support, as well as plugging a laptop directly into the NTD didn’t get me anywhere. I managed to get a DHCP lease on my laptop very briefly at one point, but otherwise the connection was completely hosed. The exceptionally courteous and helpful woman in iiNet’s Fibre team logged a fault with NBN Co, and I switched my ADSL – which I had kept for just such an emergency – back on, so that I could get some work done.

The next day everything was green, so I plugged back into the NTD. About an hour after this, I got a call from the same woman at iiNet saying they’d noticed I had a connection again and had thus cancelled the fault, but that she was going to keep an eye on things for a little while, and would check back with me over the next week or two. I also received an SMS with her direct email address, so I can advise her of any further trouble. I’ve since got smokeping running here against what I hope is a useful set of remote hosts so I’ll notice if anything goes freaky while I’m asleep or otherwise away from my desk:

I really wish I knew how to get a console on the NTD, or view some logs. I was told it’s basically a dumb box, but surely it knows a bit more internally than it indicates on the blinkenlights. As I said on twitter the other day, I feel like a mechanic trapped in the drivers seat of a car, with access only to the dashboard lights. Oh, well, fingers crossed…

Rebooted Pork: A Recipe

Recording this for posterity.

Ingredients:

  • Leftover roast pork (optimally home grown free range and happy, about which another post when I find the time), diced or thinly sliced.
  • A few onions, a couple of potatoes, some mushrooms, all thinly sliced.
  • Capsicum (red or green or both), diced.
  • Crushed garlic.
  • BBQ sauce (plus possibly tomato paste – see how you go).
  • Butter.
  • Salt.
  • Pepper (bonus points for Tasmanian bush pepper).
  • Oregano.
  • Cheese, chilli sauce, tortillas (optional).

Procedure:

  1. Heat frypan, add pork.
  2. Add onions and garlic, fry for a while.
  3. Add butter, potatoes, mushrooms, BBQ sauce, salt, pepper, oregano, fry some more.
  4. Cover, lower heat, add water and/or tomato paste if it seems necessary.
  5. Go away for at least half an hour. Check your email. Read Twitter. Do some actual useful work. But make sure you’re within smelling distance of the kitchen, just in case.
  6. Add capsicum.
  7. Wait a bit more, depending on how well done you like your capsicum.
  8. Serve, either in a bowl or wrapped in a tortilla, with or without cheese and chilli sauce according to taste.

Achievement Unlocked: A Cautionary Tale

This is the story of how I rolled my car and somehow didn’t die on the way to the last day of PyCon Au 2013. I’m writing this partly for catharsis (I think it helps emotionally to talk about events like this) and partly in the hope that my experience will remind others to be safe on the roads. If you find descriptions of physical and mental trauma distressing, you should probably stop reading now, but rest assured that I am basically OK.
Continue reading