To any #Linode customers out there: stay as far as possible from the London (EU-West) data center.

Some of you may have noticed yesterday that many of my services (including my personal website, my Akkoma instance and even the Gaza archive) have been offline for a couple of hours.

Already for the past 3 days I've been experiencing issues with my Linode nginx gateway running in London (and I've experienced intermittent connectivity issues with that node a lot over the past couple of years, especially after Linode was acquired by Akamai).

When the gateway can't communicate with my home network over VPN, any services it exposes aren't reachable and you get Bad Gateway errors.

So this is how my day went yesterday:

  • I noticed for the 3rd day in a row connectivity issues to my Linode (but this time much more severe, and they were not transient).

  • I filed a support ticket to Linode with a tracepath output from my network already showing the root cause (a back-and-forth loop between London and Amsterdam that shouldn't have been there):

$ tracepath fabiomanganiello.com
 1?: [LOCALHOST]                      pmtu 1500
 1:  _gateway                                              0.990ms
 1:  _gateway                                              0.772ms
...
10:  ae3.r22.ams01.mag.netarch.akamai.com                 11.603ms asymm  9
11:  ae1.r22.lon02.mag.netarch.akamai.com                 18.534ms
12:  ae24.r01.ams01.icn.netarch.akamai.com                 9.094ms asymm  8
13:  ae7.r22.lon01.icn.netarch.akamai.com                 17.209ms asymm 11
14:  no reply
15:  no reply
...
     Too many hops: pmtu 1500
     Resume: pmtu 1500

  • Their support gaslighted me into all kind of alternative bullshit - "you're on a shared CPU, perhaps you've got noisy neighbours?", "could be it that there are some services respawning a lot on your machine?", "could it be that you don't know how to manage bridged interfaces and promiscuous mode, and after 12 years spent operating this server now everything is suddenly breaking?"

  • I lost my hopes of getting any decent amount of technical support from those folks and just went down my own way - purchased another Linode in Amsterdam, and spent the whole day migrating everything (DNS, mail server, XMPP, Gaza archive...) out of London and to Amsterdam, on a very wobbly connection to my old Linode.

  • By the time I was done (about 3:30 AM) I decided to take a look at the ticket I had opened and I found a "Connectivity Issue in London" alert. No communication from the technical support - just a badge randomly thrown there on the page reaching the same conclusions I reached, from the outside of their network, about 12h earlier.

  • Guess what? Soon afterwards connectivity to my old Linode resumed as if nothing happened, without actually me touching anything on my Linode. But in the meantime of course I had already wasted a day migrating everything out of it, while their technical support was throwing some "did you turn it off and on?" bullshit at me.

To be clear, I've been a Linode user for nearly two decades and I've never had a single issue with machines running in German or Dutch data centers. But London is a dumpster fire and you should not host anything there if you care about your users and your sanity. And the technical level of their customer support is also something that has been consistently declining over the years.

(Reminder for me: use Terraform for these things. Just because a server has been running for 12 years it doesn't mean that the moment where you need to migrate out of it won't come. And you should make sure that when it does you have the least amount of error-prone manual actions to take as possible).

Link Preview Image