Another BGP Incident Impacts TWC, Rogers, Charter and Others

Just like an incident last week, today the Internet saw another routing incident in the global BGP routing system. Here we will dig into today’s incident and – once again – explain why it’s important to implement good routing security practices like we call for in MANRS for Network Operators.

Today, 30 July, at 1:15:38 UTC, BGPStream reported a “Possible Hijack” by AS10990.

After a few more alerts, it was obvious that another BGP incident was in the making. The first announcement from AS10990 (ATLDC – Tulix Systems, Inc.) was recorded in the BGP updates.

Most of the prefixes in the above snapshot are from Rogers Communications (Canada) and advertised from AS812. None of the prefixes is announced with the same prefix-length. For example, AS10990 announced 99.225.224.0/19 whereas AS812 announced multiple de-aggregates of 99.224.0.0/11 but 99.225.224.0/19 has never been advertised.

It was the same case with most of the other prefixes I checked. Interestingly, all the neighboring aggregates were announced by Rogers. 99.228.0.0/17 announced by AS10990, whereas Rogers announced 99.228.0.0/16. These announcements lasted around five minutes and at 01:19:49 all prefixes were withdrawn. After that, AS10990 started announcing its own IPv4-only prefix 208.79.152.0/22. 179 prefixes were announced in this 4 minutes and 11 seconds. Rogers 22, Charter 16, TWC 42, VideoTron 16 and 83 others from US and Canada.

But the issue wasn’t fixed yet. AS10990 started announcing the prefixes once again at 1:35:51 UTC. These erroneous announcements lasted for almost an hour, and final withdrawal came at 2:27:27 UTC. This time there were 145 prefixes, some old prefixes were dropped and new ones were added all from the same ranges.

It is not clear why AS10990 announced all those prefixes; they peer with AS7219 which is likely the parent company of AS10990. Unfortunately, none of the announced prefixes had any ROAs created. If there were valid ROAs, these announcements would not have propagated. If you look at the AS_Path, you see AS1299 (Telia) which drops invalid ROAs. It is important to note that AS1299 (Telia Carrier) should have done a better job by filtering at all levels rather than only dropping ROA invalids.

This looks like how most BGP Route Optimizers behave. An old blog post (2016) from INAP suggests that Tulix Systems is using INAP’s MIRO (Managed Internet Route Optimizer), though MIRO is no longer listed as a product on INAP’s website. INAP does have a new Route Optimization Product called “Performance IP” and here is the snippet of what this product can do:

Performance IP ® is an outbound bandwidth service powered by our patented route optimization engine, which can reduce your network’s latency by upwards of 44%. Border Gateway Protocol (BGP) — the standard way internet traffic is routed — doesn’t take the fastest network path. It takes the shortest. Performance IP enhances BGP by optimizing route selection roughly 500 million times per day across our global network.

https://www.inap.com/network/performance-ip/

Juha Saarinen wrote about BGP Optimizers in his post, “BGP Optimisers seem a good idea until they bring down the internet” last year, highlighting the issue these ‘solutions’ pose to global Internet routing.

It is extremely important that network operators implement effective route filtering based on verifiable information about which networks are legitimately authorised to originate which number resources (AS numbers and IP prefixes). A simple prefix-limit would have solved this problem. It is also important that network operators have established and well-advertised communication channels in order to quickly resolve issues when they happen.

MANRS is an industry-supported initiative that builds on well-established best practices by bringing together actions that can address most common threats in the global routing system. Network operators, Content Providers and Internet Exchange Points that join the initiative are taking collective responsibility for the resilience and security of a critical part of the Internet infrastructure by agreeing to implement and adhere to these actions that include route filtering, global validation of number resources, coordination and anti-spoofing.

More information on how to implement these actions and join the MANRS initiative is available on the MANRS website.