Major Route Leak by AS28548 – Another BGP Optimizer?

It’s usually not good news to get a Twitter notification from Qrator on a Saturday morning. Unfortunately, this one was no exception. Now that a severe incident has happened, let’s analyze what happened and learn from it together. 

The tweet came from Radar by Qrator, a monitoring system which detects a wide range of network anomalies, such as BGP leaks and hijacks:

The post suggested there was a major route leak incident by AS28548 (Cablevisión, S.A. de C.V) in Mexico. Thanks to Doug Madory, Director of Internet Analysis at Kentik, for providing more insights into the issue as follows:

With these data points we could start digging into some raw data. The best place to look at it is the Isolario MRT dumps. First, let us look at the connectivity of AS28548 here. According to RIPE Stats, it is a stub network and does not have any downstream networks. They peer with 4 ASNs (AS3356, AS3549, AS14178, and AS18734).

Looking at the current routing table entries, it is clear that AS18734 is their preferred path for most of their 151 IPv4 and 14 IPv6 routes they announce from AS28548.

At 4:36:41 AM UTC (10:36 PM in Mexico) they made some changes in the BGP, most likely adding an AS Path prepend towards AS18734 in order to prefer AS14178 transit. In doing so they started leaking all routes they were receiving from AS14178 and started announcing to AS18734.

The data I have captured from Isolario BGP updates suggests they leaked more than 7,200 unique IPv4 prefixes via AS18734. More than 1,290 ASNs were impacted from around 100+ countries (as per the IRR country code).

The country hit hardest by the incident was obviously Mexico, but many other networks from far away countries as mentioned in the above map were also impacted. Here is the list:

The last update from AS28548 came at 04:48:21:

 And after that they announced their own prefixes at 04:50:10 UTC:

There is some interesting information in the announcements below.

Take example of 201.157.24.0/24, this prefix belongs to AS22566 (Maxcom Telecom), but they have never advertised /24 before and they only announce /17 aggregate. They have only announced the following prefixes and have route objects for it as per bgp.he.net:

This looks like how most BGP Route Optimizers behave. We have all seen this in the past and covered in MANRS blogs (we analyzed a major incident last year here). We have warned the community about the possible dangers of using one in your network.

AS18734 should have blocked these announcement easily through AS filtering, knowing this is a stub network.

It is extremely important that network operators implement effective route filtering based on verifiable information about which networks are legitimately authorized to originate which number resources (AS numbers and IP prefixes). A simple prefix-limit would have solved this problem. It is also important that network operators have established and well-advertised communication channels in order to quickly resolve issues when they happen.

MANRS is an industry-supported initiative that builds on well-established best practices by bringing together actions that can address the most common threats in the global routing system.

By being part of MANRS, close to 600 network operators, Internet exchange points, and cloud and content delivery network providers take concrete actions to contribute to the resilience and security of a critical part of the Internet infrastructure. The actions include route filtering, global validation of number resources, coordination, and anti-spoofing.

For more information on how to implement these actions and join the MANRS initiative, visit the MANRS website.