Lesson Learned: Twitter Shored Up Its Routing Security
With so many eyes on networks and traffic flows around Ukraine and Russia, any misleading announcement is quickly investigated. That’s what happened yesterday, when AS8342 (RTComm) started announcing 104.244.42.0/24, a prefix assigned to Twitter Inc.
Doug Madory from Kentik highlighted it in this post:
The same incident was also picked up by the Georgia Tech BGP Observatory:
A nearly identical incident happened on 5 February 2021, when an ISP in Myanmar tried to hijack the exact same prefix. Here is our detailed analysis of that incident. Why this same prefix again? Twitter’s DNS A records point to this prefix, so disrupting traffic toward it would potentially disrupt traffic toward Twitter.
This time, though, it went differently, because Twitter made some key changes.
These are all the DNS A records for twitter.com:
twitter.com. 776 IN A 104.244.42.193
twitter.com. 776 IN A 104.244.42.65
twitter.com. 763 IN A 104.244.42.129
twitter.com. 1400 IN A 104.244.42.1
Per RIPE RIS data, AS8342 started originating 104.244.42.0/24 on Monday, 28 March 2022 at 12:06:11 (UTC).
1648469171.000000 104.244.42.0/24 199524 8359 8342 1648469171.000000 104.244.42.0/24 28917 8359 8342 1648469171.000000 104.244.42.0/24 31027 8359 8342 1648469171.000000 104.244.42.0/24 57695 60068 8359 8342 1648469171.000000 104.244.42.0/24 8359 8342 1648469174.000000 104.244.42.0/24 199524 8359 8342 1648469174.000000 104.244.42.0/24 8660 1267 8359 8342 1648469176.000000 104.244.42.0/24 25091 8359 8342 1648469178.000000 104.244.42.0/24 56665 8359 8342 1648469184.000000 104.244.42.0/24 59414 13030 8359 8342 1648469184.000000 104.244.42.0/24 8359 8342 1648469185.000000 104.244.42.0/24 21232 13030 8359 8342 1648469185.000000 104.244.42.0/24 25091 8359 8342 1648469191.000000 104.244.42.0/24 59605 8359 8342 1648469193.000000 104.244.42.0/24 41095 8359 8342 1648469195.000000 104.244.42.0/24 15435 8359 8342 1648469196.000000 104.244.42.0/24 8607 8359 8342 1648469199.000000 104.244.42.0/24 137409 8359 8342 1648469199.000000 104.244.42.0/24 25091 8359 8342 1648469199.000000 104.244.42.0/24 9304 8359 8342 1648469200.000000 104.244.42.0/24 22652 8359 8342 1648469201.000000 104.244.42.0/24 137409 8359 8342 1648469202.000000 104.244.42.0/24 19151 8359 8342 1648469207.000000 104.244.42.0/24 6720 8447 8359 8342 1648469207.000000 104.244.42.0/24 8660 1267 8359 8342 1648469210.000000 104.244.42.0/24 51185 8359 8342 1648471814.000000 104.244.42.0/24 31027 8359 8342 1648471824.000000 104.244.42.0/24 199524 8359 8342 1648471844.000000 104.244.42.0/24 19151 8359 8342
AS8342 is connected to DataIX Internet Exchange, and its direct upstreams as per bgp paths are AS12389 (Rostelecom), AS8920 (RTComm), AS8359 (MTS), AS25091 (IP-Max), AS6939 (Hurricane Electric) etc. The above bgp update messages show that the propagation happened through AS8359 (MTS) only, while others filtered those erroneous/malicious BGP announcements successfully.
MTS (Mobile TeleSystems PJSC) is a very well-connected network, and it propagated this route across the global routing table. Interestingly, this MTS looking glass shows that routers in Moscow have the individual IPs related to Twitter’s DNS A records in the routing table with a /32 mask originated by a private ASN, meaning they are actively redirecting Twitter traffic internally.
RIPE RIS data suggest that this hijacked announcement was accepted, via MTS, by many networks and then further propagated to their neighbours, but it failed to propagate globally and didn’t create a global outage.
The reason behind this is Twitter embracing RPKI.
A Route Origin Authorisation (ROA) is a cryptographically signed object that states which AS is authorized to originate a particular IP address prefix or set of prefixes. ROAs are important protection from mis-origination.
Below are some of the routes originated by Twitter that are covered by a valid ROA.
ASN Route RPKI Status 13414 69.195.190.0/24 Valid 13414 69.195.190.0/24 Valid 13414 69.195.190.0/24 Valid 13414 69.195.191.0/24 Valid 13414 69.195.191.0/24 Valid 13414 104.244.40.0/24 Valid 13414 104.244.47.0/24 Valid 13414 104.244.44.0/24 Valid 13414 104.244.41.0/24 Valid 13414 104.244.42.0/24 Valid 13414 69.195.190.0/24 Valid 13414 69.195.190.0/24 Valid 13414 69.195.190.0/24 Valid 13414 104.244.47.0/24 Valid 13414 104.244.44.0/24 Valid 13414 199.59.148.0/22 Valid 13414 104.244.45.0/24 Valid 13414 104.244.42.0/24 Valid 13414 69.195.191.0/24 Valid 13414 69.195.191.0/24 Valid 13414 104.244.40.0/24 Valid 13414 199.16.156.0/23 Valid 13414 199.16.156.0/22 Valid 13414 104.244.46.0/24 Valid 13414 199.59.148.0/22 Valid 13414 104.244.45.0/24 Valid 13414 199.16.156.0/23 Valid 13414 199.16.156.0/22 Valid 13414 104.244.46.0/24 Valid 13414 104.244.41.0/24 Valid
Many network operators have implemented Route Origin Validation (ROV) and will not accept any routes with an “Invalid” ROA status. The network 104.244.42.0/24 originated by AS8342 was considered invalid since its data didn’t match the Twitter-created ROA. The route was not accepted by many operators because they filtered it and prevented it from propagating further to the whole routing table.
Another encouraging aspect is that Twitter has created ROAs for most (but not all) of its routes. The table below shows when they started creating ROAs.
ASN Prefix Max Length Date Created 13414 69.195.190.0/24 24 2021-08-10T04:00:00 13414 69.195.191.0/24 24 2021-08-10T04:00:00 13414 69.195.190.0/23 24 2021-08-10T04:00:00 13414 69.195.190.0/23 23 2021-08-10T04:00:00 13414 69.195.191.0/24 24 2021-08-10T04:00:00 13414 69.195.190.0/24 24 2021-09-01T04:00:00 13414 69.195.190.0/23 24 2021-09-01T04:00:00 13414 104.244.42.0/24 24 2021-11-25T05:00:00 13414 104.244.44.0/24 24 2021-11-25T05:00:00 13414 199.16.159.0/24 24 2021-11-25T05:00:00 13414 69.195.190.0/24 24 2021-11-25T05:00:00 13414 199.59.148.0/24 24 2021-11-25T05:00:00 13414 199.59.151.0/24 24 2021-11-25T05:00:00 13414 199.16.156.0/22 24 2021-11-25T05:00:00 13414 104.244.40.0/24 24 2021-11-25T05:00:00 13414 199.16.157.0/24 24 2021-11-25T05:00:00 13414 69.195.191.0/24 24 2021-11-25T05:00:00 13414 69.195.190.0/23 24 2021-11-25T05:00:00 13414 199.59.149.0/24 24 2021-11-25T05:00:00 13414 104.244.47.0/24 24 2021-11-25T05:00:00 13414 199.16.156.0/23 24 2021-11-25T05:00:00 13414 104.244.45.0/24 24 2021-11-25T05:00:00 13414 199.16.156.0/24 24 2021-11-25T05:00:00 13414 104.244.41.0/24 24 2021-11-25T05:00:00 13414 104.244.46.0/24 24 2021-11-25T05:00:00 13414 199.59.150.0/24 24 2021-11-25T05:00:00 13414 199.59.148.0/22 24 2021-11-25T05:00:00 13414 199.16.158.0/24 24 2021-11-25T05:00:00 13414 69.195.190.0/24 24 2021-12-14T05:00:00 13414 104.244.43.0/24 24 2022-01-06T05:00:00
Just like the February 2021 incident, the incident yesterday was likely a failed attempt to block a service in the country or for their own network, and it lead to accidentally leaking those routes to the global routing table. Thankfully, the industry has learned some valuable lessons from the past and is taking strong steps to fix problems like these. Last year, Twitter had not created ROAs for any of its IP resources. This year it has, and it is now much more difficult for a bogus announcement to propagate.
It is extremely important for network operators to implement effective route filtering based on verifiable information about which networks are legitimately authorised to originate which number resources (AS numbers and IP prefixes). This is what RPKI was designed for.
Having a valid Route/Route6 Object is important, but these days when many major network operators are doing ROV, it is even more important to have a valid ROA for all your resources.
This is what MANRS has been promoting. MANRS is an industry-supported initiative that builds on well-established best practices by bringing together actions that can address the most common threats in the global routing system. With more and more operators joining MANRS, incidents like these will have smaller and smaller impact.
Leave a Comment