The Road to Publishing Action 1 NetOps Scores

In July 2023, we started publishing the MANRS Readiness Scores for Actions 2, 3, and 4 (IRR and RPKI) for participants in the NetOps program. We didn’t publish Action 1 scores at the time due to legitimate concerns within the community that these scores are not (yet) an accurate reflection of routing security.

Since then, we have been working hard on changes to the MANRS Observatory and the Action 1 score to be confident in publishing these scores alongside the others. In the New Year, we will make the following changes to the Action 1 score calculation and the metrics involved to reduce the number of false positives and improve the accuracy of these scores. Please refer to the description of the MANRS measurement framework for a description of the following metrics.

• M1 and M1C metrics will no longer be included in the Action 1 score calculation, but they will still be visible for your information in the detailed report in the MANRS Observatory. There is a low level of confidence in the accuracy of the source data, and we haven’t found any ways to improve it yet.
• M2 and M2C metrics derived from BGPStream data have been removed completely from the MANRS Observatory (and thus the version of the Action 1 score calculated using them). As mentioned above, there was a low confidence level in these scores; given that we have the alternative derived from the GRIP data, they have been removed completely.
• M2 and M2C metrics derived from GRIP data will be filtered to remove many false positives. Any GRIP prefix events with a valid ROA or, if there is no ROA, an IRR object in one of the RIR databases will be filtered out. We have seen 119 prefix events identified as false positives in October 2023 and 51 in November 2023, corresponding to 24 route mis-originations being completely removed from the October data and 21 from the November data.
• M3 and M3C metrics will also be filtered. We will split all bogon prefixes into ‘full’ and ‘administrative’ bogons. The reason for doing so is that the operational practice is that ‘administrative bogons’ are still being routed in some cases. Where there is an AS0 TAL (that is, for APNIC and LACNIC), we will use this to define what is a ’full’ bogon or not a bogon. Where there is no AS0 TAL (that is, AFRINIC, ARIN, and RIPE NCC), ’full’ bogons are anything with an NRO status of ‘available’ or ’ianapool’ or any prefix listed as reserved in the RFCs, and ’administrative’ bogons are anything where the prefix has had an NRO status of ’reserved’ for at least six months. Only metrics M3 and M3C will be included in the Action 1 calculation (only ’full’ bogons). There will be new metrics M3 (Admin) and M3C (Admin), including any ‘administrative’ bogons, but this will be for reference only.
• M4 and M4C metrics will be similarly filtered into ’full’ and ’administrative bogons.’ Any bogon AS with an NRO status of ’available’ or ’ianapool’ will be counted as a ‘full’ bogon, and anything with an NRO status of ’reserved’ for at least six months will be counted as an ‘administrative’ bogon. There will be new metrics M4 (Admin) and M4C (Admin) that will count the ‘administrative’ bogons, but only M4 and M4C (that is, ‘full’ bogons) will be used for the Action 1 calculation.
• The bogon filtering above has resulted in a more than 80% reduction in bogon announcements recorded by the MANRS Observatory.
• We will add endpoints to the MANRS API, allowing you to view the data in your conformance report email or the complete data for your ASN in the MANRS Observatory.

These changes will lead to higher confidence in the data and the scores. We plan to deploy these changes in January 2024 and apply them retrospectively for December 2023 so you can see their impact on your scores. That will also allow us to collect feedback from the community and prepare for publishing scores for Action 1 later next year.

We always welcome feedback on the MANRS Observatory and any of its data, whether due to these changes or for any other reason. You can email [email protected] or submit feedback via the website.

Alternatively, if you think any route leak or route mis-origination (metrics M1, M1C, M2, or M2C) incident is a false positive, you can log into your MANRS Observatory account and report it directly.