NANOG 88 - Seattle Trip Notes

Four of us from ISC attended the NANOG 88 meeting in Seattle, Washington on June 12- 14th. NANOG is a great place to meet independent network operators from across the US, and as usual, we talked with a number of current and prospective BIND 9 and Kea DHCP users, as well as at least one IX hosting an F-root node.

ISC contributions to the event included a day-long DNS class, using DIG, delivered by Eddy Winstead. T. Marc Jones, who has joined the NANOG DEI committee, participated in a panel discussion on dealing with bias at the DEI Committee’s inaugural Embracing Equity event at the Seattle Spheres.

The NANOG DEI Committee Panel Discussion, with T. Marc Jones from ISC

Vicky Risk delivered a lightning talk on migrating away from the legacy ISC DHCP. In addition ISC helped sponsor both the Monday lunch and the snacks for the Monday evening Embracing Equity event.

The ISC information table at lunch. Left to Right: Jason Lasky, Eddy Winstead, Vicky Risk, T. Marc Jones

The DHCP packets we handed out at lunch. Len Bosack liked them.

The conference was three full days long (it used to be only two and a half), with a focus on RPKI, CDNs, DNS, and network automation. There were also several talks focused on inclusive hiring. The full agenda is posted, but the recordings won’t be available for a while. Here are notes on a few of the talks that stood out for this participant.

It turns out, baseball has some demanding networking requirements, with all of the real time flows to gigantic replay screens, lots of semi-custom equipment, and many locations making on-the-fly changes right before or even during events. Jeremy Schulman, who does networking for Major League Baseball, gave a talk on Design Driven Network Assurance. Jeremy developed a CI system for automating network design verification that he is quite proud of and which actually sounds very promising. He didn’t go into detail on the tooling for accomplishing this, but instead evangelized for the concept of verifying all of the network design details vs the actual network, rather than looking at configuration files. He argued that, although this is a lot of work, you can start with verifying the physical layer (for example, checking for the expected status of each physical ports and interface), adding more sophisticated checks in layers. What you end up with is a valuable resource for verifying the network. Given how indispensable continuous integration testing is in software development, this seems like a very worth-while investment for a network operator.

Pavel Odintsov, the author of the open source DDoS detection system FastNetMon delivered a talk on Network Telemetry on modern routers in which he compared the various flavors and versions of Netflow and IPFIX, explaining the limitations and benefits of each.

A talk about The New, Encrypted Protocol Stack & How to deal with it, by Andreas Enotiadis (MIG Sales CTO) and Bart Van de Velde (Sr. Director, Engineering, Networking CTO Office) included several interesting updates about the growth of QUIC on the Internet. QUIC is by far the dominant transport among the top 5 traffic providers; YouTube, Instagram, Netflix, Facebook Video & FaceBook. As a result 1/3 of traffic in the US is now QUIC, and >40% in the EU and LATAM regions. QUIC is designed to optimize the throughput of QUIC flows, and will ‘hog’ most of the bandwidth for QUIC, so that as more TCP flows are added, those flows share the same bandwidth ’leftover’ from QUIC, with increasingly small amounts of bandwidth per TCP flow. Eventually, this will mean that TCP traffic will struggle to survive as QUIC increases as a proportion of overall traffic. A final point was, despite the fact that this traffic is encrypted, by observing traffic characteristics, it is possible to identify the application using the (encrypted) flow, and to tell whether it is a download or streaming media (important to ensure both are handled correctly from the client perspective).

I have to admit I actually missed most of Steve Meuse’s talk on Embedded CDNs in 2023 -A status, which included an overview of the options for steering clients towards the closest cache for their high-bandwidth data. These included using BGP routing, anycasting, client-subnet data passed via the DNS, or the DNS resolver location. He discussed some of the finer points of making geo-location work.

However afterwards, in the “hallway track”, several CDN operators expressed frustration about the need to manage multiple different methods for steering client traffic because there is no one mechanism supported broadly across all their peers. It occurred to me that one reason for a relatively low rate of provision for EDNS Client Subnet among ISPs could be due to ISC’s decision to reserve that feature for the closed-source version of BIND. At the time, we felt that enabling CDNs was not a part of our mission, managing a per-subnet cache was a more complicated task, and that attaching subnet information was a privacy leak, so we didn’t want to promote it. This decision has been helpful for ISC commercially, as ISPs have to subscribe for technical support to access the BIND 9 Subscription Edition.

One of the last talks was on IP Address range hijacking by Matthew Schneider. This talk was not recorded, and was shared with in-person attendees on the basis that all details were confidential. Although there were humorous bits, as the speaker explained how he and others unraveled one such scam, overall the talk was infuriating. The bottom line message was, as the value of unused IPv4 addresses has risen, criminals have invested in elaborate schemes to steal and resell these addresses. These parasites are persistent, they are resourceful, and they are successfully executing multi-year scams to gain legal ownership of unused addresses. It seems as if the most vulnerable space, in the US at least, are addresses that were distributed prior to the founding of ARIN, which are considered ‘legacy’ address space. It is upsetting to see how vulnerable the Internet is to brazen thievery, especially given how many people have dedicated so much of their careers to developing it as a public good.