2016 Accomplishments – Another Year of Open Source Networking Software

“ISC is dedicated to developing software and offering services in support of the Internet infrastructure.”

Once a year, we attempt to catalog what we did the prior year towards supporting the infrastructure. We do have a small team who are very busy keeping F-Root going, as well as the hosting services we still provide for some non-profits, but they are too busy to give me a list of accomplishments, so this blog focuses on our software-related contributions.

8 New Internet Standards

We are huge believers in standards. If there are three legs to the Internet, one is Open Standards, one is Open Source, and the third is, I don’t know, maybe relationships? It used to be relationships, today it might be bandwidth, subscribers or “eyeballs.”

ISC developers participate vigorously in the creation and evolution of the Internet standards. In 2016, eight of the drafts ISC staff co-authored reached RFC status. Of course, we reviewed and contributed to many other drafts that we did not author. We have already implemented RFC 7873, and most of RFC 7766 in BIND 9. We published the design for Catalog Zones as implemented in BIND 9.11 as an IETF draft, in the hope that other vendors will implement for interoperability.

RFC 7766 DNS Transport over TCP - Implementation Requirements, J. Dickinson, S. Dickinson, R. Bellis, A. Mankin, D. Wessels

RFC 7793 Adding 100.64.0.0/10 Prefixes to the IPv4 Locally-Served DNS Zones Registry, M. Andrews

RFC 7819 Privacy Considerations for DHCP, S. Jiang, S. Krishnan, T. Mrugalski

RFC 7824 Privacy Considerations for DHCPv6, S. Jiang, S. Krishnan, T. Mrugalski

RFC 7828 The edns-tcp-keepalive EDNS0 Option, P. Wouters, J. Abley, S. Dickinson, R. Bellis

RFC 7844 Anonymity Profiles for DHCP Clients, C. Huitema, S. Krishnan, T. Mrugalski

RFC 7873 Domain Name System (DNS) Cookies, D. Eastlake 3rd, M. Andrews

RFC 7969 Customizing DHCP Configuration on the Basis of Network Topology, T. Lemon, T. Mrugalski

4,500 people visited our online EDNS compatibility checker at ednscomp.isc.org to test their servers in 2016. This is a grassroots effort to stimulate improved Internet compatibility for advanced DNS features such as DNSSEC and DNS cookies that appears to be modestly useful.

16 Conference Presentations

ISC staff presented at NANOG, APRICOT, RIPE, PLNOG, ICANN, and a DNS BOF in Japan in 2016, plus we put on four webinars. I realize not all of you can attend conferences, but the proceedings are generally published and this is what we do instead of “marketing.”

Slides for all of these are posted on the ISC website under Community -> Presentations.

In addition to the presentations we have given, we make other contributions to industry organizations. Tomek Mrugalski is co-chair of the IETF DHC working group, Ray Bellis is currently Program Committee chair for DNS-OARC, Eddy Winstead has given several NANOG On-the-Road tutorials in 2016, Cathy Almond is vice-chair of the UKNOF Programme Committee, Fred Baker and Warren Kumari represent ISC on the Root Server System Advisory Committee (RSSAC), Ray Bellis and Mukund Sivaraman are members of the RSSAC Caucus, and Jeff Osborn is on the ISOC Advisory Committee.

24 BIND Releases Issued

BIND continues to be an important open source resource for the Internet. The most recent estimate we can find, from one of Geoff Huston’s experiments in October 2015, showed that 55% of all DNS queries on the Internet originate from BIND resolvers.

The majority of our development and support efforts are spent on BIND. The team continues to crank out a number of maintenance and security releases every year.

• We issued maintenance versions on the 9.9, and 9.10 branches, plus our subscriber edition, based on 9.9.
• We published 10 security vulnerabilities and issued 19 patch releases. A number of the vulnerabilities were found in house, through our fuzz testing program.

We have transitioned to using the new version 3 of the CVSS scoring system because the rest of the industry was migrating, but we don’t see significant improvements in the accuracy of scoring for DNS vulnerabilities. The CVSS system is the only general standard for scoring vulnerabilities, but it can result in high-severity BIND vulnerabilities despite what seem to us fairly unusual circumstances. Also, advances in fuzz testing have increase the rate at which we are seeing vulnerability reports, so we need a better system to identify those vulnerabilities that are real threats to your DNS, versus those that are extreme corner cases.

In 2016 we held all the vulnerabilities discovered between late November and the end of the year for announcement in 2017 to give operators a break with applying BIND patches. We would have broken our silence for a zero-day vulnerability, but it seems to have worked out well.

Although we have developed what we think is a good process over the years, handling security vulnerabilities consumes a lot of our resources, interrupting our work and delaying releases. We are also very much aware that patching vulnerabilities is a burden for our users. We are considering further changes to our process going forward, to reduce the volume of security releases, while protecting the installed base.

One of the biggest changes in 2016 was the transition to the MPL2.0 open source license for BIND 9. We asked for and received a number of comments about the choice of license, and eventually made the change with the 9.11.0 release.

Over a dozen new features in the new 9.11 branch

The focus for 9.11 was on provisioning improvements for authoritative operators. Major new features in BIND 9.11 include Catalog zones, dnstap support, a DNSSEC key management utility, negative trust anchors for DNSSEC, support for standardized DNS cookies, new RNDC commands for viewing and updating zones, an embedded database option for dynamic zones, nxdomain-redirect, two features facilitating migration to IPv6, a message-compression option, transfer-message-size, and many smaller feature changes. We worked with RedHat on a significant patch, the Dyndb interface, which integrates BIND with the FreeIPA software system.

BIND 9.12 will be a much shorter cycle

BIND 9.10 was released in May of 2014. The next major version, 9.11 wasn’t ready for final release until October of 2016. The time between versions was so long (29 months) that some contributed patches became obsolete and we had added integration problems. The delay was frustrating for users who were waiting for new features that had been in planning for several years. We have decided to significantly shorten our new branch cycles for the next major BIND version in order to have a “fresher” major version. We expect to release BIND 9.12 in December, 2017, 14 months after 9.11. This could have an impact on our EOL planning that we haven’t fully realized yet, because more frequent releases + refactoring initiative = more versions with more variance between versions => higher maintenance burden.

BIND Testing - Added 2 dozen new complex system test environments

• We transitioned our on-going continuous integration testing for BIND9 from Robie to the newer Jenkins system. This has enabled us to add a number of additional operating systems and configurations to our regular testing.
• We now test on Debian 7.7, FreeBSD 10.2, FreeBSD 11, OSX 10.11, OpenBSD 5.7, Solaris 10 (Sparc), Ubuntu 13.10, Ubuntu 16.04, and Windows Server 2012. Adding support for Windows was a significant effort, spurred by the observation that we have a large number of downloads of the free Windows executables from our website.
• We added about two dozen new complex system test environments (for example, with and without threads, openssl, dnstap, libtool, and “enable-developer”(which is the kitchen sink mode) to our automated build testing.
• We continue with internal fuzz testing, still using the AFL tool. This has been a very helpful and fruitful addition to our regular quality process for BIND.
• Towards the end of 2016, we kicked off a BIND refactoring project, aimed at improving maintainability and quality by reducing code complexity.

2 New IOS Apps

We released a DIG application on the Apple iTunes store for iPhone and iPad applications. This is a port of the dig utility from BIND, intended as a convenience for experienced users. As a companion to this, in 2017 we have added a DNS Checker utility, which checks resolver support for interoperability, including EDNS support.

ISC DHCP

Despite the fact that DHCP is a well-established protocol, there continue to be new issues, new feature requests, and new products incorporating ISC DHCP. Cloud architectures drive some of these changes. At the Open Compute networking day at Facebook we learned that ISC DHCP is a fundamental component of Canonical’s new “Metal As A Service” tool, aimed at bootstrapping bare metal into application containers (maas.io).

We issued two DHCP maintenance releases in 2016, DHCP 4.3.4 and 4.1-ESV-R13. Version 4.3.4 included numerous submitted patches, including a cluster of patches for the contributed LDAP back end, and added experimental support for DHCPv4 over DHCPv6. After releasing 4.3.4 we started working on a 4.3.5 and 4.4.0.

We continue to support the 4.1 branch, even though it is past its announced EOL date, in order to keep options open for users. ISC DHCP 4.1 versions are also much smaller footprint than 4.3 versions, so keeping 4.1 around is useful for embedded applications.

We published one security vulnerability in ISC DHCP in 2016. An attacker could abuse the remote management interface, OMAPI, exhausting resources available for TCP connections.

Towards the end of 2016, Thomas Markwalder took over as the lead maintainer for ISC DHCP.

Kea

After releasing Kea 1.0 in December of 2015, we continued active development on Kea, releasing Kea 1.1 in the fall of 2016.

The major feature in Kea 1.1 was a more complex client classification system that now supports “test expressions” to determine what class a client belongs in. Kea client classification now has 3 levels, compared to the 6-7 levels of hierarchy in ISC DHCP. Kea 1.1 expanded support for retrieving host reservations from a database backend. We also added parameter passing for communicating with Kea “Hooks” extensions.

We have seen quite a bit of user interest in the overall opportunity of maintaining DHCP configuration in a shared database, and since we don’t plan to add DHCP failover to Kea if we can help it, we hired a consultant to help us test Kea in a high-availability configuration, using a MySQL database cluster backend for “failover.” Results of this test showed us that we have several opportunities for improving both Kea’s integration with the database, and our documentation and ease of use for deployment with the database backend.

Kea 1.1 included experimental support for DHCPv4-over-DHCPv6 - RFC7341, as well as preliminary support for another backend, Cassandra. The Cassandra support was contributed by a contractor working for Deutsche Telekom. We are thrilled to have DT using Kea in production and funding this external effort on Kea. Cassandra is quite different from our MySQL or the contributed PostgreSQL backend, and requires quite a bit of incremental development, testing and documentation.

At the Berlin IETF in July, a team including core Kea developers Tomek and Marcin worked with engineers from the Sysrepo project and Deutsche Telekom on experimental Yang support. Read Tomek’s blog post for more information.

2016 saw a substantial uptick in the operating system packaging for Kea. There are now packages available for RedHat, CentOS, Ubuntu, Debian, Fedora, Arch Linux, FreeBSD, OpenBSD, and NetBSD. We also see a steady stream of messages on the Kea-users mailing list from new adopters asking questions and requesting additional features. We are hopeful that 2017 will be the year in which we finally see substantial adoption of Kea.

Kea is our most OPEN open source program at ISC:

• Kea welcomes contributions via Github.
• Kea uses an open issue database that anyone can review at kea.isc.org, which includes our requirements and roadmap.
• The Kea continuous integration testing is done using Jenkins automation. The current test status is always visible at https://jenkins.isc.org.

Despite the efforts we are making to engage with the user community, we are still struggling with the financial support model for DHCP open source. 2016 was disappointing as far as getting Kea support subscribers. We created a commercial hook application to help support the project, the forensic logging application, but so far, have not figured out how to effectively market it. However, late in 2016 we received a $100,000 award from the Mozilla Open Source Software program, which will fund work on a remote management interface for Kea in 2017.

Summary Statistics

* Additions net of deletions, January 2016 - January 2017 (measured via OpenHub)

** Issues include defects, feature requests, and project tasks, so this is more of an overall activity index than a quality index.

*** Kea stats are November 2015 - November 2016