For companies running their own datacenter, setting up internal DNS infrastructure is essential for performance and ease of maintenance. Setting up a single DNS server for occasional requests is pretty straightforward, but scaling and distributing requests across multiple data centers is challenging. In this post, we’ll describe the evolution of our DNS infrastructure from a simple setup to a more distributed configuration that is capable of reliably handling a significantly higher request volume.

DNS primer

In order to talk to each other via network computers are assigned ip addresses (http://en.wikipedia.org/wiki/IP_address). IP address can be either IPv4 (e.g. 192.168.1.100) or IPv6 (e.g. fe80::3e97:eff:fe3a:ef7c). Humans are not that good in memorizing IP addresses so the Domain Name System (DNS) was created (http://en.wikipedia.org/wiki/Domain_Name_System). DNS translates human readable domain names to IP address. This allows you to type www.hulu.com in your browser to watch your favorite show instead of typing http://165.254.237.137. DNS can be used to resolve different types of names. The most important examples are:

  • A name corresponding to an IPv4 address
  • AAAA name corresponding to an IPv6 address
  • CNAME alias to another name
  • PTR ip address, corresponding to another name

The DNS response contains a response code. A few important examples are:

  • NOERROR name was resolved ok
  • NXDOMAIN there is no ip address for given name
  • NODATA there is no information for given name and request type, but given name has information for other request types.

DNS zone is any distinct, contiguous portion of the domain name space in the Domain Name System (DNS) for which administrative responsibility has been delegated to a single manager. The domain name space of the Internet is organized into a hierarchical layout of subdomains below the DNS root domain.

DNS servers will be referred to as nameservers through the rest of this document. Nameservers can be recursive if they forward a request further in case they can’t respond by themselves or authoritative if they respond to a given query without additional requests. Recursive DNS servers are called recursors.

For this document, we’ll call servers that send DNS requests to nameservers clients. Linux uses a resolv.conf file located at /etc/resolv.conf that contains DNS configuration, including the list of nameservers to use for DNS requests.

Initial configuration

Initially we had a very straightforward DNS infrastructure configuration:

  • 4 nameservers in each datacenter behind a load balancer
  • Each nameserver was running a PowerDNS (https://www.powerdns.com/) authoritative service and recursor service
  • The recursor service was configured to serve internal zones using PowerDNS server on the same host
  • Every client had a resolv.conf configured with the load balancer DNS virtual IP address and 2 ip addresses of backend nameservers. If a timeout occurred for a DNS request sent through the load balanced endpoint, a request would be sent directly to the nameservers.

This worked, but DNS uses UDP which does not guarantee delivery. While the overall DNS infrastructure worked ok, sometime names failed to resolve due to network issues or excessive traffic. As a result of the occasional failures, service owners tended to replace names with ip addresses in their configurations after seeing the occasional failures. This was especially true for services with extremely high request rates.

On the nameservers.

/etc/powerdns/pdns.conf:

default-soa-name=a.prd.hulu.com                                                                                                    launch=gsqlite3                                                                                                    gsqlite3-database=/opt/dns/sqlite_db                                                                                                    local-port=5300log-failed-updates=yes

/etc/powerdns/recursor.conf:

allow-from=127.0.0.0/8, 192.168.1.0/24, 10.0.0.0/8, 1.2.3.0/24                                                                                                    forward-zones=10.in-addr.arpa=127.0.0.1:5300,prd.hulu.com=127.0.0.1:5300,stg.hulu.com=127.0.0.1:5300,tst.hulu.com=127.0.0.1:5300                                                                                                     # /etc/powerdns/empty_zone is an empty file                                                                                                    auth-zones=hulu.com.prd.hulu.com=/etc/powerdns/empty_zone,hulu.com.stg.hulu.com=/etc/powerdns/empty_zone,hulu.com.tst.hulu.com=/etc/powerdns/empty_zone                                                                                                    local-address=0.0.0.0                                                                                                    log-common-errors=yes                                                                                                    max-negative-ttl=60threads=2
umair-akbar-0*UvH5f5xWL qpi o  - DNS infrastructure at Hulu

On the clients

/etc/resolv.conf:

nameserver 10.1.1.53 # dns ip on load balancer                                                                                                    nameserver 10.1.1.120 # nameserver behind load balancer                                                                                                    nameserver 10.1.1.121 # another nameserver behind load balancersearch prd.hulu.com stg.hulu.com tst.hulu.com
umair-akbar-0*BbLhB6lKETug bgI - DNS infrastructure at Hulu

Adding local DNS caching

In order to minimize the chance of DNS resolution failure because of network issues, we decided to setup a local DNS cache on each client. We compared several popular DNS caches such as unbound, pdnsd, PowerDNS recursor and nscd. Unbound demonstrated the best performance, excellent stability and ease of configuration, so we gradually rolled it out to all clients.

Initially, resolv.conf was configured to point to the load balanced DNS nameservers. We updated it to list unbound first, followed by the load balanced nameserver in case unbound failed. It is important to note that if you have multiple nameserver entries in resolv.conf by default they would be used in the order they are listed with 2 retries and 5 second timeout (see man resolv.conf for more details). This means, that each failing nameserver in resolv.conf would cause 10 seconds delay in resolving name. However, this is not the case for a local nameserver running on 127.0.0.1. Resolver detects that port udp 53 on localhost is not listening almost instantaneously and tries next nameserver from the list so delay would be negligible. However, it’s noteworthy that Python, Ruby, Java and Node.js cache nameservers IPs from resolv.conf. So, services running in those languages can see resolution delays when restarting the local unbound nameserver.

Results were very good, traffic decreased and there were no failures while resolving DNS names.

On the clients.

/etc/resolv.conf:

nameserver 127.0.0.1 # unbound                                                                                                    nameserver 10.1.1.53 # dns ip on load balancersearch prd.hulu.com stg.hulu.com tst.hulu.com

/etc/unbound/unbound.conf:

server:                                                                                                        access-control: 0.0.0.0/0 allow                                                                                                        
  
prefetch: yes                                                                                                        do-ip6: no                                                                                                        local-zone: "10.in-addr.arpa." transparent                                                                                                    remote-control:                                                                                                        control-enable: no                                                                                                    forward-zone:                                                                                                        name: "."  forward-addr: 10.1.1.53
umair-akbar-0*GLp0pt29dmcoPm S - DNS infrastructure at Hulu

Blocking AAAA queries

We noticed that a significant part of responses were nxdomain for names like host.prd.hulu.com.tst.hulu.com. This was caused by 2 reasons: search list in resolv.conf and AAAA queries from python. We don’t use IPv6 in our internal network, so AAAA queries were only adding unnecessary traffic and increasing the chance that a valid request would not be answered due to the fact that DNS uses UDP and UDP is a protocol without delivery guarantee. After a close look at unbound’s features, we blocked most of the bad queries by creating zones, which unbound would serve locally.

On the clients.

/etc/unbound/unbound.conf:

server:                                                                                                        access-control: 0.0.0.0/0 allow                                                                                                        prefetch: yes                                                                                                        do-ip6: no                                                                                                        local-zone: "com.prd.hulu.com" static                                                                                                        local-zone: "com.stg.hulu.com" static                                                                                                        local-zone: "com.tst.hulu.com" static                                                                                                        local-zone: "10.in-addr.arpa." transparent                                                                                                    remote-control:                                                                                                        control-enable: no                                                                                                    forward-zone:                                                                                                        name: "."  forward-addr: 10.1.1.53

Making unbound daemons talk directly to nameservers

Everything was working OK until we had a failure in the load balancer we were using for DNS. Services were not able to resolve DNS names and started to fail. Given how important DNS is, we decided that DNS should not be tied to the reliability of a load balancer, so we decided to remove the load balancer from the DNS infrastructure. This was easy to accomplish: instead of pointing to the load balancer VIP, we reconfigured unbound to talk directly to the nameservers.

On the clients.

/etc/unbound/unbound.conf:

server:                                                                                                        access-control: 0.0.0.0/0 allow                                                                                                        prefetch: yes                                                                                                        do-ip6: no                                                                                                        local-zone: "com.prd.hulu.com" static                                                                                                        local-zone: "com.stg.hulu.com" static                                                                                                        local-zone: "com.tst.hulu.com" static                                                                                                        local-zone: "10.in-addr.arpa." transparent                                                                                                    remote-control:                                                                                                        control-enable: no                                                                                                    forward-zone:                                                                                                        name: "."                                                                                                        forward-addr: 10.1.1.120                                                                                                        forward-addr: 10.1.1.121                                                                                                        forward-addr: 10.1.1.122  forward-addr: 10.1.1.123
umair-akbar-0*Yf seBcODtoKBgtf - DNS infrastructure at Hulu

Making DNS resolution immune to internet connection outages

Some time after that, we had an internet connection issue involving outbound traffic in one of our datacenters. It is important to note, that authoritative DNS for our externally facing DNS names is hosted by our CDN. Resolution of these names in our datacenter also use the CDN. We do this in order to have unified DNS resolution from the internal Hulu network and from the internet (from Hulu’s customers perspective). This unified approach allows us to easily reproduce any customer reported issues related to DNS. But this also means that without an internet connection we can’t resolve our external DNS names, even when the IPs that they resolve to physically reside in our datacenters. Because names couldn’t be resolved many services in this datacenter stopped working even though they didn’t have a dependency on the outside internet. We decided to modify DNS so that a failure of the internet connection in a single datacenter wouldn’t affect DNS. This was a significant change to the nameservers. What we did:

  • The original PowerDNS recursor service was retired, and we switched to unbound
  • Incoming DNS requests to the nameservers were served by unbound, configured with appropriate zones.
  • Requests for internal zones were forwarded to the PowerDNS authoritative services. Each unbound instance talked to all PowerDNS servers, so failure of one PowerDNS authoritative server would have a small performance penalty
  • External requests were forwarded to another unbound layer. When the internet connection worked, this layer talked to the unbound instances in the local datacenter. If the outbound internet connection fails, the unbound instances from other datacenter are added, so that external names can continue to be resolved. It is important to note that using unbound recursors from other datacenter can result in wrong geo-location information for external names. Instead of getting an address close to datacenter where request originated from, an address close to datacenter with alive internet connection is returned. This can increase round trip times and service latencies.

On the nameservers.

/etc/unbound/unbound.conf:

server:                                                                                                        interface: 0.0.0.0                                                                                                        access-control: 0.0.0.0/0 allow                                                                                                        prefetch: yes                                                                                                        rrset-roundrobin: yes                                                                                                        do-ip6: no                                                                                                        do-not-query-localhost: no                                                                                                        extended-statistics: yes                                                                                                        local-zone: "com.prd.hulu.com" static                                                                                                        local-zone: "com.stg.hulu.com" static                                                                                                        local-zone: "com.tst.hulu.com" static                                                                                                        local-zone: "10.in-addr.arpa." transparent                                                                                                    forward-zone:                                                                                                        name: "10.in-addr.arpa"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "prd.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "stg.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]5300                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "tst.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "."                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]  forward-addr: [email protected]

/etc/unbound/unbound-5301.conf:

server:                                                                                                        interface: 0.0.0.0                                                                                                        port: 5301                                                                                                        access-control: 0.0.0.0/0 allow                                                                                                        prefetch: yes                                                                                                        do-ip6: no                                                                                                        extended-statistics: yes                                                                                                        pidfile: /var/run/unbound-5301.pid                                                                                                    remote-control:  control-port: 8954

/opt/dns/bin/dns-monitor.sh:

#!/bin/bash                                                                                                    set -u                                                                                                    dc1_ips="10.1.1.120 10.1.1.121 10.1.1.122 10.1.1.123"                                                                                                    dc2_ips="10.2.1.120 10.2.1.121 10.2.1.122 10.2.1.123"                                                                                                    case "$(hostname -s|cut -f1 -d-)" in                                                                                                              dc1) here_ips=$dc1_ips; there_ips=$dc2_ips ;;                                                                                                            dc2) here_ips=$dc2_ips; there_ips=$dc1_ips ;;                                                                                                              * ) exit 1 ;;                                                                                                    esac                                                                                                    PID_FILE=/var/run/$(basename $(readlink -f $0)).pid                                                                                                    all_ips=$(echo $dc1_ips $dc2_ips|sed -e 's/ /\n/g'|sort)                                                                                                    check_upstream() {                                                                                                            for ip; do                                                                                                                    [ "$(dig @$ip -p 5301 +tries=1 +time=1 +short \                                                                                                                        google-public-dns-a.google.com)" == "8.8.8.8" ] && return 0                                                                                                            done                                                                                                            return 1                                                                                                    }                                                                                                    set_zone() {                                                                                                            forwarders=()                                                                                                            for ip; do                                                                                                                    forwarders+=(${ip}@5301)                                                                                                            done                                                                                                            /usr/sbin/unbound-control forward_add . ${forwarders[@]}                                                                                                    }                                                                                                    run_check() {                                                                                                            current_zone=$(/usr/sbin/unbound-control list_forwards|grep '^\. '| \                                                                                                                    sed -e 's/.*forward: //' -e 's/ /\n/g'|sort)                                                                                                            here_status=down                                                                                                            there_status=down                                                                                                            cross_dc_status=down                                                                                                            check_upstream $here_ips && here_status=up                                                                                                            [ "$current_zone" == "$all_ips" ] && cross_dc_status=up                                                                                                            [ "${here_status}${cross_dc_status}" == "upup" ] && {                                                                                                                    set_zone $here_ips                                                                                                                    return                                                                                                            }                                                                                                            check_upstream $there_ips && there_status=up                                                                                                            [ "${here_status}${cross_dc_status}${there_status}" == "downdownup" ] && {                                                                                                                    set_zone $all_ips                                                                                                                    return                                                                                                            }                                                                                                    }                                                                                                    get_lock() {                                                                                                            touch ${PID_FILE}.test || {                                                                                                                    echo "Can't create ${PID_FILE}"                                                                                                                    exit 1                                                                                                            }                                                                                                            rm -f ${PID_FILE}.test                                                                                                            while true; do                                                                                                                    set -- $(LC_ALL=C ls -il ${PID_FILE} 2>/dev/null)                                                                                                                    if [ -z "${1:-}" ] ; then                                                                                                                            ln -s $$ $PID_FILE && return 0                                                                                                                    else                                                                                                                            ps ${12} >/dev/null 2>&1 && return 1                                                                                                                            find $PID_FILE -inum ${1} -exec rm -f {} \;                                                                                                                    fi                                                                                                             done                                                                                                    }                                                                                                    get_lock || exit 1                                                                                                    exec &>/dev/null                                                                                                    while sleep 5; do                                                                                                        run_checkdone

/etc/cron.d/dns-monitor:

SHELL=/bin/bash
* * * * * root /opt/dns/bin/dns-monitor.sh
umair-akbar-0*sZCelIaQppklIgPr - DNS infrastructure at Hulu

Serving programmatically generated names

In order to test certain services, we needed a testing host to have a name in the *.hulu.com domain. In case when testing is done using a workstation it is not always possible to use a dedicated name. We decided to use special domain ip.hulu.com. Names in form A.B.C.D.ip.hulu.com are resolved to ip address A.B.C.D. This can be done using unbound python extension. Another useful feature that can be implemented in python extension is datacenter aware names. For example if we have service.dc1.prd.hulu.com in datacenter dc1 and service.dc2.prd.hulu.com in dc2 we can have virtual domain dc.prd.hulu.com so that service.dc.prd.hulu.com would resolve to the proper name local to this datacenter.

On nameserver:

/etc/unbound/unbound.conf:

server:                                                                                                        interface: 0.0.0.0                                                                                                        access-control: 0.0.0.0/0 allow                                                                                                        prefetch: yes                                                                                                        rrset-roundrobin: yes                                                                                                        do-ip6: no                                                                                                        do-not-query-localhost: no                                                                                                        extended-statistics: yes                                                                                                        local-zone: "com.prd.hulu.com" static                                                                                                        local-zone: "com.stg.hulu.com" static                                                                                                        local-zone: "com.tst.hulu.com" static                                                                                                        local-zone: "10.in-addr.arpa." transparent                                                                                                    forward-zone:                                                                                                        name: "10.in-addr.arpa"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "prd.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "stg.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "tst.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "dc.prd.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "dc.stg.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "dc.tst.hulu.com"                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                    forward-zone:                                                                                                        name: "."                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]                                                                                                        forward-addr: [email protected]
  
forward-addr: [email protected]

/etc/unbound/unbound-5301.conf:

server:                                                                                                        interface: 0.0.0.0                                                                                                        port: 5301                                                                                                        access-control: 0.0.0.0/0 allow                                                                                                        prefetch: yes                                                                                                        do-ip6: no                                                                                                        extended-statistics: yes                                                                                                        pidfile: /var/run/unbound-5301.pid                                                                                                        module-config: "validator python iterator"                                                                                                    python:                                                                                                        python-script: "/etc/unbound/unbound-5301.py"                                                                                                    remote-control:  control-port: 8954

/etc/unbound/unbound-5301.py:

DC = "dc1" # or dc2 depending which datacenter we are in                                                                                                    def init(id, cfg): return True                                                                                                    def deinit(id): return True                                                                                                    def inform_super(id, qstate, superqstate, qdata): return True                                                                                                    def create_response(id, qstate, in_rr_types, out_rr_type, pkt_flags, msg_answer_append):                                                                                                            # create instance of DNS message (packet) with given parameters                                                                                                            msg = DNSMessage(qstate.qinfo.qname_str, out_rr_type, RR_CLASS_IN, pkt_flags)                                                                                                            # append RR                                                                                                            if qstate.qinfo.qtype in in_rr_types:                                                                                                                    msg.answer.append(msg_answer_append)                                                                                                            # set qstate.return_msg                                                                                                            if not msg.set_return_msg(qstate):                                                                                                                    qstate.ext_state[id] = MODULE_ERROR                                                                                                                    return True                                                                                                            # we don't need validation, result is valid                                                                                                            qstate.return_msg.rep.security = 2                                                                                                            qstate.return_rcode = RCODE_NOERROR                                                                                                            qstate.ext_state[id] = MODULE_FINISHED                                                                                                            return True                                                                                                    def operate(id, event, qstate, qdata):                                                                                                            if (event == MODULE_EVENT_NEW) or (event == MODULE_EVENT_PASS):                                                                                                                    a = qstate.qinfo.qname_str.split('.')                                                                                                                    if len(a) > 5 and a[-1] == '' and a[-2] == 'com' and a[-3] == 'hulu':                                                                                                                            if len(a) == 8 and a[-4] == 'ip' \                                                                                                                                        and 0 <= int(a[-5]) <= 255 \                                                                                                                                        and 0 <= int(a[-6]) <= 255 \                                                                                                                                        and 0 <= int(a[-7]) <= 255 \                                                                                                                                        and 0 <= int(a[-8]) <= 255:                                                                                                                                msg_answer_append = "{0} 300 IN A {1}.{2}.{3}.{4}".format(qstate.qinfo.qname_str, a[-8], a[-7], a[-6], a[-5])                                                                                                                                create_response(id, qstate, [RR_TYPE_A, RR_TYPE_ANY], RR_TYPE_A, PKT_QR | PKT_RA | PKT_AA, msg_answer_append)                                                                                                                                return True                                                                                                                        if a[-5] == 'dc':                                                                                                                                a[-5] = DC                                                                                                                                msg_answer_append = "{0} 300 IN CNAME {1}".format(qstate.qinfo.qname_str, '.'.join(a))                                                                                                                                create_response(id, qstate, [RR_TYPE_CNAME, RR_TYPE_A, RR_TYPE_ANY], RR_TYPE_CNAME, PKT_QR | PKT_RA, msg_answer_append)                                                                                                                                return True                                                                                                                    else:                                                                                                                            # pass the query to validator                                                                                                                            qstate.ext_state[id] = MODULE_WAIT_MODULE                                                                                                                            return True                                                                                                            if event == MODULE_EVENT_MODDONE:                                                                                                                    log_info("pythonmod: iterator module done")                                                                                                                    qstate.ext_state[id] = MODULE_FINISHED                                                                                                                    return True                                                                                                            log_err("pythonmod: bad event")                                                                                                            qstate.ext_state[id] = MODULE_ERROR                                                                                                            return True

Early blocking of unwanted queries

After more research, we figured out that blocking of unwanted queries can be done early with unbound extension. Specifically in our case we wanted to block resolving of all IPv6 names from Hulu domain (since we are not using IPv6 in our internal network). Here is how this can be done:

On the clients.

/etc/unbound/unbound.conf:

server:                                                                                                    server:                                                                                                        access-control: 0.0.0.0/0 allow                                                                                                        prefetch: yes                                                                                                        do-ip6: no                                                                                                        rrset-roundrobin: yes                                                                                                        chroot: ""                                                                                                        local-zone: "com.prd.hulu.com" static                                                                                                        local-zone: "com.stg.hulu.com" static                                                                                                        local-zone: "com.tst.hulu.com" static                                                                                                        local-zone: "10.in-addr.arpa." transparent                                                                                                        module-config: "validator python iterator"                                                                                                    python:                                                                                                        python-script: "/etc/unbound/unbound.py"                                                                                                    remote-control:                                                                                                        control-enable: no                                                                                                    forward-zone:                                                                                                        name: "."                                                                                                        forward-addr: 10.1.1.120                                                                                                        forward-addr: 10.1.1.121                                                                                                        forward-addr: 10.1.1.122  forward-addr: 10.1.1.123

/etc/unbound/unbound.py:

def init(id, cfg): return True                                                                                                    def deinit(id): return True                                                                                                    def inform_super(id, qstate, superqstate, qdata): return True                                                                                                    def operate(id, event, qstate, qdata):                                                                                                            if (event == MODULE_EVENT_NEW) or (event == MODULE_EVENT_PASS):                                                                                                                    qtype = qstate.qinfo.qtype                                                                                                                    qname_str = qstate.qinfo.qname_str                                                                                                                    if (qtype == RR_TYPE_AAAA and qname_str.endswith(".hulu.com.")):                                                                                                                            # create instance of DNS message (packet) with given parameters                                                                                                                            msg = DNSMessage(qname_str, qtype, RR_CLASS_IN, PKT_QR | PKT_RA | PKT_AA)                                                                                                                            # set qstate.return_msg                                                                                                                            if not msg.set_return_msg(qstate):                                                                                                                                    qstate.ext_state[id] = MODULE_ERROR                                                                                                                                    return True                                                                                                                    # we don't need validation, result is valid                                                                                                                    qstate.return_msg.rep.security = 2                                                                                                            qstate.return_rcode = RCODE_NXDOMAIN                                                                                                            qstate.ext_state[id] = MODULE_FINISHED                                                                                                            return True                                                                                                    else:                                                                                                            # pass the query to validator                                                                                                            qstate.ext_state[id] = MODULE_WAIT_MODULE                                                                                                            return True                                                                                                    if event == MODULE_EVENT_MODDONE:                                                                                                            # log_info("pythonmod: iterator module done")                                                                                                            qstate.ext_state[id] = MODULE_FINISHED                                                                                                            return True                                                                                                    log_err("pythonmod: bad event")                                                                                                    qstate.ext_state[id] = MODULE_ERRORreturn True

Cleanup of search list in resolv.conf

Finally, we removed the search list from /etc/resolv.conf on the clients. Search list is useful when you want to use short names instead of fully qualified domain names (e.g. myservice instead of myservice.prd.hulu.com). But this is definitely a bad practice since if you have myservice.prd.hulu.com and myservice.stg.hulu.com if you use short name you would get the one which is first in the search list. Instead of a search list, we are now using domain resolv.conf directive, which still allows usage of short names, but is limited to a single domain, so there will be no ambiguity in name resolution.

On the clients.

/etc/resolv.conf:

nameserver 127.0.0.1 # unbound                                                                                                    nameserver 10.1.1.53 # dns ip on load balancerdomain prd.hulu.com

DNS naming conventions

We found that once a DNS name starts being used, it is extremely hard to deprecate. Consequently, it’s best to think carefully about naming schemas and follow them rigorously. We found that it’s generally a good idea to have domains specific to production, testing, etc:

  • prd.company.com
  • tst.company.com
  • dev.company.com

If you have multiple data centers it may be useful to have data center specific domains:

  • dc0.company.com
  • dc1.company.com

Using high level names like git.company.com, help.company.com etc for internal names seems a good idea to have short names and save on typing, but their support may become quite complicated with time. It’s often better to use domains specific to production, testing, and dev (e.g. git.prd.company.com etc).

Closing thoughts

In the first portion of our upgrade we added local caches that talked directly to nameservers and proxied traffic between datacenters. This immediately minimized the effect of internal or external network outages on name resolution. Once we had a more stable setup, we increased the performance of the overall system by minimizing a significant amount of irrelevant queries. We’re continuing to improve on our DNS infrastructure by adding programmatically generated names to allow services in a datacenter to automatically find their counterpart services and databases. We’ve been very happy with our new setup and hope that the details we’ve shared here can prove useful to others looking to scale up their own DNS infrastructure.

About the Author

USA

Umair Akbar | Cloud Engineer

Umair Akbar is a Senior Information Security Engineer with over 5 years of experience leading the development and daily management of InfoSec systems.

View All Articles