DNS and CoreDNS in Kubernetes: The Indispensable Phonebook for Your Cluster

As you start your journey with K8s, you will quickly realize that communication between different components in the cluster is extremely important. Pods need to talk to Pods, Pods need to call Services. Just like we need a phone book to find someone else’s number, applications in K8s need a mechanism to find each other’s IP addresses. That mechanism is DNS (Domain Name System) .

In this article, we will learn together:

1. What is DNS and How Does It Work? (Basics and How to Check)
2. Why is DNS important in Kubernetes?
3. How Kubernetes DNS works.
4. What is CoreDNS and what role does it play?
5. CoreDNS configuration and operation.
6. How to check and troubleshoot basic DNS issues in K8s.

This article is designed for beginners and those who have some experience with Kubernetes. We will try to explain the concepts in the simplest and most understandable way to understand how DNS works, which is an important foundation before we dive into how Kubernetes uses and manages DNS for applications inside the cluster.

Part 1: DNS - "The Internet's Phonebook" and How It Works

1.1. What is DNS?

Simply put, DNS is like the "Phone Book of the Internet" . It maps domain names that are easy for humans to remember, such as www.google.com, to IP addresses, such as 142.250.199.14. IP addresses are the actual locations of servers that host websites or services on the network.

Thanks to DNS, we don’t need to remember long and complicated IP numbers to access websites. We just type the domain name into our browser, and DNS takes care of the rest – finding the corresponding IP address so our computer can connect to it. This process is called domain name resolution .

1.2. How does DNS work?

But how does DNS know the IP address for a particular domain name? The answer is: a DNS server doesn’t know everything by itself. It relies on a hierarchy of different types of DNS servers working together to find the answer. These intermediate servers are often referred to collectively as DNS resolvers .

DNS Resolver: A server that stores DNS records (or knows how to find them) and responds to DNS queries.

When you type a domain name into your browser, a series of steps takes place to find the IP address:

1. Check Local Caches: The first place your computer looks.
2. Ask Recursive DNS Server: Usually the DNS server of your Internet Service Provider (ISP) or public DNS (Google DNS, Cloudflare DNS).
3. Root DNS Server: Top of the DNS hierarchy.
4. Ask Top-Level Domain DNS Server (TLD DNS Server): Manages domain name extensions such as .com, .org, .vn.
5. Ask the Authoritative DNS Server: The server that holds the final and correct information about the domain name you are looking for.

Let's go into step by step details with the example you want to access www.example.com:

Step 1: Check Local Caches

Your computer checks its own cache first, which is a temporary storage of domain name-IP address pairs that your computer has recently accessed. If the domain name you are looking for is in the cache, your computer will use that IP address and skip the next steps.

Local cache types that can contain this information:

• Browser Cache: Your browser may have saved the IP of the website you visited previously.
• Operating System DNS Cache: The operating system also maintains a DNS cache based on the TTL (Time To Live) of the DNS record.
• Hosts File: You may have configured the domain name - IP mapping yourself in your computer's hosts file (/etc/hosts on Linux/macOS, C:\Windows\System32\drivers\etc\hosts on Windows).

Step 2: Ask Recursive DNS Servers

If it is not found in the local cache, the computer will send the query to the DNS server configured in its network settings (usually your ISP's, or you can configure it to use Google DNS 8.8.8.8, Cloudflare 1.1.1.1). This server is called a Recursive DNS Server (or Recursive Resolver).

This recursive server will also check its own cache. If it has an answer, it will send it back to your computer. If not, it will start the process of "questioning" other DNS servers, starting from the top level.

Step 3: Ask the Root DNS Servers

The Recursive Resolver will ask one of the Root DNS Servers . There are 13 root server clusters around the world, managed by different organizations. They don't know the IP address of www.example.com, but they do know who manages the .com top-level domain (TLD). The Root Server will return the address of the TLD DNS server for .com.

You can see a list of servers that manage TLDs using the dig command (more on that later):

dig +short NS com dig +short NS org # ... và các TLD khác

Step 4: Ask the Top Level Domain DNS Servers (TLD DNS Servers)

The Recursive Resolver then asks the TLD DNS server for .com (the address just received from the Root Server). This TLD server does not know the IP of www.example.com, but it knows who the Authoritative DNS Server is for the entire example.com domain. The TLD Server returns the address of that Authoritative DNS Server.

You can view the authoritative nameservers for a particular domain:

dig +short NS example.com dig +short NS google.com

Step 5: Ask Authoritative DNS Servers

Finally, the Recursive Resolver asks the Authoritative DNS Server for example.com. This is where the actual DNS records for this domain are stored. The Recursive Resolver will ask for the A record (or AAAA record for IPv6) for www.example.com.

A Record is a DNS record that maps a domain name (or subdomain name) to an IPv4 address.

The Authoritative DNS Server will look up the A record for www in the example.com zone and return the corresponding IP address (e.g. 93.184.216.34) to the Recursive Resolver.

The Recursive Resolver receives this IP address, stores it in its cache (for future queries), and sends the final response back to your computer. Your browser now has an IP address and can initiate a connection to the web server www.example.com.

1.3. See DNS in action with dig

We can use the dig (Domain Information Groper) command line tool to perform DNS queries and see how this process works. dig is available on most Linux and macOS systems. On Windows, you can install it via Chocolatey :

choco install bind-tools # Cung cấp dig và các công cụ DNS khác

To find the IP address for www.example.com, we use the command:

dig +short www.example.com

The +short option tells dig to only display the answer part (IP address). The result will be similar to this:

93.184.216.34

But how does dig find this IP? It also performs the same steps as the browser: checking the cache, asking the recursive resolver... To see the details of the "questioning" process through the DNS server levels, we use the +trace option:

dig +trace www.example.com

The output of this command will be quite lengthy, showing each step of the query from your computer to the recursive resolver, then to the root server, the TLD server, and finally the authoritative server to get the answer. Try running this command to see for yourself!

1.4. Using dig to Debug Basic DNS Issues

If you're having trouble accessing a website, dig is a useful tool to check if the problem is with DNS. For example:

• dig <domain>: View detailed information about the record (including TTL).
• dig <domain> A: Query only A records (IPv4).
• dig @<dns-server> <domain>: Query a domain name using a specific DNS server (e.g. dig @8.8.8.8 google.com). This is useful for comparing results between different DNS servers.

Part 2: Why is DNS important in Kubernetes?

In Kubernetes, everything is dynamic :

• Pods are ephemeral: Pods can be created, deleted, scaled up/down at any time. Each time a Pod is recreated, it will typically have a new IP address.
• Services provide a stable virtual IP address: Kubernetes Services is an abstraction layer that provides a virtual IP address (ClusterIP) and a fixed DNS name for a group of Pods running the same application. The client only needs to know the Service name, and does not need to care about the IP of each individual Pod changing constantly.

The problem is: How can a Pod (e.g. frontend) find the IP address of a Service (e.g. backend) using only the name of that Service?

This is where Kubernetes DNS comes in. It provides an internal name resolution mechanism for the cluster, allowing Pods and Services to discover each other using DNS names instead of dynamic IP addresses.

Imagine you have a web application with 2 parts: frontend (running in Pods) and backend (running in other Pods). You create a Service called backend-service that points to the backend Pods. Now, from the frontend Pod, you just need to call the backend-service address and K8s DNS will automatically resolve it to the ClusterIP address of the backend-service, and Kube-proxy will route the traffic to one of the healthy backend Pods.

Without DNS, the frontend Pod would have to somehow keep track of the ever-changing IP addresses of the backend Pods or the Service's ClusterIP address - which is extremely complex and inefficient.

Part 3: How Kubernetes DNS Works

Kubernetes defines a standard DNS specification for naming and resolving resources in the cluster. When a Service or Pod is created, the K8s DNS server (usually CoreDNS) automatically creates the corresponding DNS records.

3.1. DNS records for Services:

This is the most commonly used record type. A Service named <service-name> in the namespace <namespace-name> will have a DNS A record (or AAAA record for IPv6) with the format:

<service-name>.<namespace-name>.svc.<cluster-domain>

• <service-name>: Name of the Service.
• <namespace-name>: Namespace containing that Service.
• svc: Specifies that this is a Service.
• <cluster-domain>: The cluster's suffix domain name, usually cluster.local by default.

For example: A Service named my-nginx in the default namespace will have a fully qualified DNS name of my-nginx.default.svc.cluster.local. This name will resolve to the Service's ClusterIP address.

Important:

• If a Pod is trying to access a Service in the same namespace , it can simply use the Service name (e.g. my-nginx). Kubernetes DNS will automatically fill in the rest (.default.svc.cluster.local) based on the search configuration in /etc/resolv.conf.
• If a Pod needs to access a Service in another namespace , it needs to use a name of the form <service-name>.<namespace-name> (for example, database-service.prod).

3.2. DNS records for Pods:

Kubernetes can also create DNS records for each Pod, although these are less directly used for service-to-service communication. The format is typically:

<pod-ip-address-dashed>.<namespace-name>.pod.<cluster-domain>

For example: A Pod with IP 10.244.1.10 in the dev namespace will have a DNS name of 10-244-1-10.dev.pod.cluster.local. This name resolves directly to the IP of that Pod.

3.3. Headless Services and SRV Records:

When you create a Headless Service (by setting clusterIP: None), K8s DNS will not create an A record for that Service itself. Instead:

• If the Service has selectors, DNS will create A records for each Pod that the Service manages, with the format: <pod-hostname>.<service-name>.<namespace-name>.svc.<cluster-domain>.
• It also creates an SRV record for the Service name (<service-name>.<namespace-name>.svc.<cluster-domain>) that points to the A records of the Pods. SRV records are often used for applications that need peer discovery (like database clusters).

3.4. How do Pods get DNS configuration?

When a Pod is created, the kubelet on the Node will configure the /etc/resolv.conf file inside that Pod's container. This file typically looks like this:

nameserver <coredns-cluster-ip> search <namespace>.svc.<cluster-domain> svc.<cluster-domain> <cluster-domain> [custom-search-paths] options ndots:5

• nameserver: Specifies the IP address of the K8s DNS server (CoreDNS service). Any DNS queries from the Pod will be sent to this address first.
• search: List of domain suffixes that will be automatically attempted when you query a short name (without a dot). For example, if you query my-service from a Pod in the my-app namespace, the DNS client will try the following names (in order):
  1. my-service.my-app.svc.cluster.local (find Service in same namespace)
  2. my-service.svc.cluster.local (find Services in other namespaces - rarely used)
  3. my-service.cluster.local
  4. my-service.[custom-search-paths] (if any)
  5. my-service (query root name - usually for external domains)
• options ndots:5: Specifies that if a domain name has less than 5 dots, it will be treated as a short name and suffixes in search will be tried before querying the root name. This helps optimize internal name resolution.

Part 4: CoreDNS - Kubernetes' Default DNS Server

From Kubernetes v1.13 onwards, CoreDNS has become the default DNS server, replacing the previous kube-dns.

What is CoreDNS?

CoreDNS is a flexible, highly extensible DNS server written in Go. CoreDNS's biggest strength is its plugin- based architecture. All of CoreDNS' functionality is implemented by plugins, and you can easily add, remove, or reconfigure these plugins to customize the DNS server's behavior.

Why choose CoreDNS over kube-dns?

• Flexible and Extensible: The plugin architecture allows for much more customization than kube-dns. You can easily integrate features like advanced logging, metrics, query rewriting, domain blocking, etc.
• Fewer components: CoreDNS runs as a single process, while kube-dns requires 3 containers (dnsmasq, sidecar, kubedns). This reduces the attack surface and simplifies management.
• Performance and Memory: Generally considered to have better performance and use less memory than kube-dns in many cases.
• Community and Development: CoreDNS is a CNCF (Cloud Native Computing Foundation) project with an active development community.

CoreDNS in Kubernetes:

In K8s, CoreDNS is typically deployed as a Deployment in the kube-system namespace. It has a corresponding Service (usually named kube-dns, for backwards compatibility reasons) with a fixed ClusterIP. This is the IP address written to the Pods' /etc/resolv.conf file.

CoreDNS continuously watches the Kubernetes API Server for information about newly created, updated, or deleted Services and Endpoints. Based on this information, it answers DNS queries for the cluster's internal domain names.

Part 5: Inside CoreDNS - Configuration (Corefile)

CoreDNS behavior is controlled by a main configuration file called the Corefile . In Kubernetes, this Corefile is typically stored in a ConfigMap named coredns in the kube-system namespace.

Corefile has a fairly simple structure:

.:53 { # Các plugin và cấu hình của chúng plugin1 [args...] plugin2 [args...] ... } another.zone:5353 { # Cấu hình cho zone khác plugin3 }

• Each block defines a zone that CoreDNS will serve and the port it will listen on (usually 53 for DNS). The . represents the root zone.
• Inside each block is a list of plugins enabled for that zone. The order of the plugins is important because the query will go through them sequentially.

An example default (simplified) Corefile in K8s:

.:53 { # Ghi log lỗi ra stdout errors # Cung cấp endpoint health check tại http://localhost:8080/health # Sẵn sàng khi tất cả plugin đều sẵn sàng health { lameduck 5s } # Báo cáo trạng thái sẵn sàng cho K8s (sử dụng cùng port với health) ready # Plugin chính xử lý các truy vấn DNS cho K8s cluster # Nó sẽ theo dõi các Service và Pods # pods insecure: Cho phép phân giải Pod A records (IP dạng gạch ngang) # upstream: Sử dụng /etc/resolv.conf của node để phân giải tên bên ngoài nếu plugin `forward` không được dùng # fallthrough: Nếu không tìm thấy bản ghi trong K8s, chuyển tiếp truy vấn cho plugin tiếp theo kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream fallthrough in-addr.arpa ip6.arpa } # Cung cấp metrics theo chuẩn Prometheus tại http://localhost:9153/metrics prometheus :9153 # Chuyển tiếp các truy vấn không thuộc cluster.local (và các zone đặc biệt khác) # đến các máy chủ DNS upstream được định nghĩa trong /etc/resolv.conf của Node. # policy sequential: Thử từng upstream theo thứ tự forward . /etc/resolv.conf { max_concurrent 1000 } # Bật bộ nhớ đệm DNS để tăng tốc độ và giảm tải # 30: Kích thước cache (mặc định là 10000 bản ghi) # success 9984, denial 30: Cache các câu trả lời thành công/thất bại cache 30 # Phát hiện vòng lặp DNS loop # Tải lại cấu hình Corefile nếu file thay đổi (mặc định 5s) reload 5s # Cân bằng tải giữa các kết nối đến upstream (khi dùng forward) loadbalance }

Explanation of important Plugins:

• errors: Handle and log errors.
• health & ready: Provides endpoint for K8s to check the health and readiness of the CoreDNS Pod.
• kubernetes: Core plugin, responsible for resolving K8s internal domain names (*.cluster.local). It connects to the K8s API server.
• prometheus: Export CoreDNS performance metrics so that monitoring systems like Prometheus can collect them.
• forward: Forwards DNS queries that CoreDNS cannot resolve (usually domains outside the cluster) to other DNS servers (called upstream resolvers). It usually reads the upstream list from the /etc/resolv.conf of the Node that the CoreDNS Pod is running on.
• cache: Cache recent DNS resolution results to speed up response times for repeated queries.
• loop: Prevent infinite DNS loops.
• reload: Automatically reload configuration when the ConfigMap containing the Corefile changes, without restarting the Pod.
• loadbalance: Distributes forwarded queries to different upstream DNS servers.

DNS query processing flow in CoreDNS:

1. Internal query (e.g. my-service.default.svc.cluster.local):
   • The Pod sends a query to the CoreDNS Service IP.
   • Query to CoreDNS Pod.
   • The cache plugin checks to see if the result is in the cache. If so, returns it immediately.
   • Otherwise, query the kubernetes plugin.
   • The kubernetes plugin finds information about the Service my-service in the default namespace from the data it monitors from the API Server.
   • The kubernetes plugin returns the ClusterIP of the Service.
   • The cache plugin saves this result to the cache.
   • CoreDNS returns the result to the Pod.
2. External queries (eg google.com):
   • The Pod sends a query to the CoreDNS Service IP.
   • Query to CoreDNS Pod.
   • Cache plugin checks cache. If not there.
   • The kubernetes plugin checks. Since google.com doesn't match cluster.local (or other local zones), it will "fallthrough" (forward to the next plugin).
   • Query the forward plugin.
   • The forward plugin sends the query to one of the upstream DNS servers (taken from Node's /etc/resolv.conf).
   • The upstream DNS server returns the IP address of google.com.
   • The forward plugin receives the result.
   • The cache plugin saves the results to the cache.
   • CoreDNS returns the result to the Pod.

Part 6: Troubleshooting - Check and fix basic DNS issues in K8s

DNS is one of the common causes of connectivity issues in Kubernetes. Here are the basic steps to check:

1. Check CoreDNS Pods status:

   kubectl get pods -n kube-system -l k8s-app=kube-dns # hoặc kubectl get pods -n kube-system -l app.kubernetes.io/name=CoreDNS (tùy label)

   Make sure the CoreDNS Pods are in Running state. If not, check their logs:

   kubectl logs -n kube-system <coredns-pod-name>

2. Check Service kube-dns:

   kubectl get service kube-dns -n kube-system

   Make sure the Service exists and has a ClusterIP.

3. Check /etc/resolv.conf inside the Pod that is having trouble connecting: Run a shell inside the Pod that is having trouble connecting:

   kubectl exec -it <your-pod-name> -n <your-namespace> -- sh # hoặc bash

   Then, view the file contents:

   cat /etc/resolv.conf

   • Check if the nameserver is the correct ClusterIP address of the kube-dns Service.
   • Check that search contains the correct Pod namespace and cluster suffix (svc.cluster.local, cluster.local).
4. Use nslookup or dig from within the Pod: This is the best way to test DNS resolution from the application's perspective. Install tools if needed (apt-get update && apt-get install -y dnsutils or bind-tools for Debian/Ubuntu, apk add bind-tools for Alpine).

   • Check internal Service resolution (same namespace):

     nslookup <service-name> # Ví dụ: nslookup kubernetes # (Service mặc định trong namespace default)

   • Check internal Service resolution (different namespace):

     nslookup <service-name>.<namespace-name> # Ví dụ: nslookup kube-dns.kube-system

   • Check internal Service resolution (full name):

     nslookup <service-name>.<namespace-name>.svc.cluster.local # Ví dụ: nslookup kube-dns.kube-system.svc.cluster.local

   • Check external domain name resolution:

     nslookup google.com

   • Query CoreDNS directly (bypassing local cache):

     nslookup google.com <coredns-service-ip> # Hoặc dùng dig: dig @<coredns-service-ip> google.com

5. Check Network Policies: If you are using Network Policies, make sure they are not blocking DNS traffic (usually port 53 UDP and TCP) from your Pods to CoreDNS Pods.
6. Check CoreDNS logs: As mentioned in step 1, CoreDNS logs can contain valuable information about resolution or configuration errors. You can increase the level of log detail by adding a log plugin to your Corefile (but be careful, it can generate a lot of logs).
7. Node DNS Issue: If internal resolution works but external resolution fails, the issue may lie in the DNS configuration on the Worker Nodes themselves (the /etc/resolv.conf file on the Node) that CoreDNS is using to forward queries (if the forwarding configuration is using /etc/resolv.conf).

Part 7: Customizing CoreDNS (For Intermediate)

One of the great advantages of CoreDNS is its customization. You can edit the coredns ConfigMap to:

• Changing Upstream Forwarders: Instead of using Node's /etc/resolv.conf, you can specify specific DNS servers (like 8.8.8.8, 1.1.1.1, or your company's internal DNS) in the forward plugin.

  forward . 8.8.8.8 1.1.1.1

• Add Stub Domains: Forward queries for a specific domain (e.g. *.mycompany.local) to another internal DNS server.

  mycompany.local:53 { errors cache 30 forward . 192.168.1.50 # DNS server của công ty } .:53 { # ... cấu hình còn lại ... # Đảm bảo plugin kubernetes đứng trước forward hoặc dùng fallthrough kubernetes cluster.local ... { fallthrough } forward . /etc/resolv.conf # Forward các tên miền khác nếu không khớp mycompany.local hoặc cluster.local # ... }

• Enable/Disable/Configure Plugins: You can add other plugins (e.g. rewrite to change query name, acl to control access) or change parameters of existing plugins (e.g. increase cache size).
• Blocklist/Allowlist: Use plugins like blocklist or combine with rewrite to block access to unwanted domains.

Important Note: Be careful when editing the CoreDNS ConfigMap. Incorrect configuration can break DNS functionality of the entire cluster. Always test thoroughly after making changes. The reload plugin will automatically apply the changes after a few seconds without restarting the Pod.

Conclude

DNS is an essential, often hidden, yet critically important component of any Kubernetes cluster. Understanding how DNS works, from the basics to specific implementations in K8s with CoreDNS, is key to building and operating robust applications. It provides a flexible service discovery mechanism that allows applications to reliably communicate with each other in the dynamic environment of K8s.