Golang disables Nagle's Algorithm by default: A performance trick on unstable networks

Hello everyone, today I want to share with you a funny and sad story that I and many of my fellow system engineers have experienced.

Imagine you’ve just completed a backend application written in Go. You test it on localhost: smooth as silk. You deploy it to the staging environment, then to production on AWS/Google Cloud: record-low latency, extremely impressive throughput. Everything is perfect until the application is released to end-users.

One day, customers using the application from areas with unstable 3G/4G mobile networks, congested home networks, or satellite connections like Starlink start complaining. The application responds slowly, connections are constantly congested, network bandwidth skyrockets abnormally despite minimal actual data being sent, and occasionally TCP connections are abruptly terminated mid-stream.

At this point, you check the server logs on the cloud, and everything looks fine. You start suspecting that the customer’s network provider is the issue. But in reality, the culprit lies in a default setting of the standard Golang library (net package) that few people notice: Golang defaults to disabling the Nagle algorithm on all TCP connections.

Let’s dive into the system principles to understand why this setting is a “performance trap” on low-quality networks, and how we can easily avoid it.

Back to Basics: Nagle’s Algorithm and Delayed ACK

To understand this trap, we need to go back in time to the early days of the Internet and explore two classic TCP optimization mechanisms: Nagle’s Algorithm and Delayed ACK.

1. What is Nagle’s Algorithm?

Proposed by John Nagle in 1984 in RFC 896, Nagle’s algorithm was designed to solve the network congestion problem caused by the “silly window syndrome”.

Imagine you’re typing a command in an SSH session. Each key you press is equivalent to 1 byte of data. If the operating system immediately packages and sends this 1 byte over the network, the TCP packet would require:

• 20 bytes for the TCP Header.
• 20 bytes for the IP Header.
• 1 byte of actual data (payload).

In total, you’d spend 41 bytes of transmitted data just to send 1 useful byte. The overhead here is 4000%! If thousands of users were doing this, the internet would quickly become overloaded with millions of tiny packets (called tinygrams).

Nagle’s algorithm solves this problem with a simple rule:

• If the data to be sent is large enough to fill the MSS (Maximum Segment Size, usually 1460 bytes on Ethernet networks), send it immediately.
• If the data is smaller than MSS, the operating system checks if there are any previously sent packets that have not been acknowledged (ACK) by the receiver.
  • If not, send the small packet immediately.
  • If yes, the operating system will hold the new data in the send buffer to accumulate more data from subsequent write commands, until it receives the ACK of the previous packet or the buffer accumulates enough data to reach the MSS size, then send it.

In simple terms, Nagle helps to combine small packets into larger ones to save bandwidth and reduce the number of packets circulating on the network.

2. What is Delayed ACK?

In contrast to the sender using Nagle, the receiver uses a optimization technique called Delayed ACK.

Instead of sending an ACK packet immediately for each received data packet (which would waste bandwidth), the receiver delays sending the ACK for a short period (up to 200ms, typically 40ms by default on Linux).

The purpose of this delay is to:

• Wait to see if the receiving application has any data to send back to the sender, to piggyback the ACK packet with the response data.
• Accumulate ACKs for multiple consecutive received packets to send only one ACK packet that acknowledges all of them.

3. The Deadly Combo

When Nagle’s algorithm (on the sender side) and Delayed ACK (on the receiver side) work together, we encounter a disaster in terms of latency for applications that send and receive small amounts of data continuously.

Let’s look at the following scenario:

1. The sender wants to send two small messages consecutively (e.g., packet_A 10 bytes and packet_B 10 bytes).
2. The sender calls the write command for packet_A. Since there are no other packets waiting for ACK on the transmission, the operating system sends packet_A immediately.
3. The receiver receives packet_A. The Delayed ACK mechanism kicks in: the receiver does not send an ACK immediately but waits to see if there is response data or waits for the next packet to acknowledge together.
4. The sender continues to call the write command for packet_B. But at this point, packet_A has not been ACKed. Because Nagle is enabled, the sender’s operating system must hold packet_B in the send buffer waiting for the ACK of packet_A.
5. A deadlock occurs: the sender waits for the ACK to send the next packet, and the receiver waits for the next packet to send a combined ACK.
6. The deadlock is only resolved when the Delayed ACK timeout at the receiver expires (typically 40ms to 200ms). The receiver sends a separate ACK packet for packet_A. The sender receives the ACK, and only then releases packet_B and sends it.

The consequence is that your application suddenly experiences an additional latency of 40ms to 200ms for each round of sending small data. For real-time systems or web applications that require fast responses, this latency is unacceptable.

Golang’s Default Decision: Disabling Nagle

To thoroughly resolve the 200ms latency issue caused by the combination of Nagle and Delayed ACK, most distributed systems, databases (such as Redis, Cassandra), and modern web servers choose the solution: Disabling the Nagle algorithm.

In the socket configuration of the operating system, disabling Nagle is done by setting the TCP_NODELAY = true flag. When this flag is enabled, the operating system will send any data written to the socket immediately, without considering whether the data is small or not, or whether there are any packets waiting for ACK.

Golang is no exception. From its early versions, the engineers developing the Go standard library decided to default to TCP_NODELAY = true for all TCP connections created by the net package.

If you look at the source code of the net library in Go (specifically the net/dial.go file), you will see this default setting:

// A symbolic part of the code in the Go net package
func (d *Dialer) DialContext(ctx context.Context, network, address string) (Conn, error) {
    // ... connection setup ...
    if tc, ok := c.(*TCPConn); ok {
        // Default to enabling TCP_NODELAY (i.e., disabling Nagle)
        tc.SetNoDelay(true)
        // Default to enabling KeepAlive
        tc.SetKeepAlive(true)
    }
    return c, nil
}

This decision is entirely reasonable in the context of microservices or API servers communicating with each other within the same datacenter (internal LAN). There, bandwidth is virtually unlimited, latency is extremely low, and we want data to be transmitted as quickly as possible without being bottlenecked by Delayed ACK.

However, this is a highly dangerous trap (footgun) when your application communicates directly with end-users over the public internet, especially on unstable mobile or satellite networks.

Hidden Footgun: When Lack of Buffer Meets TCP_NODELAY

The trap appears when we combine two factors:

1. TCP_NODELAY = true (send data immediately for each Write command).
2. In Golang, net.TCPConn does not have a implicit buffer in user-space. Each Write() command you call on the connection will perform a syscall write directly writing data into the operating system’s socket.

Imagine you write a piece of code to send JSON data over a TCP connection like this:

// Send directly into the connection without a buffer
err := json.NewEncoder(conn).Encode(data)

You think that the Encode function will convert the entire struct into a single JSON string and then write it once into the connection? Not at all.

The json.NewEncoder function in Go is designed to optimize memory. Instead of allocating a large buffer in RAM to contain the entire serialized JSON string and then writing it, it will serialize directly into small parts (opening curly bracket {, attribute name "id", colon :, value, comma ,…) and call the conn.Write() command continuously for each small part.

What’s the result? Because TCP_NODELAY is enabled by default, each small write command (even if it’s only 1-2 bytes) from json.NewEncoder will immediately trigger a syscall write to the kernel, the operating system will immediately package this small part into a full TCP/IP packet (including 40 bytes of header) and push it straight to the network.

If you send a small JSON struct of about 100 bytes, json.NewEncoder may perform 5 to 10 small write commands. On the network, this is equivalent to sending 5 to 10 separate TCP packets.

Let’s do a simple calculation:

• Actual data: 100 bytes.
• Number of packets: 10 packets.
• Header overhead: 10 packets * 40 bytes/packet = 4000 bytes header!
• Overhead ratio: 400% of bandwidth wasted just to carry 100 bytes of data.

On high-quality networks like fiber optic or datacenter internal networks, this may not cause significant problems because the hardware can handle it. But on “shitty networks” (unstable 3G/4G mobile networks, Starlink satellite networks with high variable latency and random packet loss):

1. Bufferbloat: Millions of small packets flood into the queue of low-quality modems/routers. The device cannot handle it, leading to queue overflow, increased network latency, and packet loss.
2. Severe network congestion: When packet loss occurs, TCP is forced to retransmit packets. The continuous retransmission of numerous small packets makes the network bandwidth completely congested.
3. CPU waste: The continuous calling of syscall write dozens of times for a single data sending task wastes CPU resources of both the client and server unnecessarily.

The final result is that users will see the application as extremely slow, with unstable connections and frequent disconnections.

Practical Example: Code Illustration and Solution

To see the difference clearly, let’s write a small example simulating this issue in Go.

1. Analyzing the Pitfall of Direct Writing (No Buffer)

Below is a simple TCP Client that sends JSON data directly into the connection:

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"net"
	"time"
)

type UserInfo struct {
	ID        int       `json:"id"`
	Name      string    `json:"name"`
	Email     string    `json:"email"`
	CreatedAt time.Time `json:"created_at"`
}

func main() {
	// Connect to TCP Server
	conn, err := net.Dial("tcp", "localhost:8080")
	if err != nil {
		log.Fatalf("Failed to connect: %v", err)
	}
	defer conn.Close()

	user := UserInfo{
		ID:        999,
		Name:      "Luan Dinh",
		Email:     "[email protected]",
		CreatedAt: time.Now(),
	}

	fmt.Println("Starting to send data without buffer...")
	
	// THE PITFALL: json.NewEncoder writes directly into the socket connection.
	// Each small JSON token will trigger a syscall and be sent immediately due to TCP_NODELAY being enabled by default.
	encoder := json.NewEncoder(conn)
	err = encoder.Encode(user)
	if err != nil {
		log.Fatalf("Send error: %v", err)
	}

	fmt.Println("Send completed!")
}

If you use a packet monitoring tool like Wireshark or tcpdump to capture packets transmitted through port 8080 when running the code above, you will be surprised to see the number of packets generated for this single struct is much larger than you think. Each packet carries a tiny payload with a bulky overhead header.

2. The Solution: bufio.NewWriter

The solution to this problem is extremely simple and doesn’t require more than 2 lines of your code: Using a buffer at the application layer (user-space buffer).

Go provides us with the bufio package for this purpose. Instead of writing directly into net.Conn, we wrap the connection with a bufio.Writer.

package main

import (
	"bufio"
	"encoding/json"
	"fmt"
	"log"
	"net"
	"time"
)

type UserInfo struct {
	ID        int       `json:"id"`
	Name      string    `json:"name"`
	Email     string    `json:"email"`
	CreatedAt time.Time `json:"created_at"`
}

func main() {
	// Connect to TCP Server
	conn, err := net.Dial("tcp", "localhost:8080")
	if err != nil {
		log.Fatalf("Failed to connect: %v", err)
	}
	defer conn.Close()

	user := UserInfo{
		ID:        999,
		Name:      "Luan Dinh",
		Email:     "[email protected]",
		CreatedAt: time.Now(),
	}

	fmt.Println("Starting to send data WITH buffer...")

	// SOLUTION: Wrap connection with bufio.Writer.
	// By default, bufio.Writer will allocate a 4096-byte buffer in RAM.
	writer := bufio.NewWriter(conn)

	// json.NewEncoder now writes into the RAM buffer managed by Go instead of directly into the socket.
	encoder := json.NewEncoder(writer)
	err = encoder.Encode(user)
	if err != nil {
		log.Fatalf("Send error: %v", err)
	}

	// VERY IMPORTANT: You must call Flush() to push all remaining data in the buffer 
	// down to the socket of the operating system. If you don't call Flush, data may get stuck in the buffer.
	err = writer.Flush()
	if err != nil {
		log.Fatalf("Flush error: %v", err)
	}

	fmt.Println("Send completed and Flush successful!")
}

When using bufio.Writer:

1. Small write commands from json.NewEncoder will only write into the RAM managed by the Go application.
2. Data will be accumulated until the buffer is full (default is 4096 bytes, or you can customize the size with bufio.NewWriterSize).
3. When calling writer.Flush(), all accumulated data will be sent down to the socket of the operating system through a single write syscall.
4. The operating system now receives a contiguous block of data, packages it into a single large TCP packet (or a few large MSS packets), and transmits it over the network.

The difference is impressive: The number of packets transmitted over the network decreases significantly, header overhead is minimized, reducing the load on network routers, reducing packet loss, and making the application operate much more stably on unstable connections.

Practical Advice for Backend Engineers

From the above story, we can draw out some system design rules when working with Golang:

1. Understand the deployment environment:

   • If your application is a microservice running in a closed datacenter environment (like K8s on AWS), setting the default TCP_NODELAY = true is perfect. You don’t need to change anything, as it optimizes latency extremely well.
   • If your application is a client that connects directly to end-user devices (IoT Gateway, Mobile App Backend, Game Server, VPN Server) over the public internet, be extremely cautious.

2. Always use buffering (bufio) when writing structured data:

   • Whenever you write data through encoders like json.NewEncoder, xml.NewEncoder, or when you perform multiple small write operations consecutively on a socket, wrap the connection with bufio.NewWriter.
   • Make it a habit to use the defer structure or strict error handling to ensure the Flush() function is always called before the connection is closed.

3. Customize the buffer size appropriately:

   • bufio.NewWriter defaults to using a 4KB buffer. If your data is significantly larger or smaller, consider using bufio.NewWriterSize to optimize memory allocation in RAM according to your application’s payload characteristics.

Conclusion

Disabling the Nagle algorithm by default in Golang is a smart move to optimize latency for the cloud and microservices era. However, it also brings a silent side effect if we lack buffering at the application layer when transmitting data over an unstable internet connection.

Hope this article helps you understand more about the operation of TCP, the Nagle algorithm, and how Golang handles socket connections under the hood to avoid unnecessary performance issues in real-world projects.

Have you ever encountered performance issues related to TCP_NODELAY or unstable networks when writing Go? Share your experience in the comments section below!