Graceful Shutdown: Benefits and Reasons to Have It

There was a time when our team deployed a new version at 11 pm. After running kubectl rollout restart, 30 seconds later, PagerDuty alerted: 200 502 errors in 5 seconds. Customers were placing orders when they encountered a timeout. We quickly rolled back, then checked the logs - it turned out that the old pod was terminated with SIGTERM, the HTTP server shut down immediately, and 50 requests being processed were cut off entirely.

The error was not with Kubernetes. The error was with our service: it didn’t have a graceful shutdown.

After that night, I spent 2 days implementing a graceful shutdown for the entire service. Below is the pattern I used, along with runnable Go code and the things Kubernetes needs to sync.

What happens when you kill -9?

Before diving into the code, let’s understand the death of a process:

1. Request being processed is interrupted → client receives 502/503, data inconsistency
2. DB/Redis connections are not closed → connection leak, pool exhausted
3. Message queue is abandoned → message loss or duplication
4. LB hasn’t updated yet → traffic still being sent to the dead pod

In simple terms: a graceful shutdown is like closing a coffee shop - not accepting new customers, serving the last cup, washing the cup, turning off the machine, and then locking the door. While kill -9 is like turning off the lights and running away.

Deployment in Go: from basics to production

Step 1: Signal handling

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    srv := &http.Server{
        Addr:    ":8080",
        Handler: router,
    }

    // Start server in goroutine
    go func() {
        log.Println("Server starting on :8080")
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("listen: %v", err)
        }
    }()

    // Block until signal is received
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    sig := <-quit
    log.Printf("Received signal: %v. Shutting down...", sig)

    // Give server 30s to finish processing requests
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("Server forced to shutdown: %v", err)
    }

    log.Println("Server exited gracefully")
}

server.Shutdown(ctx) does 3 things: closes the listener (stops accepting new requests), waits for ongoing requests to finish, and then exits. If the timeout is exceeded, it forces a shutdown.

Step 2: Cleaning up multiple resources - in order

In reality, you don’t just have an HTTP server. You also have a database, Redis, RabbitMQ, and background workers. All of these need to be cleaned up in the correct order:

type Closer func(ctx context.Context) error

func GracefulShutdown(timeout time.Duration, closers ...Closer) {
    ctx, cancel := context.WithTimeout(context.Background(), timeout)
    defer cancel()

    for _, close := range closers {
        if err := close(ctx); err != nil {
            log.Printf("cleanup error: %v", err)
        }
    }
}

func main() {
    // ... initialize resources ...

    GracefulShutdown(30*time.Second,
        httpserver.Shutdown,    // 1. Stop accepting new requests
        worker.Stop,            // 2. Stop background jobs
        db.Close,               // 3. Close database pool
        redis.Close,            // 4. Close Redis
        mq.Close,               // 5. Close message queue
    )
}

I used to do it backwards: closing the database before the HTTP server. The result: requests being processed called the database → connection closed error → 500. The order of cleanup is more important than you think.

Integration with Kubernetes

Graceful shutdown in Go is only half the story. Kubernetes needs to be configured in sync.

PreStop hook - the key to survival

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]

preStop runs BEFORE SIGTERM arrives. Purpose: to allow LB/Ingress to update the routing table and remove the pod from the pool. No PreStop → LB still sends traffic while the pod is shutting down → 502.

terminationGracePeriodSeconds

terminationGracePeriodSeconds: 45

Must be greater than the timeout in Go code + PreStop sleep. For example:

• PreStop: 5s
• Go shutdown timeout: 30s
• → terminationGracePeriodSeconds minimum: 40s (set to 45 for buffer)

If the pod does not exit within 45s, Kubernetes force kills it with SIGKILL. If cleanup is not done in time, the error is correct.

Readiness probe - remove pod from service before shutdown

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

In Go code, when receiving SIGTERM, set readiness to false:

var isReady atomic.Bool

func readinessHandler(w http.ResponseWriter, r *http.Request) {
    if isReady.Load() {
        w.WriteHeader(http.StatusOK)
        return
    }
    w.WriteHeader(http.StatusServiceUnavailable)
}

// In signal handler:
isReady.Store(false)
time.Sleep(2 * time.Second) // wait for LB to update
srv.Shutdown(ctx)

Set readiness → false → LB stops sending traffic → 2s buffer → shutdown. This is the pattern used for all production services.

Checklist for Graceful Shutdown

• Capture SIGTERM/SIGINT
• Call server.Shutdown(ctx) with timeout
• Clean up resources in the following order: HTTP → workers → DB → Redis → MQ
• Kubernetes: PreStop hook ≥ 5s
• Kubernetes: terminationGracePeriodSeconds > shutdown timeout
• Readiness probe fails when shutting down
• Log all received signals and each cleanup step
• Use atomic.Bool for the readiness flag, do not use mutex (not necessary)

Bonus: Verify Graceful Shutdown

# Start service
go run main.go &

# Send continuous requests
for i in $(seq 1 100); do
  curl -s http://localhost:8080/api/health &
done

# Send SIGTERM
kill -TERM $(pgrep main)

# Check: no connection refused or 502 errors

Bottom line

Graceful shutdown is something “nobody cares about until 200 502 errors hit you in the face at 11pm.” It takes 30 minutes to implement, saving dozens of hours of debugging incidents. If your service doesn’t have it - stop reading this blog, go implement it now. Come back here and read on afterwards.

Have you ever encountered an incident due to a lack of graceful shutdown? Do you have any better patterns that I haven’t tried? Tell me about your disastrous deployment story - lessons from incidents are always remembered the most. 🛑