Blog Posts

Home /
Blog Posts

Graceful Shutdown: Benefits and Reasons to Have It

Golang

There was a time when our team deployed a new version at 11 pm. After running kubectl rollout restart, 30 seconds later, PagerDuty alerted: 200 502 errors in 5 seconds. Customers were placing orders when they encountered a timeout. We quickly rolled back, then checked the logs - it turned out that the old pod was terminated with SIGTERM, the HTTP server shut down immediately, and 50 requests being processed were cut off entirely.

Goroutine management: errgroup, leak-proof, backpressure

Golang

Last month, one of our team’s services suddenly slowed down like a turtle. The p50 latency jumped from 50ms to 3 seconds. Upon checking Grafana, there were 120,000 goroutines running - normally, there are only 200. Someone had just pushed code and forgotten to call defer cancel() in a batch processing loop.

When to use cache and when not to

Backend , Golang , System design

There is a production bug that I still remember vividly: user A cancels an order, but the app still displays “delivering” for the next 10 minutes. Support receives 30 tickets in one morning. The reason: cache TTL is 10 minutes, but no one invalidates it when the order status changes.

API Filtering: retrieving data like a coffee connoisseur

Backend , Golang

In my first year of working, I once wrote an endpoint GET /api/menus that returned… the entire menu. 200 items every time it was called. The JSON was 1.2MB heavy. The frontend only needed the name and price of 10 active dishes. I remember the first thing my lead said: “You’re sending the entire warehouse to someone who just needs to view the menu, aren’t you?”

Context in Go: passing data, deadline, cancel - use correctly or die of performance

Golang

When I first coded in Go, I encountered a strange production bug: after running for 3-4 days, the service suddenly consumed 8GB of RAM and caused an OOM error. I spent the entire Friday afternoon debugging and finally discovered the issue: a goroutine never ended because I forgot to pass the context to the gRPC call. Each request leaked a goroutine, and after 100K requests, the server was overwhelmed.