Graceful Shutdown: Benefits and Reasons to Have It
There was a time when our team deployed a new version at 11 pm. After running kubectl rollout restart, 30 seconds later, PagerDuty alerted: 200 502 errors in 5 seconds. Customers were placing orders when they encountered a timeout. We quickly rolled back, then checked the logs - it turned out that the old pod was terminated with SIGTERM, the HTTP server shut down immediately, and 50 requests being processed were cut off entirely.
Read More
Goroutine management: errgroup, leak-proof, backpressure
Last month, one of our team’s services suddenly slowed down like a turtle. The p50 latency jumped from 50ms to 3 seconds. Upon checking Grafana, there were 120,000 goroutines running - normally, there are only 200. Someone had just pushed code and forgotten to call defer cancel() in a batch processing loop.
Read More
API Filtering: retrieving data like a coffee connoisseur
In my first year of working, I once wrote an endpoint GET /api/menus that returned… the entire menu. 200 items every time it was called. The JSON was 1.2MB heavy. The frontend only needed the name and price of 10 active dishes. I remember the first thing my lead said: “You’re sending the entire warehouse to someone who just needs to view the menu, aren’t you?”
Read More
Context in Go: passing data, deadline, cancel - use correctly or die of performance
When I first coded in Go, I encountered a strange production bug: after running for 3-4 days, the service suddenly consumed 8GB of RAM and caused an OOM error. I spent the entire Friday afternoon debugging and finally discovered the issue: a goroutine never ended because I forgot to pass the context to the gRPC call. Each request leaked a goroutine, and after 100K requests, the server was overwhelmed.
Read More