How I Keep Myself Alive Using Golang: Automated Incident Management for Type 1 Diabetes

How I Keep Myself Alive Using Golang: Automated Incident Management for Type 1 Diabetes

Table of Contents

How many grams of carbohydrates are in a pint of beer or a seemingly healthy salad? For most people, the answer is “who cares,” unless they are following a strict diet. But for approximately 8 million people living with Type 1 Diabetes (T1D) worldwide, this is a continuous, daily life-or-death question.

Type 1 Diabetes is an autoimmune condition where the pancreas produces little to no insulin - the essential hormone required to convert carbohydrates into energy. If blood glucose levels remain too high over long periods, it leads to severe organ damage. Conversely, if blood glucose drops too low (Hypoglycemia) even for a brief window, it can cause loss of consciousness, temporary paralysis, or death if left untreated.

This article explores the architectural solution built by Matt Boyle (Staff Engineer), who applied distributed systems design, Go microservices, Prometheus telemetry, and incident escalation workflows to automate the real-time monitoring of his own vital signs.

The Human Body as an Incident Management Problem

In system engineering, we frequently discuss 99.99% SLAs, Mean Time to Detect (MTTD), and Mean Time to Resolve (MTTR). For a T1D patient, personal health management is fundamentally a 24/7 Incident Management Loop:

  1. Detection: Continuously measure blood glucose levels in real time.
  2. Analysis: Evaluate trend direction based on active insulin, recent carbohydrate intake, and physical exercise.
  3. Mitigation: Execute emergency carbohydrate intake whenever blood glucose enters the critical zone (< 4.0 mmol/L).

Warning

The Commercial Software Failure Mode: Commercial continuous glucose monitors (CGMs) wear an arm sensor paired with a smartphone app. However, during rapid blood sugar drops (the exact moment of maximum danger), commercial apps often encounter anomaly detection errors and crash or suppress readings, leaving the patient completely unmonitored at the most critical time.

To eliminate this single point of failure, the engineering objective was clear: Reverse-engineer the sensor telemetry stream and build a custom, highly available monitoring pipeline in Golang.

Overall Telemetry & Escalation Architecture

The architecture strictly separates Data Ingestion, Observability, Alerting Engine, and Escalation Workflows.

graph TD
    subgraph "Physical Boundary (IoT Edge)"
        Sensor[Libre Arm Sensor] -->|NFC/Bluetooth| Bridge[MiaoMiao / Tomato App]
    end

    subgraph "Logical Boundary (Go Backend Core)"
        Bridge -->|HTTP POST 2-min| EchoServer[Go Echo Server / Encore.dev]
        EchoServer -->|Update Gauge| Prom[Prometheus Metrics]
        EchoServer -->|Query| Grafana[Grafana Dashboard]
        
        Cron[Go Alert Cron - 5min] -->|Check Threshold < 4.0| EchoServer
        TelegramBot[Telegram Bot - Manual Panic] -->|User Command| IncidentSvc[Incident Microservice]
        Cron -->|Trigger Incident| IncidentSvc
    end

    subgraph "External Alerting & Escalation"
        IncidentSvc -->|HTTP POST + Idempotency Key| IncIO[incident.io API]
        IncIO -->|SMS Alert| Self[Personal Phone]
        IncIO -->|Escalate after 20min no ACK| Family[Emergency Contacts / Family]
    end

Step 1: Intercepting Telemetry & Building a Go Echo Server

While exploring the configuration of the intermediary Tomato app, Matt discovered a data sync setting intended for Nightscout (an open-source monitoring tool). Instead of pointing to Nightscout, he redirected the endpoint URL to his own Go backend.

Using Encore.dev, he deployed a lightweight Go HTTP handler to capture incoming payloads:

  • Go Echo Handler
  • Sensor JSON Payload
// encore:api public raw method=POST path=/id/:id/api/v1/devicestatus
func Echo(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", req.Header.Get("Content-Type"))
    
    // Copy the request body directly to ResponseWriter for inspection
    if _, err := io.Copy(w, req.Body); err != nil {
        http.Error(w, "Failed to echo request", http.StatusInternalServerError)
        return
    }
}
{
  "date": 1696171541297,
  "sgv": 73,
  "delta": 0,
  "sysTime": 1696171541381,
  "dateString": "2023-10-01T14:45:41.297Z",
  "_id": "dOUXaI8HcaulCGrQfxe23UE0",
  "type": "sgv",
  "device": "Tomato",
  "direction": "Flat"
}

Key fields parsed:

  • sgv (Sensor Glucose Value): Raw blood sugar reading (divided by 18 in the UK to convert to mmol/L).
  • direction: Trend vector (Flat, SingleDown, FortyFiveDown).
  • Push frequency: Every 2 minutes.

Step 2: Telemetry & Observability with Prometheus & Grafana

Once validated, the Go backend updates a Prometheus Gauge metric:

var BloodSugar = metrics.NewGauge[float64](
    name: "blood_sugar",
    metrics.GaugeConfig{},
)

func ProcessReading(sgv float64) {
    mmolValue := sgv / 18.0
    BloodSugar.Set(mmolValue)
}

Visualizing readings on Grafana with annotation layers (carbs consumed, insulin doses, physical exercise) turns raw biological signals into structured, context-aware observability.

Step 3: Alert Engine & Idempotent Escalation

Observability is only half the battle. If a hypoglycemic event occurs while sleeping, the system must trigger active alerts automatically.

1. Automated Go Cron Job

A background job checks vital metrics every 5 minutes:

var _ = cron.NewJob("monitor-blood", cron.JobConfig{
    Title:    "Monitor blood to check if there is reason to open an incident",
    Every:    5 * cron.Minute,
    Endpoint: BloodIncidentCron,
})

func BloodIncidentCron(ctx context.Context) error {
    currentReading := getLatestReading()
    const BloodLowerLimit = 4.0

    if currentReading < BloodLowerLimit {
        if err := triggerIncident(ctx, currentReading); err != nil {
            return fmt.Errorf("failed to trigger incident: %w", err)
        }
    }
    return nil
}

2. Manual Panic Override via Telegram

A fallback Telegram Bot handler accepts "i need help" to trigger an incident manually if symptoms appear rapidly:

if strings.ToLower(update.Message.Text) == "i need help" {
    _, err := incident.Create(req.Context(), "low")
    if err != nil {
        rlog.Error("error creating incident", "err", err)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
    writeBackToTelegram(req.Context(), "Ack, incident opened. Godspeed.")
}

3. Incident.io Integration with Idempotency Keys

A critical architectural pattern here is using UUIDv4 Idempotency Keys on the incident.io API calls. Under unstable mobile network conditions or repeated cron retries, the system must never storm duplicate incidents.

// encore:api public path=/incident/:blood
func Create(ctx context.Context, blood string) (*Response, error) {
    idemKey, _ := uuid.NewV4()
    
    payload := Payload{
        IdempotencyKey: idemKey.String(),
        Mode:           "standard",
        Name:           fmt.Sprintf("Matt's blood sugar is currently critical: %s", blood),
        SeverityID:     "severity_critical",
        Summary:        "Matt's blood sugar dropped below 4.0 mmol/L threshold!",
        Visibility:     "public",
    }
    
    data, err := json.Marshal(payload)
    if err != nil {
        return nil, fmt.Errorf("failed to marshal incident request: %w", err)
    }

    req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.incident.io/v2/incidents", bytes.NewBuffer(data))
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", secrets.IncidentAPIKey))

    resp, err := client.Do(req)
    if err != nil {
        return nil, fmt.Errorf("failed incident.io req: %w", err)
    }
    defer resp.Body.Close()

    return &Response{Message: blood}, nil
}

Dynamic Escalation Policy: Two-Tier Fail-Safe

Using enterprise incident platforms like incident.io enables automated escalation workflows:

Tip

Escalation Policy Flow:

  1. Trigger Incident: SMS alert is dispatched immediately to Matt’s phone.
  2. ACK Timeout (20 Minutes): If Matt fails to acknowledge or resolve the incident within 20 minutes (indicating incapacitation), the system automatically escalates.
  3. Emergency Broadcast: Workflows trigger automated SMS and calls to family members with location details.

System Design Takeaways for Real Life

By combining backend engineering principles with IoT health telemetry, Matt created a life-saving fail-safe while gathering long-term incident telemetry. Analyzing incident rate trends enables data-driven medical consultations to refine insulin treatment regimens.

Engineering capabilities extend far beyond building enterprise software or tuning database queries. Applied purposefully, software engineering serves as a powerful tool for real-world risk management.

Share :

Related Posts

When to use cache and when not to

When to use cache and when not to

There is a production bug that I still remember vividly: user A cancels an order, but the app still displays “delivering” for the next 10 minutes. Support receives 30 tickets in one morning. The reason: cache TTL is 10 minutes, but no one invalidates it when the order status changes.

Read More
When an Autonomous AI Agent Left Its Owner a $6,500 AWS Bill Scanning DN42

When an Autonomous AI Agent Left Its Owner a $6,500 AWS Bill Scanning DN42

Over the past few days, the DevOps and networking communities have been buzzing about a rather tragicomic incident. An operator confidently gave an autonomous AI agent an AWS API key with full administrator access. The mission seemed simple: join and scan DN42. A few days later, the operator woke up to a $6,531.30 AWS invoice, declared bankruptcy, and took to the DN42 mailing lists and Matrix channels begging for cryptocurrency donations.

Read More
Golang disables Nagle's Algorithm by default: A performance trick on unstable networks

Golang disables Nagle's Algorithm by default: A performance trick on unstable networks

Hello everyone, today I want to share with you a funny and sad story that I and many of my fellow system engineers have experienced.

Read More