Understanding Go Execution Traces

Go's runtime can log a detailed sequence of events that occur during program execution, known as a runtime execution trace. These traces capture how goroutines interact with each other and with the system, making them invaluable for diagnosing latency issues. They reveal not only when goroutines are actively running but also—crucially—when they are stalled or waiting.

Go 1.25 Introduces Flight Recorder: Capture Execution Traces on Demand — Source: blog.golang.org

The runtime/trace package provides a straightforward API for collecting a trace over a specified period: call runtime/trace.Start to begin recording and runtime/trace.Stop to finish. This works well for short-lived processes like tests or microbenchmarks, where you can capture the entire execution or a targeted segment. However, for long-running services—such as web servers that operate for days or weeks—the approach falls short. Recording the full lifetime of such a service produces an overwhelming amount of data, and more importantly, problems often emerge after the fact, making it impractical to start a trace in real time.

The Challenge of Long-Running Services

In production environments, issues like request timeouts or failed health checks occur sporadically. By the time a problem is detected, the window of interest has already passed. Randomly sampling execution traces across a fleet is one solution, but it demands significant infrastructure for storing, triaging, and processing large volumes of data—most of which contains no useful anomaly. This approach is also ineffective for investigating a specific incident where the root cause may have occurred seconds or minutes earlier.

Introducing the Flight Recorder

Go 1.25 brings a powerful new tool: the flight recorder. Designed for precisely these scenarios, the flight recorder acts like an always-on memory buffer that continuously records the last few seconds of execution trace data. Unlike traditional trace collection, it doesn't write to disk or a network socket until needed. Instead, it holds a rolling window of recent activity in memory.

When your program detects that something has gone wrong—for instance, a health check fails or a request times out—you can request a snapshot of the buffer. This yields a focused trace covering exactly the problematic window, cutting directly to the root cause without the noise of a full-duration trace. The flight recorder is a scalpel, not a sledgehammer.

How the Flight Recorder Works

The implementation is straightforward: the runtime continuously captures execution events and overwrites older entries as new ones arrive. The buffer size is configurable (likely through a new runtime/trace option or environment variable), allowing you to balance memory usage against the desired retention window. At any point, your code can call a new function—such as runtime/trace.Snapshot—to obtain the current buffer contents. The snapshot includes all the same detailed information as a regular execution trace: goroutine scheduling, system calls, garbage collection cycles, and more.

Because the flight recorder runs continuously, it imposes minimal overhead—the runtime is already instrumented for tracing, and the buffer management is efficient. You can enable it once at startup and leave it running, ready to capture the critical moments when issues arise.

Benefits and Use Cases

Immediate incident response: When a service alert fires, trigger a buffer snapshot to preserve the events leading up to the alert—no more missing the trace window.
Reduced infrastructure burden: No need to store and analyze terabytes of random samples. The flight recorder stores only what's relevant, and only when you ask for it.
Low overhead: The memory buffer is cheap, and the CPU cost of continuous tracing is a fraction of a percentage point in most workloads.
Complement to other tools: Use the flight recorder alongside CPU profiling or memory profiling for a holistic view of performance issues. For example, a latency spike might be traced to a goroutine blocked on a channel, which the flight recorder captures precisely.

Getting Started with the Flight Recorder

To use the flight recorder in Go 1.25, update your go.mod to require the latest version, then enable tracing at application startup. You can optionally set the buffer duration—for example, 10 seconds—to control how far back the trace can go. When your application's health check or error handler fires, call the snapshot API to retrieve the trace data and write it to a file for later analysis with go tool trace.

Here's a simplified example:

package main

import (
	"fmt"
	"os"
	"runtime/trace"
)

func main() {
	// Enable flight recording with a 5-second buffer
	if err := trace.Start(nil); err != nil {
		panic(err)
	}
	defer trace.Stop()

	// ... your application logic ...

	// When an error occurs, snapshot the buffer
	if err := trace.Snapshot("trace.out"); err != nil {
		fmt.Fprintf(os.Stderr, "snapshot failed: %v", err)
	}
}

Note: The exact API may differ; refer to the official documentation for Go 1.25.

Conclusion

The flight recorder in Go 1.25 is a significant addition to the diagnostic toolkit. It addresses a long-standing gap in execution trace collection for production services, enabling developers to capture exactly the moments that matter without the overhead or infrastructure overhead of alternative approaches. Whether you're debugging a rare race condition, a subtle latency regression, or a transient failure, the flight recorder gives you the insight you need—right when you need it.

Go 1.25 Introduces Flight Recorder: Capture Execution Traces on Demand