The code I show here is from the Go repository and is copyrighted by the Go authors.

Sometimes your program needs to do something periodically, like flush a buffer, send pending log messages or accumulated metrics somewhere, etc. The simplest way to accomplish that is to start a goroutine. That goroutine has a time.Ticker, a second channel to signal that the goroutine should stop, and selects on the two in a loop. But if you don't stop that goroutine, it'll hold on to the buffer or other memory indefinitely. The idea in this section is this: schedule a single flush, and only when new data is added.

I first saw this pattern in the net/http/reverseproxy package. The reverse proxy buffers writes for efficiency. To avoid potentially unbounded latency if the buffer doesn't fill up, the reverse proxy periodically flushes the buffer. The implementation used to look like this:

type maxLatencyWriter struct {
	dst     writeFlusher
	latency time.Duration

	mu   sync.Mutex // protects Write + Flush
	done chan bool
}

func (m *maxLatencyWriter) Write(p []byte) (int, error) {
	m.mu.Lock()
	defer m.mu.Unlock()
	return m.dst.Write(p)
}

func (m *maxLatencyWriter) flushLoop() {
	t := time.NewTicker(m.latency)
	defer t.Stop()
	for {
		select {
		case <-m.done:
			if onExitFlushLoop != nil {
				onExitFlushLoop()
			}
			return
		case <-t.C:
			m.mu.Lock()
			m.dst.Flush()
			m.mu.Unlock()
		}
	}
}

func (m *maxLatencyWriter) stop() { m.done <- true }

One big flaw here is that if you forget to defer m.stop(), then the goroutine running m.flushLoop will never exit. That goroutine is going to hold on to the backing buffer for the writer. If such a goroutine is started for every HTTP request the proxy receives, the program will eventually run out of memory.

CL 137335 from Brad Fitzpatrick changed the implementation. There were a few motivations for that change, but one benefit of the new implementation is getting rid of that goroutine. Here's the new implementation:

type maxLatencyWriter struct {
	dst     io.Writer
	flush   func() error
	latency time.Duration // non-zero; negative means to flush immediately

	mu           sync.Mutex // protects t, flushPending, and dst.Flush
	t            *time.Timer
	flushPending bool
}

func (m *maxLatencyWriter) Write(p []byte) (n int, err error) {
	m.mu.Lock()
	defer m.mu.Unlock()
	n, err = m.dst.Write(p)
	if m.latency < 0 {
		m.flush()
		return
	}
	if m.flushPending {
		return
	}
	if m.t == nil {
		m.t = time.AfterFunc(m.latency, m.delayedFlush)
	} else {
		m.t.Reset(m.latency)
	}
	m.flushPending = true
	return
}

func (m *maxLatencyWriter) delayedFlush() {
	m.mu.Lock()
	defer m.mu.Unlock()
	if !m.flushPending { // if stop was called but AfterFunc already started this goroutine
		return
	}
	m.flush()
	m.flushPending = false
}

func (m *maxLatencyWriter) stop() {
	m.mu.Lock()
	defer m.mu.Unlock()
	m.flushPending = false
	if m.t != nil {
		m.t.Stop()
	}
}

The writer has a timer. When bytes are written, the timer is set to flush later based on the desired max latency. The timer is reset if it was already scheduled, so the writer will only flush if there are no more write calls for a long enough time. If and when the timer fires, the buffer is flushed one time and then that's it. That flush happens in a new goroutine created when the timer fires, but we know it just does one thing and thus we know the goroutine won't leak.^[1] Nice!

In the reverse proxy, the code still needs to ensure that stop is called. This is because the writer is going to be used later without synchronization, and thus the code needs to ensure that the flush scheduled by the timer doesn't touch it. However, even if the writer was safe to use concurrently, we're still better off because we've avoided the goroutine leak.

The second example comes from the Go toolchain. The toolchain has lots of cases where it can parallelize work. It's not necessarily good to start a goroutine per work item, though. For one, if there are lots of them (like one per package) and they each use a decent amount of memory, the program can run out of memory. And if the work items are compute heavy, there's no benefit do doing more than "the number of processors" work items concurrently. The typical approach here would be to create a "pool" of workers. And in the typical implementation, workers are goroutines that receive work items from a channel. Usually the goroutines would be started up-front. Then the goroutines range over that channel and exit when the channel is closed. But if we don't close the channel, the workers never stop.

Now, unlike the reverse proxy example, I don't think I've personally seen unbounded goroutine leaks from worker pools. They're usually created once, at the start of the program. It's not the end of the world if they don't exit because they'll be around for the whole program run anyway. But I still think this next example is neat, as an example of avoiding leaks by design. Maybe you'll find a good use for the pattern.

CL 247766 from Bryan Mills introduced a Queue struct for scheduling work with bounded parallelism. The code is a little longer so I'll break it up.^[2] Here are the data structures and how they're created:

// Queue manages a set of work items to be executed in parallel. The number of
// active work items is limited, and excess items are queued sequentially.
type Queue struct {
	maxActive int
	st        chan queueState
}

type queueState struct {
	active  int // number of goroutines processing work; always nonzero when len(backlog) > 0
	backlog []func()
	idle    chan struct{} // if non-nil, closed when active becomes 0
}

// NewQueue returns a Queue that executes up to maxActive items in parallel.
//
// maxActive must be positive.
func NewQueue(maxActive int) *Queue {
	if maxActive < 1 {
		panic(fmt.Sprintf("par.NewQueue called with nonpositive limit (%d)", maxActive))
	}

	q := &Queue;{
		maxActive: maxActive,
		st:        make(chan queueState, 1),
	}
	q.st <- queueState{}
	return q
}

Work is submitted to the queue as parameterless functions to call. The functions are stored in a first-in first-out backlog, implemented using a slice here. Note that this implementation uses a buffered channel (st) as a kind of mutex to protect the queue state. Here's how work is submitted to the queue:

// Add adds f as a work item in the queue.
//
// Add returns immediately, but the queue will be marked as non-idle until after
// f (and any subsequently-added work) has completed.
func (q *Queue) Add(f func()) {
	st := <-q.st
	if st.active == q.maxActive {
		st.backlog = append(st.backlog, f)
		q.st <- st
		return
	}
	if st.active == 0 {
		// Mark q as non-idle.
		st.idle = nil
	}
	st.active++
	q.st <- st

	go func() {
		for {
			f()

			st := <-q.st
			if len(st.backlog) == 0 {
				if st.active--; st.active == 0 && st.idle != nil {
					close(st.idle)
				}
				q.st <- st
				return
			}
			f, st.backlog = st.backlog[0], st.backlog[1:]
			q.st <- st
		}
	}()
}

There's a little more going on here. Basically, if there are fewer than maxActive worker goroutines, the function starts one. Otherwise work items are added to the backlog. The worker goroutines remove items from the backlog and run them.

The exit condition is very different from the "range over a channel" implementation. With a channel, the program needs to close the channel for the goroutines to exit. This invites user error, especially if this is part of an API other people will use. With this implementation, we don't have that problem. The worker goroutines will exit as soon as the backlog is ever empty. This should happen eventually if the workers can keep up with the incoming work. And if the worker goroutines stop and there's new work later, we just start new ones. Goroutines are cheap to create. Practically speaking, I think knowing the code is free of one kind of bug is worth at least some of the cost of making new goroutines every now and then.^[3]

The thing these examples have in common is that the exit conditions for the goroutines are clear. In the first example, it's that we're only going to flush one time. In the second example, it's running out of work. There's no other function to remember to call. In both examples, we know how the goroutines will stop when we start them.

1. Unless the write somehow blocks forever... I recall having issues with network file systems in the past. Back
2. I won't show the code here, but there's also an Idle method which lets the caller synchronize on all the pending work being finished. It basically just returns the idle channel, with some logic to handle if there is no active work. Back
3. If you're going to use this pattern for a long-lived process, with unbounded amounts of work, don't copy this-as is. This implementation uses a slice for the backlog, appending new items to the end and slicing the next item off the front. This will grow the backing memory without bound. Prefer a linked list or other implementation in this case (see Wikipedia for an example.) Back

The short version:

• The most flexible option is to build a toolchain distribution from (modified) source. You'll need to do this if you're changing the compiler, linker, or other tools.
• If you're only modifying the runtime or standard library, use go-patch-overlay to apply patches at build time.

The first method is to build a custom Go toolchain, and use it to build your program. By toolchain, I mean the collection of tools (go command, compiler, linker, etc), along with the runtime and standard library sources. There are ways to change the runtime and standard library sources when you build your program. But if you want to change the tools, you'll need to build them ahead of time.

You first check out the Go source code:

git clone https://go.googlesource.com/go

If you're making a change to a specific Go release version, use git switch --detach {release tag} to check it out. Then you make the changes you want.

If you want to build the toolchain to use locally, there's a script to build it:

cd src
./make.bash

Then add /path/to/go/bin to your PATH. After that, running the go command will use the toolchain you built.

What if you want to give this toolchain to someone else, or use it in your CI? My colleague Nayef has done this at Datadog. Here's how he's done it:

You'll need a VERSION file to make a distribution:

echo go1.x.y > VERSION

Replace x and y with numbers. For example, if you were testing a fix for go1.24.4, you could do go1.24.999. That'll give you a valid version that's also unlikely to collide with a real Go version.

Then you'll actually build the distribution:

cd src/
GOOS=linux GOARCH=amd64 ./make.bash -distpack 

Set GOOS and GOARCH as needed. You'll end up with a distribution like

../pkg/distpack/go1.X.Y.linux-amd64.tar.gz

To actually use this, you'll essentially follow the steps in the Go installation docs. Copy go1.X.Y.linux-amd64.tar.gz to wherever you want to install it, and then extract it. You'll need to update the PATH environment variable to include the bin subdirectory of the resulting directory. Then when you run the go command you'll be using the custom toolchain.

If you have a CL you'd like to use (from go.dev/cl), there's a tool that can build the toolchain for you. It's called gotip. This might apply to you if you've reported a bug, and a Go maintainer has a potential fix for you to try. The gotip readme explains it, but here's a concrete example:

Say you want to try out go.dev/cl/12345. The "CL number" is 12345. You can then use gotip like so:

go install golang.org/dl/gotip@latest
gotip download 12345

This will download the Go sources from that CL number, and then build the Go toolchain from those sources. Then you can use gotip to build your program in place of the normal go command.

If you build your Go code with Bazel, then ignore the rest of this post and just refer to this section. Full disclosure: At the time of writing this post, I haven't actually used this method. I don't know if this works for compiler/linker/etc patches or just runtime/library code. But we have some Bazel stuff at Datadog so it seemed worth mentioning here. The rules_go rules support patching the tools and libraries out of the box. It's described here. Here's an example straight from the manual:

go_download_sdk(
    name = "go_sdk",
    goos = "linux",
    goarch = "amd64",
    version = "1.18.1",
    sdks = {
        "linux_amd64": ("go1.18.1.linux-amd64.tar.gz", "b3b815f47ababac13810fc6021eb73d65478e0b2db4b09d348eefad9581a2334"),
        "darwin_amd64": ("go1.18.1.darwin-amd64.tar.gz", "3703e9a0db1000f18c0c7b524f3d378aac71219b4715a6a4c5683eb639f41a4d"),
    },
    patch_strip = 1,
    patches = [
        "//patches:cgo_issue_fix.patch",
    ]
)

I've bolded the relevant part. You just provide a list of patches, and they're applied when Bazel builds and installs the toolchain.

As mentioned previously, if you want to change the compiler, linker, or other tools, you'll need to rebuild them. But if your changes are just to the runtime or standard library, you have other options. Your Go toolchain includes the runtime and standard library source code. The code for your toolchain is under the toolchain's GOROOT:

% go env GOROOT
/usr/local/go
% go1.24.4 env GOROOT
/Users/nick/sdk/go1.24.4

There are a few ways you can modify the source code. The first way is to edit it directly. You can use a text editor, or if you have a patch, you can apply it to the source code. Don't do this to your local toolchain installation! It's a pain to undo your changes. This method is suitable for doing during a build in a Docker container.

There are a few ways to get a patch:

• Somebody gives you one
• You check out the Go sources and edit them. Then run git diff > change.patch
• You get one from a CL. You can visit the CL page, where there should be a link to download the change as a patch. Or you can download the patch via the API

  CL_NUMBER=12345
  curl -XGET "https://go-review.googlesource.com/changes/go~${CL_NUMBER}/revisions/current/patch?raw" > ${CL_NUMBER}.patch
  

Assuming you have a patch, you can apply it to the Go toolchain sources like so:

pushd $(go env GOROOT)
git apply /path/to/patch1.diff /path/to/patch2.diff ...
popd
go build ...

Whatever changes you make to the sources will be reflected in the program you build. Expect your build to be a bit slower if you change widely depended-upon packages, like runtime.

Turns out there's another way to change the Go sources at build time, without needing to actually modify your installation. The go build command accepts an -overlay argument. The argument is a path to a JSON file like this:

{
  "Replace": {
    "/path/to/source1.go": "/other/path/to/replacement.go",
    "/path/to/source2.go": "",
    ...
  }
}

Basically, an overlay is a key-value map where the keys are file paths seen by the compiler, and the values are replacement files. This can even include new or deleted files. The overlay feature was originally added to support gopls, where the language server needs to tell the compiler about files whose contents haven't been written to disk. We can use it to try out patches.

There's a tool that can generate an overlay for you from patches: go-patch-overlay. My colleague Felix wrote this a few years ago. I wrote a very similar program during a recent team research week with a few fixes and extra features, before I knew about Felix's program, so we incorporated my changes into his program. You use the tool like this:

go install github.com/felixge/go-patch-overlay@main
go build -overlay=$(go-patch-overlay /path/to/patch.diff)

It works like this:

• The go-patch-overlay command reads the patch and determines which files it modifies.
• Then the tool runs go env GOROOT to find where the toolchain sources are installed. It copies any modified files into a new temporary directory.
• Then the tool applies the patch to the sources it copied from the toolchain GOROOT.
• Then the tool creates an overlay JSON file mapping the original sources to the modified sources. The tool then prints the path to the JSON file to standard out.

The file replacements in the overlay only apply to that specific go build invocation. Your toolchain's sources are unchanged. You can also pass the -overlay argument to go run and go test.

I personally like this method the best for runtime/standard library changes. It's lightweight, non-destructive, and easy to use both locally and in CI.

There's one more tool worth mentioning here: Orchestrion. Orchestrion automatically instruments Go code. It was originally built for distributed tracing, profiling, and security monitoring instrumentation. It works by rewriting source code at build time. It's a really powerful tool, kind of a low orbit ion cannon compared to the hammer of applying a patch. I won't go into the details here. If the changes you want to make are simple, I'd start with patches and go-patch-overlay. But keep Orchestrion in mind if you need something more powerful.

Don't clobber the frame pointer

Home - RSS

Posted 2024-12-24

Recently I diagnosed and fixed two frame pointer unwinding crashes in Go. The root causes were two flavors of the same problem: buggy assembly code clobbered a frame pointer. By "clobbered" I mean wrote over the value without saving & restoring it. One bug clobbered the frame pointer register. The other bug clobbered a frame pointer saved on the stack. This post explains the bugs, talks a bit about ABIs and calling conventions, and makes some recommendations for how to avoid the bugs.

Here's the short version of what you should do when writing assembly for Go to avoid the problems discussed in this post:

• First, read the Go assembly guide.
• Prefer an assembly generator like Avo. Tools like Avo are aware of the underlying rules for using registers and manipulating the stack, and generally make writing non-trivial amounts of assembly easier.
• If your assembly function calls Go functions, prefer not to use the frame pointer registers at all (BP for AMD64, R29 for ARM64). Leaving a non-frame pointer value in those registers prior to calling a Go function can crash the runtime execution tracer and profilers.
• Otherwise, if your function has a non-zero frame size, the assembler will correctly make a stack frame for you, growing the stack if needed, and properly save and restore the frame pointer. The means the frame pointer will be safe to use. Consider giving your function a small, non-zero frame size even if you don't need to if you want to use the frame pointer. ^[1]
• Otherwise, if you need to use the frame pointer and don't want to have a frame, save the original frame pointer on the stack yourself and restore it when the function returns. Take care on ARM64: Go compiled functions will save the frame pointer one word below their stack frame, so the first 8 bytes of stack frame are off-limits, and the stack pointer must be 16-byte aligned.

The first issue was reported at go.dev/issue/69629. After upgrading to Go 1.23, and building with profile-guided optimization (PGO), the block profiler consistently crashed when the program was under load. The crash happened when collecting a call stack via frame pointer unwinding. This affected the program when built for the amd64 architecture, but not when it was built for arm64. I had contributed frame pointer unwinding for the block and mutex profilers for Go 1.23, so I took a look at the issue.

I won't go into the full investigation here. There were many dead ends. Thanks to the issue reporter, Liz Fong-Jones (@lizthegrey), for being so willing to collect and share information. The key clue turned out to be the address of the faulting memory access, 0x30bc5b31, which was the same in every occurrence of the crash. The program was reading 8 bytes above a frame pointer, so the supposed frame pointer was 0x30bc5b29. In a moment of desperation (or inspiration?) I put this value into GitHub code search and turned up several Metrohash implementations.

The github.com/dgryski/go-metro implementation uses assembly. Turns out the affected program was using this library! Here's some assembly from the library:

// func Hash64(buffer_base uintptr, buffer_len int64, buffer_cap int64, seed uint64) uint64
TEXT ·Hash64(SB),4,$0-40
	MOVQ seed+24(FP), AX
	MOVQ buffer_base+0(FP), BX
	MOVQ buffer_len+8(FP), CX
	MOVQ $3603962101, DX
	IMULQ DX, AX
	MOVQ $5961697176435608501, DX
	ADDQ DX, AX
	CMPQ CX, $32
	JLT after32
	MOVQ AX, DX
	MOVQ AX, DI
	MOVQ AX, SI
	MOVQ AX, BP

The MOVQ AX, BP instruction is a problem. The BP register is used as a "frame pointer" register for AMD64. Frame pointers form a linked list used to efficiently collect backtraces, and the head of the list lives in that register. The function also has no call frame (see the $0 in the signature), so the assembler won't add any code to save and restore the frame pointer. That MOVQ instruction overwrites BP with a value used for the hash function, without saving the old value. So the old value is gone forever, and BP now holds an invalid frame pointer. Later in the function, there is a MOVQ $817650473, BP instruction. This is the last write to BP before the function returns. And 817650473 == 0x30bc5b29, our mystery value.

Because this function leaves a non-frame pointer value in the frame pointer register, any unwinding that passes through it between when when this function returns and when its caller returns (restoring the caller's caller's presumably valid frame pointer) will fail. The affected program crashed with PGO enabled because the function which called this buggy hash function was inlined, extending the scope where the frame pointer register was invalid. I reported the problem.

The assembly in the go-metro repo is not written by hand. It was generated programmatically with PeachPy. With PeachPy, you write Python scripts which generate assembly. The go-metro PeachPy code doesn't explicitly use the frame pointer register. Rather, PeachPy treats it as a general-purpose register and uses it in the generated code. So this is really a bug in PeachPy.

Despite this bug in PeachPy, writing assembly with generator tools is a really good idea. Assembly generators handle register allocation, so you can ask for registers and refer to them by a useful names, rather than having to remember that CX is your loop counter and DX is the hash seed, etc. The generator can learn which registers are special and save you from breaking things. The generator can automatically allocate the right size stack frame based on locals your functions uses. You can encapsulate repeated logic in functions used by the generator, which are way better than preprocessor macros since they actually have type checking, and they end up inlined in the assembly output so you don't pay for an actual function call.

While PeachPy doesn't appear to be actively maintained, the Avo assembly generator is. And Avo does know that the frame pointer register needs to be saved. Avo specifically targets Go assembly and only supports AMD64 as of this writing. I recommend using Avo for Go assembly when you can. I opened a PR to port go-metro's assembly generation to Avo.

Also worth noting: go vet has a check for overwriting the frame pointer register without saving it. The check unfortunately misses this bug, though. The check was written to be simple and conservative to avoid false positives, and opted to stop checking a function at the first branch. But in this case, the frame pointer is clobbered after the first branch. I opened go.dev/issue/69838 to track improving this check.

I found the next bug while looking for occurrences of the previous bug at Datadog. One of our programs was crashing with stack dumps like this:

SIGSEGV: segmentation violation
PC=0x69f0e0 m=30 sigcode=1 addr=0xc3ffa97ffc05

goroutine 0 [idle]:
        runtime.fpTracebackPCs(...)
        runtime.traceStack()
        runtime.traceLocker.stack(...)
        runtime.traceLocker.GoStop()
        runtime.traceLocker.GoPreempt(...)
        runtime.goschedImpl(...)
        runtime.gopreempt_m()
        runtime.mcall()

goroutine 321 [running]:
        runtime.asyncPreempt2()
        runtime.asyncPreempt()
        github.com/apache/arrow/go/v13/arrow/memory.memory_memset_neon()
        github.com/apache/arrow/go/v13/arrow/memory.Set(...)
[ ... ]

Seeing the execution tracer and asynchronous preemption gave me flashbacks to "the one-instruction window". The memory_memset_neon function jumped out at me. I recognized "neon" as the Neon SIMD extension for ARM. I smelled assembly and took a look at the code.

The memory_memset_neon function calls this _memset_neon function, implemented in assembly. Here's the beginning of the function:

// func _memset_neon(buf unsafe.Pointer, len, c uintptr)
TEXT ·_memset_neon(SB), $0-24

        MOVD	buf+0(FP), R0
        MOVD	len+8(FP), R1
        MOVD	c+16(FP), R2

        WORD $0xa9bf7bfd // stp    x29, x30, [sp, #-16]!
        WORD $0x8b010008 // add    x8, x0, x1
        WORD $0xeb00011f // cmp    x8, x0
        WORD $0x910003fd // mov    x29, sp
        BLS	LBB0_7
[ ... ]

This assembly is extra odd since it's (partly) automatically translated from the output of a C compiler. The WORD instructions are the bytecode for machine instructions. This is usually used for instructions the Go assembler doesn't yet support, but the assembly translator probably does it for simplicity. The original instructions are in the comments.

The first stp instruction is key here. It decrements the stack pointer by 16, then saves the current values of x29 and x30 on the stack starting at the newly decremented stack pointer. The x29 register is the frame pointer register for ARM64. So, it looks like this function saves the frame pointer as it should, right?

Turns out it's wrong in a non-obvious way. The Go compiler saves the caller's frame pointer one word below the function's stack frame. ^[2] See "the one-instruction window" for some diagrams that illustrate how the frame is laid out. So, the top word of a function's stack frame on entry actually belongs to the caller. That stp instruction is actually saving x30 (the link register) where the caller saved the frame pointer. When the function returns, it will restore its caller's frame pointer to the frame pointer register, and that frame pointer now points to an invalid frame pointer (an instruction address from x30). That invalid frame pointer will be there until the caller returns. This includes if the caller is asynchronously preempted while the execution tracer is running. When that happens, the frame pointer will be visited when collecting a backtrace, hence the crash.

I reported the bug, and sent a PR to fix it. At the time I sent the PR, I did what seemed like the least intrusive thing to fix the problem. I decremented the stack pointer a little bit more so the caller's saved frame pointer wouldn't be clobbered.

However, while writing this post I realized there are probably better ways to deal with the problem. For one, the functions don't even need stack frames (see again the $0) so the frame pointer and link register save/restore logic could just be eliminated. But it can be nice to have the frame pointer set up so that you get complete stack traces from something like Linux perf.

It's probably okay to manually create the call frame and save the registers like the code does. There will typically be room below the stack pointer in any function call ^[3], so we're unlikely to write out of bounds. But it's better as a programmer not to rely on implementation details like that. It would be just as easy to give each function a non-zero frame size, even if they don't have any locals. That way, the assembler can insert the frame pointer save and restore. If the frame size is small enough, and the compiler/assembler know that small frames are safe to allocate without stack growth, extra bounds check or growth logic can be safely eliminated.

Of course, the underlying source of the bug is the assembly translator used for that code. The assembly was translated from the output of a C compiler. The C compiler isn't going to know about the requirements for Go assembly code, though. And certainly not something like the top word of the stack frame being off-limits.

There are Apache Arrow clients for many programming languages besides just Go, and it makes sense to try to share low-level, performance-sensitive code where possible, rather than writing the same primitives multiple times. And calling assembly functions is much, much cheaper than calling C functions. So I can understand why that approach was taken. PeachPy even tried to do something similar, supporting generating assembly for several different targets from a single script. Unfortunately there doesn't seem to be a tool that supports that well right now.

These kinds of rules for assembly-level code—which registers are special, when registers need to be saved and restored, how to pass arguments to and return values from functions, the exact layout of call frames and other data—are part of an "application binary interface" (ABI). There are many ABIs, even for a single processor architecture: The System V ABI is used by Linux and some other Unix-descendent operating systems. Microsoft has its own ABI. Programming languages can also have their own ABIs, on top of the operating system ABIs.

As a Go programmer, you should refer to the Go assembly guide. This describes the stable ABI for Go assembly, which is what you should target.

If you're ever looking at assembly generated by the compiler, or hand-written assembly in the runtime, refer to the internal ABI guide.

If you're planning to write Go assembly, be aware of the details of the Go ABI, but also lean on tools like Avo to handle the subtle details for you so you can focus on the code you actually want to write.

1. With a frame pointer, the function will show up more explicitly in profiles collect with tools like Linux perf. The compiler can also omit the stack growth check for functions with very small frames, so it's cheap to declare a small frame even if you don't need it. Back
2. Contrast this with how it works on AMD64. There, the caller's frame pointer is saved at the top of the frame, via a PUSHQ BP instruction, and restored before any return via POPQ BP. This is almost the same as ARM64, in the sense that the first word of the frame is a frame pointer and the word above it is a return address. But on AMD64 that word is the caller's frame pointer, rather than the caller's caller's frame pointer, and the function only writes strictly within its own frame. Frame pointer support for ARM64 was added in 2018. At the time, the Go compiler already saved the link register (R30) at the bottom of a function's call frame. By convention, the frame pointer is stored one word below the return address. So, for simplicity, the frame pointer is saved one word below the function's stack frame. It was done this way for ARM64 for simplicity, since the link register is already at the bottom of the frame. As for why that is done, I haven't yet found any record of the rationale. In a sense it's similar to pushing the return address to the stack for AMD64 CALL. But again, it's different for ARM64 since the return address is in a register, and the previous return address goes on the stack... Back
3. Sometimes known as a "red zone". Back

The one-instruction window

Home - RSS

Posted 2024-08-07

This post is about how I investigated a frame pointer unwinding crash in the Go runtime. The bug was specific to 64-bit ARM: asynchronous preemption at a specific instruction, in certain functions, led to a kind of use-after-free bug which caused the runtime execution tracer to crash when collecting a traceback. The Go runtime is low-level, and can be intimidating and mysterious. But digging into a tricky bug is, in my experience, the best way to learn how a complex system really works. I wrote this post to remind myself how I diagnosed the bug, and to share some of the things I learned while doing so.

A few Datadog engineers reported crashes like this (lightly edited):

SIGSEGV: segmentation violation
PC=0x471c64 m=38 sigcode=1

goroutine 0 [idle]:
        runtime.fpTracebackPCs(...)
        runtime.traceStackID()
            pc=0x471c64
        runtime.traceEventLocked()
            pc=0x4712d0
        runtime.traceEvent()
            pc=0x471030
        runtime.traceGoPreempt(...)
        runtime.gopreempt_m()
            pc=0x457270
        runtime.mcall()
            pc=0x482224
        
goroutine 549321 [running]:
        runtime.asyncPreempt2()
            pc=0x44e26c
        runtime.asyncPreempt()
            pc=0x4853ec
        golang.org/x/net/http2.(*PingFrame).Header()
            pc=0x9ab404
        golang.org/x/net/http2.(*Framer).checkFrameOrder()
            pc=0x97f854
        golang.org/x/net/http2.(*Framer).ReadFrame()
            pc=0x97f5f8
        google.golang.org/grpc/internal/transport.(*http2Client).reader()

[ ... ]

I was pinged to help because my job at Datadog is to maintain our continuous profiling capability for Go. My colleague Felix Geisendörfer and I had recently worked on the Go runtime execution tracer, which is where the crash was happening. We did this work as part of building Datadog's profiling timeline feature for Go, a tool for understanding latency and finding bottlenecks. We were running execution tracing widely in production. This was the first time we had seen any production crashes from the tracer. Check out Felix's talk How To Win Frames And Influence Pointers for a lot more on our work on execution tracing. We collaborated with the Go runtime maintainers a lot for this project. Datadog supports contributing to open-source projects like Go. Datadog uses Go heavily, our customers use Go, and I personally think the Go project is awesome. I also love debugging. I'm very happy to be able to do this kind of work in service of contributing to such a great project.

The crashing function, runtime.fpTracebackPCs, was a function Felix and I had recently contributed to the Go runtime. Its purpose is to efficiently collect a traceback. It does so using a technique called frame pointer unwinding. We added it specifically for the runtime execution tracer, where traceback collection was the largest source of overhead. Frame pointer unwinding involves maintaining a linked list of return instruction addresses (PCs), updated as functions are called and return, which is traversed when a traceback is requested.

Frame pointer unwinding breaks when the linked list is incorrectly manipulated. So, my approach here was to figure out:

• Which call stack the runtime was trying to collect a traceback for
• How the functions on that call stack manipulate frame pointers
• How the call stack itself may have been manipulated, for example by the runtime scheduler or garbage collector

The crash traceback gives a pretty clear signal for which call stack we're unwinding: the crash happened during a call to runtime.traceGoPreempt, so we were tracing a "preempt" event. And on top of that, we see another goroutine with runtime.asyncPreempt on its call stack. So, it seems likely that we failed to unwind that goroutine's call stack.

As the name implies, runtime.asyncPreempt is doing "asynchronous preemption". I had to take a slight detour to figure out what that actually is. Austin Clements gave an excellent talk, Pardon the Interruption, on what asynchronous preemption is and why it exists. Basically, a goroutine is forced to yield execution so that other work can happen. "Asynchronous" means this can happen at basically any point in its execution. It's implemented using signals. A signal is delivered to a goroutine to interrupt it. Then, instead of running the next instruction, a call to runtime.asyncPreempt is simulated, as if the goroutine called the function itself. That function then yields the goroutine's execution.

I wondered where precisely the goroutine in question was preempted. The full crash traceback included instruction addresses, and I also had access to the binary of the crashing process. The function preceding runtime.asyncPreempt in the traceback is golang.org/x/net/http2.(*PingFrame).Header. I used objdump to get the instructions for that function. Here is where the preemption happened, in bold:

% objdump --disassemble-symbols='golang.org/x/net/http2.(*PingFrame).Header' ./binary

./binary:	file format elf64-littleaarch64

Disassembly of section .text:

00000000009ab3b0 :
  9ab3b0: fe 0f 1d f8  	str	x30, [sp, #-48]!
  9ab3b4: fd 83 1f f8  	stur	x29, [sp, #-8]
  9ab3b8: fd 23 00 d1  	sub	x29, sp, #8
[ ... body elided ... ]
  9ab400: ff c3 00 91  	add	sp, sp, #48
  9ab404: fd 23 00 d1   sub  x29, sp, #8 
  9ab408: c0 03 5f d6  	ret
[ ... ]

The Go internal ABI specifies how function calls and call frames work for 64-bit ARM. Here's what the code does:

• str x30, [sp, #-48]!
  • Simultaneously allocate a 48-byte stack frame and save the link register at the bottom of the new frame
• stur x29, [sp, #-8]
  • Save the caller's frame pointer register (x29 on ARM) in the word below the new stack frame
• sub x29, sp, #8
  • Update the frame pointer register to point to where the previous one was just saved

Here's a sketch of the state of the goroutine's stack at this point:

Note that frame pointers are saved below the function's call frame. So the first value is the frame pointer saved by the Header function's caller, and the "saved frame pointer" value is the one saved by the Header function itself.

Then at the end of the function, we have the following cleanup:

• add sp, sp, #48
  • Free the stack frame
• sub x29, sp, #8
  • Restore the frame pointer register to its previous value ^[1]

The function was interrupted after freeing the stack frame, but before restoring the frame pointer register. Now the call stack looks like this:

So, the frame pointer register points into now-freed stack memory. This seems problematic! What if the preemption handler uses some of that memory?

At this point I had basically completely identified the bug. I was quite lucky to have gotten so far with just a crash traceback and the binary. These kinds of bugs tend to be much more difficult to make a dent in. I could have packaged this up into a bug report and sent it to the Go team, and one of the experienced maintainers would have probably quickly found the root cause.

However, I suffered the all-too-common debugger's curse: tunnel vision. We can get fixated on a possible explanation, or a particular aspect of the bug that takes us far off the path to a solution. This curse can also be a gift: we can learn a lot in the struggle to understand something, and we can sometimes find impactful ideas that go deeper than the original problem. I learned some things in the process that will no doubt help me in the future. But I could have just as easily burned all of my energy on a dead end.

I wanted to recreate the crash reliably under a debugger. The golang.org/x/net/http2.(*PingFrame).Header showed up in basically every instance of this crash. I was fixated on this exact function getting preempted at that exact instruction. At the time I was debugging this, once I found out where the preemption was happening, I jumped straight into a debugger and tried to reproduce the crash. It wasn't until later that I more carefully considered the state of the frame pointers at that particular point in the function.

The Header function just returns a struct. Here's the source. I created a program with a basically identical struct, and a function that returns the struct. I then inspected the compiled output and confirmed the instructions were basically the same as the function from the crash. I wrote a test which calls the function in a tight loop, and then ran the test with the execution tracer enabled, hoping that eventually the function would get preempted at the same instruction. And... no crash. Here's the code:

package fpcrash

import "testing"

type BigStruct struct {
        a bool
        b, c uint8
        d, e uint32
}

//go:noinline
func MakeBigStruct() BigStruct {
        return BigStruct{}
}

var sink BigStruct

func TestCrash(t *testing.T) {
        for {
                sink = MakeBigStruct()
        }
}

I decided to try and make the preemption happen myself. For problems like this where assembly-level stuff is involved, I prefer the GNU Debugger (GDB). I learned that GDB can deliver signals to the program you're debugging. From Austin's talk, I knew that the runtime uses the SIGURG signal for asynchronous preemption. GDB can also set breakpoints at specific instruction addresses. I disassembled the function I wanted to preempt, found the specific instruction where I wanted the preemption to happen, and set a breakpoint there. When the breakpoint hit, I then delivered the SIGURG signal that triggers asynchronous preemption.

(gdb) disas 'fpcrash.MakeBigStruct' [2]
Dump of assembler code for function fpcrash.MakeBigStruct:
   0x0000000000100830 <+0>:     str     x30, [sp, #-32]! [3]
   0x0000000000100834 <+4>:     stur    x29, [sp, #-8]
   0x0000000000100838 <+8>:     sub     x29, sp, #0x8
   0x000000000010083c <+12>:    stur    xzr, [sp, #12]
   0x0000000000100840 <+16>:    str     wzr, [sp, #20]
   0x0000000000100844 <+20>:    ldrb    w0, [sp, #12]
   0x0000000000100848 <+24>:    mov     x1, xzr
   0x000000000010084c <+28>:    mov     x2, x1
   0x0000000000100850 <+32>:    mov     x3, xzr
   0x0000000000100854 <+36>:    mov     x4, x3
   0x0000000000100858 <+40>:    add     sp, sp, #0x20
   0x000000000010085c <+44>:    sub     x29, sp, #0x8 Here's where I want to interrupt
   0x0000000000100860 <+48>:    ret
   0x0000000000100864 <+52>:    udf     #0
   0x0000000000100868 <+56>:    udf     #0
   0x000000000010086c <+60>:    udf     #0
End of assembler dump.
(gdb) b *0x000000000010085c
Breakpoint 1 at 0x10085c: file /home/ec2-user/fpcrash/fpcrash_test.go, line 13.
(gdb) r -test.trace=/dev/null
Starting program: /home/ec2-user/fpcrash/fpcrash.test -test.trace=/dev/null
[ ... ]
Thread 1 "fpcrash.test" hit Breakpoint 1, 0x000000000010085c in fpcrash.MakeBigStruct (~r0=..., ~r0=...) at /home/ec2-user/fpcrash/fpcrash_test.go:13
13              return BigStruct{}
(gdb) signal SIGURG
Continuing with signal SIGURG.

This still didn't crash. I wanted to see what the sequence of frame pointers looked like. So after hitting the first breakpoint, I set a breakpoint in runtime.asyncPreempt2 and then delivered the signal. I hit the second breakpoint, and I could now try following the frame pointers myself. For every frame pointer address there is a return instruction address right next to it:

Thread 1 "fpcrash.test" hit Breakpoint 1, 0x000000000010085c in fpcrash.MakeBigStruct (~r0=..., ~r0=...) at /home/ec2-user/fpcrash/fpcrash_test.go:13
13              return BigStruct{}
(gdb) b 'runtime.asyncPreempt2'
Breakpoint 2 at 0x4750c: runtime.asyncPreempt2. (2 locations)
(gdb) signal SIGURG
Continuing with signal SIGURG.

Thread 1 "fpcrash.test" hit Breakpoint 2, runtime.asyncPreempt2 () at :1
[ ... step forward a few instructions to have runtime.asyncPreempt2 set up its call frame ... ]
(gdb) p/x $x29
$2 = 0x400009fd18
(gdb) x/g 0x400009fd18
0x400009fd18:   0x000000400009fd38
(gdb) x/a 0x400009fd18+8
0x400009fd20:   0x7a2dc <runtime.asyncPreempt+156>
(gdb) x/g 0x000000400009fd38
0x400009fd38:   0x400009ff18
(gdb) x/a 0x000000400009fd38+8
0x400009fd40:   0x10085c <fpcrash.MakeBigStruct+44>
(gdb) x/g 0x400009ff18
0x400009ff18:   0x0

The x/g lines dereference the given address. This is how I'm following the frame pointer linked list. The x/a lines dereference the given address and treat the value as an instruction address. The return addresses that make up the call stack are stored 8 bytes above the frame pointers. So, this is how I reconstruct the call stack with frame pointer unwinding. I reached a 0 frame pointer, which makes frame pointer stop normally. So, this explains why it's not crashing. But the traceback is incomplete. We should see the TestCrash function, at least. Something is indeed broken!

I spent a long time making small tweaks to my program, under the incorrect assumption that the MakeBigStruct method was somehow manipulating the frame pointer incorrectly. I would tweak the code, trigger the preemption, and do frame pointer unwinding. I did the unwinding step often enough by hand that it was getting tedious. I learned that GDB can be scripted with Python, so I wrote a bit of code to do the unwinding:

(gdb) py
>def fpunwind():
>    _ptrptr = gdb.lookup_type('void').pointer().pointer()
>    fp = gdb.selected_frame().read_register('x29')
>    while fp != 0:
>        fp = fp.reinterpret_cast(_ptrptr)
>        pc = (fp+1).dereference()
>        fp = fp.dereference()
>        print(fp, pc)
(gdb) py fpunwind() # while at a breakpoint where I think the frame pointers are good
0x40000a7f38 0x10088c <fpcrash.TestCrash+28>
0x40000a7f58 0xc9f38 <testing.tRunner+232>
0x40000a7fa8 0xcad4c <testing.(*T).Run.func1+44>
0x0 0x79744 <runtime.goexit+4>

I finally figured out what was happening when I read back through my initial investigation notes. The asynchronous preemption handling code was going to use previously-freed stack space. I decided to check what the code was doing. Sure enough, the initial preemption handler code, run on the goroutine's stack, spills a bunch of registers. The preemption handler is like a normal function, and saves the frame pointer register, which will be the frame pointer register at the point where the function was interrupted. So the call stack looks like this:

I just needed to see which registers were saved around 32 bytes from the top of the preemption handler's frame, where the frame pointer register was pointing prior to preemption. It turned out to be some floating point registers. My test code didn't do any floating point operations, so those registers must have been 0! GDB lets you set register values, so it was easy enough to force non-zero values in the registers. With that, I was able to recreate the crash!

Thread 1 "fpcrash.test" hit Breakpoint 1, 0x000000000010085c in fpcrash.MakeBigStruct (~r0=..., ~r0=...) at /home/ec2-user/fpcrash/fpcrash_test.go:13
13              return BigStruct{}
(gdb) set $d30.s = 0xdeadbeef
(gdb) signal SIGURG
Continuing with signal SIGURG.

Thread 1 "fpcrash.test" received signal SIGSEGV, Segmentation fault.
runtime.traceStackID (mp=<optimized out>, pcBuf=..., skip=<optimized out>, ~r0=<optimized out>) at /home/ec2-user/sdk/go1.21.0/src/runtime/trace.go:987
987                             nstk += 1 + fpTracebackPCs(unsafe.Pointer(curgp.sched.bp), pcBuf[2:])
(gdb) disas to show where we're faulting
[ ... ]
0x0000000000069370 <+544>:   b       0x69350 <runtime.traceStackID;+512>
=> 0x0000000000069374 <+548>:   ldr     x8, [x4, #8]
0x0000000000069378 <+552>:   str     x8, [x6, x0, lsl #3]
[ ... ]
(gdb) p/x $x4
$9 = 0xdeadbeef

Success! I modified my test program to put non-zero values in those registers. With that, the test crashed reliably. I had a reproducer and was ready to file a bug report.

I didn't do this at the time, but there's another useful GDB feature that would have helped me find where the old frame pointer was being overwritten. GDB supports setting watchpoints on specific memory addresses. Under the assumption that we had a frame pointer register pointing to freed stack space, I could have set a watchpoint on the address in the frame pointer register:

Thread 1 "fpcrash.test" hit Breakpoint 1, 0x000000000010085c in fpcrash.MakeBigStruct (~r0=..., ~r0=...) at /home/ec2-user/fpcrash/fpcrash_test.go:13
13              return BigStruct{}
(gdb) p/x $x29
$11 = 0x400009af18
(gdb) watch *0x400009af18
Hardware watchpoint 4: *0x400009af18
(gdb) signal SIGURG
Continuing with signal SIGURG.

Thread 1 "fpcrash.test" hit Hardware watchpoint 4: *0x400009af18

Old value = 634680
New value = 0
runtime.asyncPreempt () at /home/ec2-user/sdk/go1.21.0/src/runtime/preempt_arm64.s:47
47              CALL ·asyncPreempt2(SB)
(gdb) disas
[ ... ]
0x000000000007a2d0 <+144>:   stp     d28, d29, [sp, #456]
0x000000000007a2d4 <+148>:   stp     d30, d31, [sp, #472]
=> 0x000000000007a2d8 <+152>:   bl      0x7b280 <runtime.asyncPreempt2>

So I would have seen that the memory the frame pointer register was pointing to was overwritten when the d30 and d31 were saved by runtime.asyncPreempt.

One more thing to note here: the floating point registers weren't special here. For my reproducer, the frame pointer register happened to point to where those registers were saved. But that depends on the frame size of the function that's getting preempted. The important thing is that the frame pointer register, prior to preemption, was pointing to stack memory that was just freed. Any junk could go on the stack where the frame pointer register was pointing.

I filed an issue to report this bug to the Go project. I figured the fix would be to rearrange the frame cleanup instructions. It seemed like the fix would need to happen in the compiler, which I don't know anything about. I didn't want to hold up getting the problem fixed while I ramped up on the compiler.

Keith Randall, one of compiler maintainers, quickly put together a fix. The fix was to rearrange the call frame cleanup instructions, making sure the frame pointer is restored before the call frame is freed. With that, the instructions look like this:

000000000010cc10 <fpcrash.MakeBigStruct>:
        10cc10:       f81e0ffe        str     x30, [sp, #-32]!
        10cc14:       f81f83fd        stur    x29, [sp, #-8]
        10cc18:       d10023fd        sub     x29, sp, #0x8
[ ... ]
        10cc38:       910063fd        add     x29, sp, #0x18
        10cc3c:       910083ff        add     sp, sp, #0x20
        10cc40:       d65f03c0        ret

And if the function were preempted at the next-to-last instruction, the call stack would look like this, and the frame pointer register would be safe to use:

I was surprised at first not to see my reproducer (or something similar) as a regression test in the fix. But my reproducer might not have actually been robust enough. It reliably crashed eventually on my machine each time I ran it. It can't run forever in CI, though. How long would the actual test wait until it could declare no bug? And would the test catch the same general kind of bug if it was reintroduced elsewhere, or if the runtime changed substantially? I wonder if there is a more formal, model-based way to check for the correctness of frame pointers. We can say certain qualities frame pointers should have: they should point to the value 0 or another valid frame pointer, there should be a return address stored at a fixed offset from where the frame pointer points, and a signal handler which wants to get a traceback should not observe the frame pointers in an inconsistent state. Perhaps we could build a tool to check these things comprehensively for a compiled binary?

I learned about several useful GDB features: delivering signals, breaking at specific instructions, scripting with Python. I learned about how preemption works, improving my mental model for the control flow of Go programs. I also understood a little more of the 64-bit ARM calling conventions.

I lost a lot of time to tunnel vision. Once I caught the scent of a possible cause for the crash, I jumped to a conclusion about where the problem was. I didn't fully digest what I saw based on the assembly I was reading, and overlooked a crucial fact (the function freed the frame before restoring the frame pointer), only revisiting it after hours of fruitless debugging. Debugging is about pinpointing the difference between expectation and reality. We should be thorough: we should lay out how we expect the system to work, take a full inventory of what we've observed, and then focus on the gaps in our understanding or observations. I like how Bryan Cantrill put it in his talk "The Hurricane's Butterfly".

It can be very tempting to think about debugging as the process of forming hypotheses. That is not what debugging is. Hypotheses are... Yes, a hypothesis is going to be a critical stage to debugging. It is the last stage of debugging.

When you debug, you should be iterating between questions and answers, questions and answers, questions and answers. And as you're iterating, the possible space of hypotheses gets smaller and smaller and smaller, until you don't have a choice.

Thanks to Daniel Schwartz-Narbonne, Matt Alp, and Felix Geisendörfer for reviewing this post. Special thanks to Felix for suggesting visualizations for the call stack state.

1. You might have noticed that this instruction is not reading the value previously saved on the stack. It happens to still be correct, because the caller's frame pointer would have pointed to just below the caller's stack frame, which is where sp now points! Back
2. Go symbol names have characters like "." and "/" that confuse GDB. You need to enclose the names in quotation marks. Back
3. You might notice the stack frame is 32 bytes rather than 48. I'm not totally sure where the difference comes from. In the real code, the Header function comes from one struct embedded in another. Felix suggests that a method for the embedded struct might be generated, with a bigger call frame to account for the other fields of the struct, if the struct is used via an interface? Back