compiler: preemptively indirect large params/results instead of letting LLVM do it

[jakebailey]

#5477 revealed a preexisting bug(?) with large types.

When there are too many parameters or results (common! it's not like every result can fit in registers), LLVM indirects things. It turns out that the result value it makes is a flattened struct, one element per struct field and array element. So then each of those elements then gets its own LLVM value to construct it, which means that it scales up worse and worse the more array elements there are. In SelectionDAG this also hits a hard representation limit: SDNode::NumValues is an unsigned short, and overflows at 65,536 flattened values.

https://go.dev/cl/707355 added a struct with a massive array that's passed by value:

type tokens struct {
	extraHist [32]uint16  // codes 256->maxnumlit
	offHist   [32]uint16  // offset codes
	litHist   [256]uint16 // codes 0->255
	nFilled   int
	n         uint16 // Must be able to contain maxStoreBlockSize
	tokens    [65536]token
}

This makes the compiler very unhappy. In this case, tokens has 65,858 scalar leaves, so direct lowering can crash LLVM in addition to being very slow.

The only way I can think to avoid this is to just do the indirection ourselves before handing it over to LLVM so that it doesn't hit this limit. TinyGo now does this for internal aggregate params/results above 1024 flattened scalar leaves, which is intentionally well below LLVM's 65,535-value representation limit to avoid the compile-time/memory cliff before the crash boundary. Exported function types are unchanged.

This is a pretty invasive change. I tried to split this stack into enough individual (and I think reasonable) commits with abstractions to make this reviewable one commit at a time.

Temporary backing storage goes through TinyGo's normal allocation path: non-escaping bounded allocations may still be stack-promoted by the allocation optimizer, while escaping storage remains heap-backed. It's unlikely anyone will actually hit this code, though, given before they would have been very much timing things out?

[jakebailey]

FYI @deadprogram, this is the "horrorshow" from #5477 (comment), though it's less bad when split apart.

[jakebailey]

I have to be at least a bit amused by the loops I'm replacing, which are all written completely differently despite being identical.

[deadprogram]

@jakebailey can you pull the latest dev into this branch to run the tests, please?

[deadprogram]

Here is edited version of the useful parts of some generated feedback:

createIndirectStorage calls runtime.alloc (GC-tracked heap allocation) rather than using stack allocas. For frequently-called functions with large params, this adds GC pressure. The comment in the PR description says "The indirection happens on the stack where possible" but looking at the code, createIndirectStorage always calls runtime.alloc. The getValueStorage helper does use createTemporaryAlloca for small types but falls through to getValuePointer -> createIndirectStorage for large ones. Is there a reason stack allocation (alloca) wasn't used for the indirect storage? It seems like for values that don't escape, a stack alloca would be more efficient.

isLLVMValueType filtering determines whether a Go type maps to a "pure" LLVM value type. Types containing interfaces, maps, slices etc. are excluded from indirect treatment because their LLVM representation is already a small struct of pointers. However, a struct containing a large array AND an interface field would return false from isLLVMValueType and skip indirection entirely, even though the array component is huge. Is this intentional?

prependIndirectResult in defer replay, when replaying deferred calls, prependIndirectResult allocates storage for the result, but deferred calls discard their return values. This allocates memory that's never read. It's harmless but wasteful.

getGoroutineCallArgument correctly copies indirect params to new storage (since the goroutine outlives the caller), but uses createIndirectStorage (heap alloc). For go readLargeValue(value), this means an extra 1025-byte heap allocation + memcpy. This is correct but worth noting for performance.

[jakebailey]

createIndirectStorage calls runtime.alloc (GC-tracked heap allocation) rather than using stack allocas. For frequently-called functions with large params, this adds GC pressure. The comment in the PR description says "The indirection happens on the stack where possible" but looking at the code, createIndirectStorage always calls runtime.alloc. The getValueStorage helper does use createTemporaryAlloca for small types but falls through to getValuePointer -> createIndirectStorage for large ones. Is there a reason stack allocation (alloca) wasn't used for the indirect storage? It seems like for values that don't escape, a stack alloca would be more efficient.

So, yes, but these are allocs that would normally have crashed the compiler, so I'm not sure that I'm actually regressing much?

isLLVMValueType filtering determines whether a Go type maps to a "pure" LLVM value type. Types containing interfaces, maps, slices etc. are excluded from indirect treatment because their LLVM representation is already a small struct of pointers. However, a struct containing a large array AND an interface field would return false from isLLVMValueType and skip indirection entirely, even though the array component is huge. Is this intentional?

I don't think this is true, but maybe I can add a test.

prependIndirectResult in defer replay, when replaying deferred calls, prependIndirectResult allocates storage for the result, but deferred calls discard their return values. This allocates memory that's never read. It's harmless but wasteful.

Yes, but I don't think there's a way around this one.

getGoroutineCallArgument correctly copies indirect params to new storage (since the goroutine outlives the caller), but uses createIndirectStorage (heap alloc). For go readLargeValue(value), this means an extra 1025-byte heap allocation + memcpy. This is correct but worth noting for performance.

IIRC this happened before, it's just that I'm doing it earlier, right?

[jakebailey]

I have some fixes I'll push in, but, I have to compile LLVM 22 first so that'll take me a bit 😄

[deadprogram]

@jakebailey can you please resolve the merge conflicts/rebase against the latest dev?

[dgryski]

If we're moving values passed to functions to pointers-to-allocations, I don't think we can safely use stack allocations unless we can guarantee the values don't escape. Will this be handled by the existing move-to-stack optimization pass?

[jakebailey]

To be clear, this is about how the arguments and results are stored during the call; the objects don't exist except specifically for that one call. This is basically just spilling, it's just that LLVM is slow to do that spilling for you + has a uint16 counter that maxes out.

[deadprogram]

Another merge conflict to resolve @jakebailey if you please.

[deadprogram]

Still merge conflicts here @jakebailey

[jakebailey]

Oops, I only pulled my fork's dev, not upstream.

[dgryski]

Test corpus seems happy with this change.

[deadprogram]

From my point of view, it seems like it is ready for merge.