cmd/compile: convert merge to use appl. bal. trees for sharing

This CL replaces a not-very-shared linear-sized set
representation with a much more shared representation.
For the annoying test program in question, it reduces
the heap size by 95%, and the time slightly.

However, for some programs build time is longer.

This also includes at least one bug fix for problems
uncovered while ensuring compatibility with what it
replaces.

Fixes #51543.

Change-Id: Ie7a4c6ea460775faeed2b0378ab21ddffd15badc
Reviewed-on: https://go-review.googlesource.com/c/go/+/397318
Run-TryBot: David Chase <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Than McIntosh <[email protected]>

Author: dr2chase