Compile packages independently #285
Comments
|
Some statistics when building The entire compiler runs serially at the moment, while most of this could be parallelized and/or cached.
|
|
First off, thanks for releasing TinyGo @aykevl. I keep learning so much from this project! Tonight I was curious to see how TinyGo lowered Go interfaces to LLVM IR and thus wanted to compile the following Go package: package p
type T int
func (t T) M() int {
return int(t) + 10
}
type I interface {
M() int
}
func F(i I) int {
return i.M()
}However, doing so, I ran into a type assertion panic, as detailed below. After a bit of troubleshooting it seems like this was due to TinyGo looking for and not finding the From github.com/tinygo-org/tinygo/ir/passes.go:69: main := p.mainPkg.Members["main"].(*ssa.Function)Converting package The following LLVM IR tail call fastcc void @runtime.printuint64(i64 10), !dbg !42is all that's left from compiling: package main
func main() {
var t T
println(F(t))
}
type T int
func (t T) M() int {
return int(t) + 10
}
type I interface {
M() int
}
func F(i I) int {
return i.M()
}My next attempt was to disable optimizations entirely, and to compile with Note that no panic occurs on optimization level 1, The output optimization level 1 still optimizes out the interface related code, and produces: define internal fastcc void @p.go.main() unnamed_addr section ".text.p.go.main" !dbg !104 {
entry:
tail call fastcc void @runtime.printint64(), !dbg !105
tail call fastcc void @runtime.printnl(), !dbg !105
ret void, !dbg !106
}
define internal fastcc void @runtime.printint64() unnamed_addr section ".text.runtime.printint64" !dbg !41 {
entry:
call void @llvm.dbg.value(metadata i64 10, metadata !47, metadata !DIExpression()), !dbg !48
tail call fastcc void @runtime.printuint64(i64 10), !dbg !49
ret void, !dbg !50
}While the above code produces the correct output result ( So, to summarize this night-time adventure into exploring interface handling in TinyGo, I still had a lot of fun using TinyGo and feel quite amazed to see how far the project has gotten already. I definitely welcome the intended direction of TinyGo to compile packages independently as this makes it easier to debug, interact with, and use LLVM tools to process the LLVM IR output of TinyGo, not only for One may envision using this functionality of TinyGo to compile Go packages to LLVM IR, and then invoking specific functions of the compiled LLVM IR from a I wish you all the best and happy continued coding! Cheers, |
|
Thank you for your interest in TinyGo internals! Yes, as you've discovered TinyGo does some whole-program optimizations that often result in interfaces being optimized away entirely. This is very important for code size and to enable further optimizations such as inlining, const propagation, escape analysis, etc.
While that would be quite interesting, that is difficult to do in the current design for a few reasons. First of all, most compiled code depends on the runtime package for various reasons (initialization, heap, map operations). Second, whole-program optimizations (which are in fact required in the current design of TinyGo) can obviously not be done for individual packages. However, I could imagine we'd eventually have
Yes, that has definitely complicated debugging for me sometimes. I think this optimization is the result of the following two optimization passes: putchar('1');
putchar('0');
putchar('\r');
putchar('\n');The fact that TinyGo breaks at You may want to look at the I do see that while the call void @llvm.dbg.value(metadata i64 10, metadata !47, metadata !DIExpression()), !dbg !48That should allow a debugger to still print the 10 in a backtrace when breaking inside
Thanks a lot 😄 |
|
Hi @aykevl, Thanks for the detailed reply!
The way I'd envision using this functionality would be to compile individual Go packages to LLVM IR, and let this LLVM IR module be the build artefact of the compiled package; similar to how
The final step of linking would enable whole-program optimization. No need to do it at an earlier stage. So linking Of course, I understand that if The benefit of not relying on
Great, I'll definitely take a look at Cheers, |
|
@mewmew what you propose is indeed what I intend to get to eventually, but again the structure of the TinyGo compiler at the moment makes this difficult. Quite a bit refactoring has been done already to get to that point and it's nowhere finished (although #1571 is a significant step). (this is a late reply but I didn't want to leave this entirely unanswered) |
|
@aykevl, I know progress take time, so no rush! Also, you initiated the issue, so I felt quite certain you would return to it as time would allow. Very happy to see the work on making the compiler more modular. Wish you the best of hacking and a great start of the new year! :) Cheers, |
|
This has finally been implemented, in #1612! It only took me two years to get to this point 😅 There are many things that are still left to do, but right now TinyGo compiles packages in parallel and caches the result. This provides a small performance benefit. There are many more optimizations possible:
Closing now, as the initial goal (compiling packages independently) has been reached. |
Yay! That's great. Thanks for working on this @aykevl, it will definitely be useful and also open up new use cases for TinyGo. Wish you as always happy hacking and the best of springs! Cheerful regards, |
|
I will reopen to tag for next release, then close after it is released. |
|
This was released with v0.18 so now closing. Thank you! |
Packages should be compiled independently and then linked together. This provides several advantages in the long term:
I would strive for the following architecture:
go tool compile,go tool link) that call the compiler/linker directly. This can be useful for debugging.I have taken a look at the Go build cache and I think I'd implement a similar system. Every package has a hash of the input (build tags, file list, hashes of files, output hashes of packages it depends on) and produces an output hash, which is a hash over all the artifacts that it builds (mostly the serialized exported types).
My thinking is that a cached package would be an uncompressed zip file stored in the cache directory, with the following contents as files:
This is most likely too big to do at once, so I'd do it in multiple steps. But I'm creating this issue to give a high-level overview of what I would want to achieve at some point.
The text was updated successfully, but these errors were encountered: