Compilation bottleneck

[timholy]

I don't understand the compile toolchain very well, and I haven't yet gotten good profile info (see https://groups.google.com/d/msg/julia-dev/RKAahOwppNs/Kg0TFx_SBwAJ), but at least for the SubArray tests I'm guessing that this line, via jl_compress_ast, is an enormous bottleneck in compilation. I inserted a debugging line,

diff --git a/src/dump.c b/src/dump.c
index e11bff5..9c3c96f 100644
--- a/src/dump.c
+++ b/src/dump.c
@@ -618,6 +618,7 @@ static int is_ast_node(jl_value_t *v)

 static int literal_val_id(jl_value_t *v)
 {
+    jl_printf(JL_STDOUT, "len = %d\n", jl_array_len(tree_literal_values));
     for(int i=0; i < jl_array_len(tree_literal_values); i++) {
         if (jl_egal(jl_cellref(tree_literal_values,i), v))
             return i;

and saw that most of the lengths were around 10k or so. That's a heck of a lot of calls to jl_egal. Is there anything that can be done to reduce this number?

[vtjnash]

we could put them in an ObjectIdDict, but that may make it slower in the expected usage. that table is supposed to be nearly empty, but it can get populated if code is interpolating non-ast objects into the ast.

perhaps we should switch to using a per-linfo list of literals? the list could re-use the codegen roots list -- it seems likely that these same objects are likely getting copied there by emit_linfo_root during codegen anyways.

[JeffBezanson]

At one point we switched to per-module literal lists since many values ended up being shared by several functions. But I agree it seems silly to have both that and linfo->roots.

10k is way more than I'd expect. I wonder if we can cut that down (e.g. by serializing more things). Or just fully switch to a bytecode representation.

[timholy]

It will be a little while, but to save anyone else time, I'm collecting the histogram of counts by running make testall1 with

diff --git a/src/dump.c b/src/dump.c
index e11bff5..50ab040 100644
--- a/src/dump.c
+++ b/src/dump.c
@@ -6,6 +6,7 @@
 #include <stdlib.h>
 #include <string.h>
 #include <assert.h>
+#include <stdio.h>

 #include "julia.h"
 #include "julia_internal.h"
@@ -616,8 +617,21 @@ static int is_ast_node(jl_value_t *v)
         jl_is_labelnode(v) || jl_is_linenode(v) || jl_is_globalref(v);
 }

+int tlvlen[50000] = { 0 };
+
+JL_DLLEXPORT void jl_write_tlvlen(void)
+{
+    FILE *f = fopen("/tmp/tlvlen.bin", "w+");
+    fwrite(tlvlen, sizeof(int), 50000, f);
+    fclose(f);
+}
+
 static int literal_val_id(jl_value_t *v)
 {
+    int n = jl_array_len(tree_literal_values);
+    if (n >= 50000)
+        n = 49999;
+    tlvlen[n] += 1;
     for(int i=0; i < jl_array_len(tree_literal_values); i++) {
         if (jl_egal(jl_cellref(tree_literal_values,i), v))
             return i;
diff --git a/src/julia.h b/src/julia.h
index b93c268..03545fc 100644
--- a/src/julia.h
+++ b/src/julia.h
@@ -1678,6 +1678,8 @@ JL_DLLEXPORT extern const char* jl_ver_string(void);
 JL_DLLEXPORT const char *jl_git_branch(void);
 JL_DLLEXPORT const char *jl_git_commit(void);

+    JL_DLLEXPORT void jl_write_tlvlen(void);
+
 // nullable struct representations
 typedef struct {
     uint8_t isnull;
diff --git a/test/runtests.jl b/test/runtests.jl
index 969c72e..6b7cb00 100644
--- a/test/runtests.jl
+++ b/test/runtests.jl
@@ -72,3 +72,5 @@ cd(dirname(@__FILE__)) do
     @unix_only n > 1 && rmprocs(workers(), waitfor=5.0)
     println("    \033[32;1mSUCCESS\033[0m")
 end
+
+ccall(:jl_write_tlvlen, Void, ())

Will report back when it finishes.

[JeffBezanson]

It might also make sense to count the total iterations spent in that loop (# of jl_egal calls). Or maybe you have that already?

[timholy]

Yeah, I thought of that after I launched the first one. FWIW here are the results:

Median value was 10, mean value was 254.

Want me to collect the # of jl_egal calls too?

[timholy]

Wait, I was totally borked on those statistics. Median value is 10024, mean value is 11305.

[timholy]

Plotting the cumulative histogram on a linear scale suggests there's one group of "short lists" (<300) and another with "long lists" (between about 7k and 30k):

[timholy]

If you just count jl_egal calls, it's a radically different picture:

Median value 128, mean value 1585.

[timholy]

I decided to profile the test suite. I had to exclude a few tests (profile because it calls Profile.clear(); also "cmdlineargs", "file", "libdl", "parallel" because they gave problems for reasons I didn't track down). I ran this on release-0.4 (0.4.3-pre6), because I don't trust profiling on 0.5 due to #14281. I collected one sample every 10 ms (100samples/sec).

Analysis script (whose purpose was to show "flat" results that don't suffer from the duplicates-due-to-recursion problem as illustrated by dumbsum) and results are here: https://gist.github.com/timholy/67762c65cbfe4226b43a. I only included lines that accounted for at least 100 samples, i.e., at least 1s of time. Scroll to the end to see the worst offenders.

While it's a shame that so many important lines lack good backtrace info (here's hoping that @Keno's work will help), to me it looks as if jl_egal is the single worst "low-level" offender, accounting for more than 8% of the total number of samples collected. That, combined with the fact that the number of jl_egal calls is clearly not as small as seems to have been expected, suggests there's an opportunity for measurable improvement.

[Keno]

Unfortunately, I'm probably not as much help here. I briefly implemented a more efficient/accurate profiler on top of Gallium, but it only worked on OS X and had a number of bugs that made it essentially unusable. That said, I do plan to revisit the topic of profiling once I've made some progress on the debugger front.

I can say however, that I concur with your analysis of jl_egal being a significant chunk of our compile time. I don't have the traces anymore, but I remember pointing this fact out to @JeffBezanson

[timholy]

Bummer about the backtraces. Hopefully we'll get there some day. I also just remembered that long ago I convinced myself (rightly or wrongly) that most of the borked backtraces are almost surely from our many C functions that are declared static---they're not exported in the .so file, and as a consequence our means of looking them up fail.

[Keno]

Well, the backtraces shouldn't be super borked. What OS are you on?

[timholy]

See more info here.

[timholy]

Kubuntu 14.04.3 LTS, Trusty Tahr

[Keno]

Hmm, should be fine then. On OS X you have to jump through some hoops to make the dSYM available such that we can find it to get at the debug line info.