elf: support external and supplementary debuginfo #427

philipc · 2021-06-09T05:18:30Z

Fixes #410

Doesn't check the CRC for GNU debuglink. I'm not sure how important this is, and it would require adding a dependency (crc32fast?).

Doesn't add any tests; I'm unsure how to test this in CI. I have manually tested that the backtrace example can give line number information for __libc_start_main, which tests the build ID on Fedora, and the GNU debuglink on Ubuntu. I have manually tested the supplementary object file support using the backtrace and raw examples after running the following commands:

objcopy --only-keep-debug target/debug/examples/backtrace target/debug/examples/backtrace.debug
strip -g target/debug/examples/backtrace
(cd target/debug/examples; objcopy --add-gnu-debuglink=backtrace.debug backtrace)

mkdir -p target/debug/examples/.debug
objcopy --only-keep-debug target/debug/examples/raw target/debug/examples/.debug/raw.debug
strip -g target/debug/examples/raw
(cd target/debug/examples/.debug; objcopy --add-gnu-debuglink=raw.debug ../raw)

mkdir -p .dwz
dwz -m .dwz/backtrace.debug -rh target/debug/examples/backtrace.debug target/debug/examples/.debug/raw.debug

philipc · 2021-06-09T05:21:35Z

src/symbolize/gimli/elf.rs

+
+        let stash = Stash::new();
+        let cx = Context::new(&stash, object)?;
+        Some(Mapping {


This no longer uses Mapping::mk because it needs to parse the Object beforehand. Not sure if there is any way to restructure this to allow that.

alexcrichton

Awesome, thanks for this! This is definitely tricky enough though that I'd want some tests in CI for this. If this is something that varies across unix distributions it'd be great if a variety of containers could run as well and we'd run tests in those containers.

To figure that out, though, can you describe to me how this is expected to work? Does separate debuginfo always require the intervention of a separate tool? It looks like with objcopy you're deleting the debuginfo from one file, adding it to another, and then adding a link of one form or another from the original file to the new file? I'm not entirely sure what dwz is doing...

This may not be a scenario where tests can run via cargo test, but it might be a situation where there needs to be an auxiliary crate where cargo run runs all the tests so you can cargo build, frob the debuginfo, and then run the binary manually to run all the tests.

Finally, it looks like this is handling /usr/lib things, which is presumably for system-installed debuginfo? Could the tests link to some library which has debuginfo inserted there and assert that the filenames and line numbers appear ok?

src/symbolize/gimli.rs

alexcrichton · 2021-06-09T20:50:36Z

src/symbolize/gimli/elf.rs

+                path.push(hex(byte & 0xf));
+            }
+            path.extend(BUILD_ID_SUFFIX);
+            if let Some(mapping) = Mapping::new_debug(Path::new(OsStr::from_bytes(&path)), None) {


Are these paths and this scheme linux-specific? If so should this have an if cfg! around it perhaps?

src/symbolize/gimli/elf.rs

alexcrichton · 2021-06-09T21:00:01Z

src/symbolize/gimli/elf.rs

+
+        // Try to locate a supplementary object file.
+        let tmp_stash = Stash::new();
+        if let Some(section) = object.section(&tmp_stash, ".gnu_debugaltlink") {


Oh deear, I can't seem to find documentation on this... Do you know of some good online docs to link to?

I'm curious about why this is handled here rather than above from the original object...

I don't know of any documentation for the format of this. The only documentation I found is the dwz man page, but it doesn't give this detail. I based the code on what libbacktrace does, but gdb would have been a better source. And indeed, gdb does the path resolution differently so I'll have to fix that.

It is handled here because this is a section in the external debug file, not the original object.

src/symbolize/gimli/elf.rs

alexcrichton · 2021-06-09T21:03:26Z

src/symbolize/gimli/elf.rs

+
+    // Try "/usr/lib/debug/parent/filename"
+    let parent = parent.strip_prefix("/").unwrap();
+    let f = Path::new("/usr/lib/debug").join(parent).join(filename);


Similar to above, if this is linux-specific should this be wrapped in a if cfg!?

Or perhaps something like a global AtomicBool which indicates the presence of /usr/lib/debug should prefix this so we don't keep trying on systems that don't have that directory?

gdb uses a debug-file-directory setting which defaults to "${libdir}/debug. This whole external debug scheme is distribution specific, but I think most linux distributions use it, and I think FreeBSD does too. For those that do, I don't know if any deviate from /usr/lib/debug. So we probably need a if cfg! around the whole build-id/debuglink support. I think having an AtomicBool for the presence of /usr/lib/debug would be worth it too, since I assume someone might add build-id notes to binaries for purposes other than locating debug info.

What about MinGW? Doesn't it support build-id/debuglink too?

I don't know. Do you have a link to any information about that?

philipc · 2021-06-10T07:34:32Z

Awesome, thanks for this! This is definitely tricky enough though that I'd want some tests in CI for this. If this is something that varies across unix distributions it'd be great if a variety of containers could run as well and we'd run tests in those containers.

Sure, I can work on doing that.

To figure that out, though, can you describe to me how this is expected to work? Does separate debuginfo always require the intervention of a separate tool? It looks like with objcopy you're deleting the debuginfo from one file, adding it to another, and then adding a link of one form or another from the original file to the new file?

That's pretty much it. https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html describes this fairly well, but in summary:

the linker adds a build ID note
objcopy extracts the debuginfo to a debug file, deletes it from the original file and inserts a .gnu_debuglink section with the name of the debug file
during the build, this debug file may located in either the same directory, or under a .debug directory (I do one of each of these for the commands I tested with)
the packager installs the debug file in a tree under /usr/lib/debug, and adds a symbol link to that from /usr/lib/debug/.build-id

I'm not entirely sure what dwz is doing...

dwz by itself is a DWARF compressor; it tries to rewrite the DWARF in an equivalent form that is smaller. It is run on debug files that have already been extracted from the original files. With the -m option it can also identify common DWARF entries in multiple debug files, and move these to a shared file called the supplementary object file. The DWARF entries in the debug files are modified to refer to the supplementary object file using special DWARF forms (DW_FORM_ref_sup4/8 and DW_FORM_strp_sup). dwz modifies the debug files to contain .gnu_debugaltlink sections with the name of the supplementary object file.

This may not be a scenario where tests can run via cargo test, but it might be a situation where there needs to be an auxiliary crate where cargo run runs all the tests so you can cargo build, frob the debuginfo, and then run the binary manually to run all the tests.

Sounds doable.

Finally, it looks like this is handling /usr/lib things, which is presumably for system-installed debuginfo? Could the tests link to some library which has debuginfo inserted there and assert that the filenames and line numbers appear ok?

Yeah. So firstly we'll need to ensure the container has system-installed debuginfo. And then we'll need to get a stack trace that contains that debuginfo. This is what I was asking cuviper about in #410, but I gave up and just used __libc_start_main instead.

philipc · 2021-06-10T07:44:46Z

Perhaps we could just manually install the auxiliary crate debug info into the system dir. Is that something we can do for the containers in CI, and would that be a good enough test?

alexcrichton · 2021-06-10T18:39:39Z

Ok that all sounds good! I'm happy to help out with CI too if desired.

philipc · 2021-06-18T04:31:41Z

This is ready for further review. I've added a single docker test as part of the main Ubuntu tests. It doesn't really need docker except that it lets us write to /usr/lib/debug.

alexcrichton

This looks perfect, thanks so much!

I'm gonna fiddle locally a bit afterwards to see if I can fixup some of the transmutes to 'static, I'd ideally like to contain that if possible. In any case though this all looks great to me, thanks again!

philipc commented Jun 9, 2021

View reviewed changes

philipc force-pushed the sup branch from 9abbfe8 to 71fcbc2 Compare June 9, 2021 05:27

philipc added 2 commits June 9, 2021 15:31

elf: locate debuginfo via build ID or GNU debuglink

99cc11e

elf: supplementary object file support

4208042

philipc force-pushed the sup branch from 71fcbc2 to 4208042 Compare June 9, 2021 05:31

alexcrichton reviewed Jun 9, 2021

View reviewed changes

Delete Context::new_sup

e18f530

philipc force-pushed the sup branch 3 times, most recently from fc20f4b to 13b5ed9 Compare June 18, 2021 03:57

philipc added 5 commits June 18, 2021 14:02

Reduce nesting in Mapping::new

ffb6c2b

Add documentation links for debuglink

a6ff0b1

Cache check for /usr/lib/debug and restrict to linux/freebsd

9c8edbf

Fix debugaltlink path resolution

588574c

Canonicalize for relative debugaltlink path

adaa78b

philipc force-pushed the sup branch from 13b5ed9 to e43c781 Compare June 18, 2021 04:11

Add debuglink tests

9d24115

philipc force-pushed the sup branch from e43c781 to 9d24115 Compare June 18, 2021 04:19

alexcrichton approved these changes Jun 18, 2021

View reviewed changes

alexcrichton merged commit ed3689c into rust-lang:master Jun 18, 2021

bobrik mentioned this pull request Jan 19, 2022

Support debuginfod #459

Open

philipc deleted the sup branch May 28, 2024 02:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

elf: support external and supplementary debuginfo #427

elf: support external and supplementary debuginfo #427

philipc commented Jun 9, 2021

philipc Jun 9, 2021

alexcrichton left a comment

alexcrichton Jun 9, 2021

alexcrichton Jun 9, 2021

philipc Jun 10, 2021

alexcrichton Jun 9, 2021

philipc Jun 10, 2021

bjorn3 Jun 10, 2021

philipc Jun 10, 2021

philipc commented Jun 10, 2021

philipc commented Jun 10, 2021

alexcrichton commented Jun 10, 2021

philipc commented Jun 18, 2021

alexcrichton left a comment

elf: support external and supplementary debuginfo #427

elf: support external and supplementary debuginfo #427

Conversation

philipc commented Jun 9, 2021

Choose a reason for hiding this comment

alexcrichton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philipc commented Jun 10, 2021

philipc commented Jun 10, 2021

alexcrichton commented Jun 10, 2021

philipc commented Jun 18, 2021

alexcrichton left a comment

Choose a reason for hiding this comment