@aapeliv's issue -- Let's work together!

[cpsauer]

Hi @aapeliv,

Thanks for giving this tool a whirl! I saw you'd starred. But I also noticed you'd pushed a commit to a fork, presumably to work around an issue you were running into.

Any chance we could work together to resolve in the main repo whatever problem you were running into? Would love to understand and fix what went wrong both for you and for others in the future who might run into it.

I'd love to at least hear what your setup is, and what went wrong. Bonus points if you'd be willing to help reproduce the issue or get us to a fix!

Thanks!
Chris

[aapeliv]

Hi @cpsauer! Of course, glad to help and be helped. Just didn't want to expect help for randos like me on an open source project :P. I'll add some details on this issue in a sec when I reproduce this.

[aapeliv]

Here's the failing output:

$ bazel run //:refresh_compile_commands
INFO: Analyzed target //:refresh_compile_commands (5 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:refresh_compile_commands up-to-date:
  bazel-bin/refresh_compile_commands.sh
INFO: Elapsed time: 0.431s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
 >>> Analyzing commands used in //cc/... 
Traceback (most recent call last):
  File "/home/aapeli/.cache/bazel/_bazel_aapeli/e96571fd1c49704bf5db6143f7865193/execroot/__main__/bazel-out/k8-fastbuild/bin/external/hedron_compile_commands/extract.runfiles/hedron_compile_commands/extract.py", line 208, in <module>
    for files, command in outputs:
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/aapeli/.cache/bazel/_bazel_aapeli/e96571fd1c49704bf5db6143f7865193/execroot/__main__/bazel-out/k8-fastbuild/bin/external/hedron_compile_commands/extract.runfiles/hedron_compile_commands/extract.py", line 189, in _get_cpp_command_for_files
    files = _get_files(args)
  File "/home/aapeli/.cache/bazel/_bazel_aapeli/e96571fd1c49704bf5db6143f7865193/execroot/__main__/bazel-out/k8-fastbuild/bin/external/hedron_compile_commands/extract.runfiles/hedron_compile_commands/extract.py", line 81, in _get_files
    assert len(source_files) > 0, f"No sources detected in {compile_args}"
AssertionError: No sources detected in ['/usr/bin/clang', '-U_FORTIFY_SOURCE', '-fstack-protector', '-Wall', '-Wthread-safety', '-Wself-assign', '-fcolor-diagnostics', '-fno-omit-frame-pointer', '-MD', '-MF', 'bazel-out/k8-fastbuild/bin/external/boringssl/_objs/crypto/chacha-x86_64.pic.d', '-fPIC', '-iquote', 'external/boringssl', '-iquote', 'bazel-out/k8-fastbuild/bin/external/boringssl', '-isystem', 'external/boringssl/src/include', '-isystem', 'bazel-out/k8-fastbuild/bin/external/boringssl/src/include', '-Wa,--noexecstack', '-D_XOPEN_SOURCE=700', '-Wall', '-Werror', '-Wformat=2', '-Wsign-compare', '-Wmissing-field-initializers', '-Wwrite-strings', '-Wshadow', '-fno-common', '-std=c11', '-Wmissing-prototypes', '-Wold-style-definition', '-Wstrict-prototypes', '-no-canonical-prefixes', '-Wno-builtin-macro-redefined', '-D__DATE__="redacted"', '-D__TIMESTAMP__="redacted"', '-D__TIME__="redacted"', '-c', 'external/boringssl/linux-x86_64/crypto/chacha/chacha-x86_64.S', '-o', 'bazel-out/k8-fastbuild/bin/external/boringssl/_objs/crypto/chacha-x86_64.pic.o']
 >>> Finished extracting commands for //cc/... 

I suppose that's because this is some machine-optimized/dependent assembly for encryption so there's no C/C++ source. I think the dependency might be gRPC (I can isolate this and make a minimal failing WORKSPACE as well).

[cpsauer]

You're great :)

Thanks so much for replying so fast--and scoping it to .s assembly files. I'd failed to consider that case! Also love the PR and your overall willingness to help :)

I was taking some time over here to think about what we should do to handle the other assembly file extensions--and test how clang handled them. I'm also realizing I should head off the CUDA case before it emerges. I'll accept your pull and then push up those changes, too.

[cpsauer]

Great! Done. Glad we got this fixed for you, and I'm sure it's going to help a bunch of other folks in the future. (Conversely, if it's still not working for you in some other way, please let me know!)

If you're down to share, I'd love to hear how it's working for you otherwise--and what you're using it for. Helps me know how to make it better and understand it's usefulness. If there are any rough edges you see, I'd love to hear about 'em!

[cpsauer]

Also, @aapeliv, I'm relatively new to being on the maintainer side of things. Anything I could have done, e.g. in the README, that'd have encouraged you to engage right away to help and be helped?

[aapeliv]

Thanks, appreciate it.

If you're down to share, I'd love to hear how it's working for you otherwise--and what you're using it for.

Seems to be working fine. Just using it for some HPC stuff for my PhD.

Also, @aapeliv, I'm relatively new to being on the maintainer side of things. Anything I could have done, e.g. in the README, that'd have encouraged you to engage right away to help and be helped?

Well, I had a look. I noticed the license is "odd"... so I had a read. I'd really discourage you from doing a "custom" license. There are a bunch of licenses and basically none of them open you/your company up for libability. I'd recommend you instead use the MIT license, or if you're really paranoid, use Apache 2.0; that has clauses about patents, etc. The issue with the current license is that any company looking to use this probably won't go through a manual review process through their legal dept and will just ignore it instead of potentially polluting their codebase. Honestly I wouldn't have contributed either if I realized what the license is. In short, just pick one of the standard licenses.

[cpsauer]

Sweet! Glad to hear.

Thanks for your feedback and for taking the time to reply :)

[I hear you on the license, and your advice+preference for something non-custom! Thank you.
I totally get that there's some cost to trying to innovate here, but I'd still like to give it a shot, falling back if eventually necessary. I think there might be a path to opening up more software this way, preventing traps that otherwise keep things closed.
If interesting, another original goal was to prevent rent-seeking, while still making the software maximally useful to others (implying not GPL; instead terms that would make legal smile in a review, rather than feel "polluted"). Hence this (short, readable) hybrid of ideas from MIT, good ways of making source-available from the gaming world, and a (I think) much more palatable version of preventing rent-seeking than the GPL or other options. I thought it was worth a quick try, while also (hopefully) getting everyone what they wanted, including free use of the tool. If I've failed--and there are terms blocking you from doing what you'd like--lmk so we can better it! Also always available to you (or anyone else reading this) by email--addresses listed in my commits.]

Good luck with your HPC studies, Aapeli! Cheering you on in all you do

[kgreenek]

+1 for a standard license. Using a standard permissive license like MIT or Apache protects you from liability and prevents others from even having to do a legal review at all. I don't think many people (myself included) would deem it a good use of money to pay their lawyers to review a one-off license for such a small dependency.

Anyways, this is a great project! I've been playing around with it in the shadows. Thank you for making it available.