Dependency scanner does not recognize one-line dependencies

[Lastique]

Originally reported in boostorg/filesystem#226.

If CMakeLists.txt of a library contains one-line dependency such as this:

target_link_libraries(boost_filesystem PRIVATE Boost::atomic)

the dependency scanner does not recognize this. This affects Boost.Filesystem, so issuing this command on Boost root:

cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug -DBOOST_INCLUDE_LIBRARIES=gil -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DBoost_VERBOSE=ON ..

results in this error:

CMake Error: install(EXPORT "boost_filesystem-targets" ...) includes target "boost_filesystem" which requires target "boost_atomic" that is not in any export set.

The dependency on Boost.Atomic is here.

[Flamefire]

I'll look into this. Got an idea already.

[pdimov]

The dependency scanner at the moment only recognizes "Boost::foo" in a line of its own, it doesn't contain a full CMake parser.

[Flamefire]

Checked the relevant sources of the includer/scanner. My approach would change it quite a lot, so I want to put it up for discussion first:

• Generate a list of libraries to add based on BOOST_EXCLUDE_LIBRARIES, BOOST_ENABLE_MPI, BOOST_ENABLE_PYTHON and BOOST_INCLUDE_LIBRARIES, basically what is done (minus the deps) in

  cmake/include/BoostRoot.cmake

  Line 256 in bd981bf

  if(__boost_lib IN_LIST BOOST_INCOMPATIBLE_LIBRARIES)

• For each of those call add_subdirectory and add them to a "handled" list
• Add all to a list of libraries to scan (for dependencies)
• while this list isn't empty, pop the first and do the following:
  • Get the correct target using a heuristic, e.g.

    cmake/include/BoostRoot.cmake

    Line 146 in bd981bf

    if(TARGET "Boost::${__boost_lib_target}" AND TARGET "boost_${__boost_lib_target}")

  • Get the LINK_LIBRARIES property of this target and combine with INTERFACE_LINK_LIBRARIES skipping duplicates
  • loop over this list and search for entries prefixed Boost:: that are not in the "handled" list
  • include the libraries CML
  • add it to the handled and to-scan lists

I'm sure it works for all libraries where it currently works and now also handles cases as in filesystem. It requires authors to follow a naming convention so we can convert between path-name and target name, but at least for the auto-installation to work this is already the case (although a soft failure, which I'd make a hard failure as otherwise we'll miss dependencies):

cmake/include/BoostRoot.cmake

Line 146 in bd981bf

if(TARGET "Boost::${__boost_lib_target}" AND TARGET "boost_${__boost_lib_target}")

However the current scanner includes(supposed) dependencies that are conditional, i.e. atomic, if formatted correctly: https://github.com/boostorg/filesystem/blob/41d076ace558cfae01e233f9746dc955dcf78dba/CMakeLists.txt#L195-L198

The described approach would only include atomic if it is actually added as a dependency, i.e. earlier than C++20-mode. I'd see this as a bonus.

Any comments? Possible issues?

Quick solution: Enhance the heuristic at

cmake/include/BoostRoot.cmake

Line 176 in bd981bf

if(line MATCHES "^[ ]*Boost::([A-Za-z0-9_]+)[ ]*$")

to allow cases as in filesystem. However that may break again, e.g. for multiple deps on one line.
The above described solution uses CMake to handle the deps, so all valid CMake constructs will be recognized and conditional dependencies only included, when actually added. That might cause issues too, i.e. if someone adds/links Boost::filesystem and assumes Boost::atomic to be available even in C++20 mode which due to the if-condition above won't be with the new algorithm.

[pdimov]

That's possible and I've had this in mind as an alternative, but it has its own issues, although on balance it may still be better than what we have now.

It will change the order of add_subdirectory calls, but only sometimes, so order dependence will sometimes work and sometimes not. It also doesn't handle libraries that aren't named "correctly", such as Test and Stacktrace, but maintaining an exception list for those is probably more manageable than fixing the scanner to support whatever people write.

[pdimov]

Oh, it also breaks one interesting case that currently works and is in use: setting BOOST_INCLUDE_LIBRARIES and BOOST_EXCLUDE_LIBRARIES to the same value of e.g. lib. This means, include the dependencies of lib, but not lib itself, which can't be done with the above approach.

[Flamefire]

It will change the order of add_subdirectory calls, but only sometimes, so order dependence will sometimes work and sometimes not.

I think CMLs of independent libs relying on order are ill-formed. Any dependency between 2 (boost) libs more than target_link_libraries should be changed, if there are such cases. And if not the include-dependency shouldn't matter. Am I missing anything here?

Oh, it also breaks one interesting case that currently works and is in use: setting BOOST_INCLUDE_LIBRARIES and BOOST_EXCLUDE_LIBRARIES to the same value of e.g. lib. This means, include the dependencies of lib, but not lib itself, which can't be done with the above approach.

Hm, interesting indeed. Where is this used? However I think we could say that this is be undefined behavior and worked by accident ;-) I mean: What is a system supposed to do when told to do opposites?
Anyway: Indeed this cannot be supported by the "new" algorithm as in order to get the dependencies you need to create the target so you can check it. Best we can use is to add a custom exclude-from-all list, which gets close to that behavior.

As a slight alternative: How about using:

set(boost_regex "Boost::([A-Za-z0-9_]+)")
file(STRINGS "${BOOST_SUPERPROJECT_SOURCE_DIR}/libs/${lib}/CMakeLists.txt" data REGEX .*${boost_regex}.*)
string(REGEX MATCHALL ${boost_regex} deps ${data})

And then do the loop at

cmake/include/BoostRoot.cmake

Lines 174 to 183 in bd981bf

	foreach(line IN LISTS data)
	if(line MATCHES "^[ ]*Boost::([A-Za-z0-9_]+)[ ]*$")
	string(REGEX REPLACE "^numeric_" "numeric/" dep ${CMAKE_MATCH_1})
	list(APPEND result ${dep})
	endif()
	endforeach()

over ${deps} but replace the regex by the above variable.
I.e. search the whole file for possible boost libs anywhere.

Again conditional dependencies will be always included but the only requirement would be to not use a variable (e.g. link_libraries(Boost::${thisOrThat}))

I think I still prefer the CMake-Target-approach as it does not introduce any kind of subdialect library-authors would need to follow and any valid CMake suffices.

[pdimov]

Hm, interesting indeed. Where is this used?

Here: https://github.com/boostorg/json/blob/21fcc880304c754aeb01421318e06563484c72ac/CMakeLists.txt#L42-L53

[pdimov]

As a slight alternative: How about using

I fear that this will pick up more than we need, although it's an option if all else fails.

[Flamefire]

Here: https://github.com/boostorg/json/blob/21fcc880304c754aeb01421318e06563484c72ac/CMakeLists.txt#L42-L53

Nifty! I'm actually surprised this works... Couldn't that still be done by putting that at the top without the exclude, guarding the if against re-entry and returning after including the super-project CML? Basically abuse that file(-part) as a launcher script for the super-project build.

I fear that this will pick up more than we need, although it's an option if all else fails.

For example? We filter already against the existing libs via the paths.
Ok comments could be a problem...

[Lastique]

I do like the idea of using CMake to discover all dependencies. This solution is future-proof, too.

As a slight alternative: How about...

I think, if the scanner has to parse CMakeLists.txt by itself, it should be more robust than that. I.e. it should recognize link_libraries/target_link_libraries (non-commented and written in any valid way) and parse its arguments rather than picking random mentions of Boost:: strings, whether they are actual targets or comments or strings or whatever else. Although even recognizing target_link_libraries isn't perfect, as that won't work when the target name(s) are in a variable and would not reflect conditional dependencies. Still, that would be better than the current implementation.

[pdimov]

I'm not particularly thrilled by the prospect of having to perform proper tokenization and maintain state across lines in order to parse every possible target_link_libraries call. Besides, the current approach actually works for more than that, e.g. https://github.com/boostorg/test/blob/01ffbe2521bbbc8fa646e9a4fbf68a5c89241de0/CMakeLists.txt#L11-L30

[Flamefire]

Yeah, either we simply use the enhanced regex matcher and maybe filter comment lines or do the "proper" approach with targets. Writing a cmake parser in cmake is to much.

[mloskot]

The conflict between maintainer hand rolling free-form CML and strict-form expectations from Boost CMake is inevitable.

A not entirely crazy idea: split the two sets of CML-s into mandatory generated CMakeLists.boost.txt and optional manually maintained by a lib maintainer CMakeLists.txt

[Lastique]

A not entirely crazy idea: split the two sets of CML-s into mandatory generated CMakeLists.boost.txt and optional manually maintained by a lib maintainer CMakeLists.txt

I'm not sure how that would work with non-trivial dependency selection. Such as what Boost.Filesystem does.