[CAS] Add LLVMCAS library with InMemoryCAS implementation #114096
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Created using spr 1.3.5
cachemeifyoucan
requested review from
akyrtzi,
rnk,
bogner,
benlangmuir,
rastogishubham and
dwblaikie
ilovepi
reviewed
Oct 30, 2024
There was a problem hiding this comment.
This is a large patch, so I haven't read all the C++ closely enough to say anything about it. Mostly, I've left comments on the documentation, and pointed out a few things I saw while skimming the implementation.
I think overall a bit more explanation about how some of the key classes interoperate may be warranted. For instance I really don't understand why some of the reinterpret casts should be legal in InMemroyInlineObject.
The other thing I see is a pattern of foo(){return fooImpl();}, but fooImpl() isn't private or (from what I can see) an extension point. I'm not sure I understand the design choice here, since I don't recall seeing it used this way much in LLVM. Normally the Impls are either private, virtual, or in some other module/namespace. It's a big project, so maybe I've just missed something though.
I'll try to take a more detailed look at the implementation code, but I'd encourage others to look too, since this is a very large, complicated patch set.
Created using spr 1.3.5
Created using spr 1.3.5
ilovepi
reviewed
Nov 1, 2024
Created using spr 1.3.5
6 tasks
Created using spr 1.3.6
|
Ping! I am going to pushing to upstreaming CAS implementation again with the hope to land this before next dev meeting. @ilovepi @bogner @adrian-prantl Let me know if you want to take another look while I start working on rebasing and updating all the patches. |
Created using spr 1.3.6
cachemeifyoucan
requested review from
adrian-prantl,
rastogishubham and
ilovepi
adrian-prantl
approved these changes
Aug 8, 2025
|
Thanks @adrian-prantl . I will let it sit over the weekend in case anyone wants to make comments. |
ilovepi
reviewed
Aug 8, 2025
ilovepi
reviewed
Aug 8, 2025
Created using spr 1.3.6
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/203/builds/19563 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/206/builds/4666 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/205/builds/18352 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/207/builds/5164 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/204/builds/18375 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/76/builds/12066 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/160/builds/22828 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/180/builds/22971 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/145/builds/9019 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/42/builds/5770 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/118/builds/7759 Here is the relevant piece of the build log for the reference |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/97/builds/8104 Here is the relevant piece of the build log for the reference |
|
@cachemeifyoucan I'm seeing this MSVC warning since this patch please can you investigate? |
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/130/builds/14938 Here is the relevant piece of the build log for the reference |
|
The unit test associated with this commit appears to be running infinitely inside of CASTest.BlobsParallel with our downstream toolchain running on X86 Linux. I'm not sure where to begin with respect to figuring out why other than just to turn it off. |
|
@evodius96 Do you have a stack trace for the hang? Is it really hang or just take a long time? I can disable it but we have x86 linux bot (ubuntu) that runs this test without problem. |
|
@evodius96 I have a guess. Is it possible that you config llvm with LLVM_ENABLE_THREADS=Off? I should disable test in single thread mode. |
Yes, we do -- I believe that's the issue. Thanks! |
cachemeifyoucan
added a commit
that referenced
this pull request
Aug 13, 2025
|
We have also observed hangs in this test on s390x, ppc64le and i686 (but not x86_64 and aarch64). We're not using LLVM_ENABLE_THREADS=OFF. |
|
Okay, it looks like the test hangs whenever only two cores or less are available. Trivially reproducible using |
|
Checking getMaxConcurrency() would work, but really I don't think using ThreadPool is appropriate for what you are doing here. These are not "tasks" where it's okay to schedule the later ones only after the earlier ones have finished. You should probably be using std::thread directly. |
nikic
added a commit
that referenced
this pull request
Aug 18, 2025
This test hangs forever if executed with less than three cores available, see: #114096 (comment)
|
Test commented out in ba45ac6. |
llvm-sync bot
pushed a commit
to arm/arm-toolchain
that referenced
this pull request
Aug 18, 2025
This test hangs forever if executed with less than three cores available, see: llvm/llvm-project#114096 (comment)
|
@nikic thanks for reporting. I guess we were not yielding the thread so it dead-lock when the parallelism is only two. Maybe the simpler fix is to just call producer first since the concurrency behavior that is being tested wasn't guaranteed either way. |
|
Try fix and enable: #154151 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add llvm::cas::ObjectStore abstraction and InMemoryCAS as a in-memory
CAS object store implementation.
The ObjectStore models its objects as:
And each CAS Object can be idenfied with an unqine ID/Hash.
ObjectStore supports following general action:
It also introduces following types to interact with a CAS ObjectStore:
print/compare CASIDs.
implementation defined so it can be optimized for
read/store/references depending on the implementation.
inside CAS Object. It bundles a ObjectHandle and an ObjectStore
instance.