[df] Support MT snapshotting to RNTuple #19599
Conversation
Test Results 20 files 20 suites 3d 19h 1m 23s ⏱️ For more details on these failures, see this check. Results for commit 6fd522f. ♻️ This comment has been updated with latest results. |
hahnjo
force-pushed
the
df-ntuple-snapshot-mt
branch
from
ced6d97 to
8a09b36
Compare
I am confused by the wording of this part of the commit log. |
Agreed, it's not well formulated. What I'm trying to say is that for sequential snapshotting (that is already supported before the PR), changing from the |
hahnjo
force-pushed
the
df-ntuple-snapshot-mt
branch
from
8a09b36 to
a0e8be9
Compare
Yes, I wasn't sure either. We're of course massively benefiting that we use the exact same model creation and filling code that you already wrote all the tests for. Additionally, we have the challenge of MT scheduling, so even the best test case will have to deal with non-determinism and potentially still not test the relevant thing because all events end up on a single thread... |
That's fair, not to forget about the fact that the behavior of the parallel writer is already tested in isolation. I'm okay with leaving this like this then! |
... instead of the default entry. Then we also only need a bare model.
This is less expensive than string comparisons of field names during every call to Exec().
hahnjo
force-pushed
the
df-ntuple-snapshot-mt
branch
from
a0e8be9 to
bd8b5ae
Compare
Switch the existing code to use the RNTupleParallelWriter with one RNTupleFillContext per slot. For sequential snapshotting, this should be (almost) as efficient as the RNTupleWriter (one additional cloned RNTupleModel for the only fill context), but save quite a bit of code duplication and in testing effort.
Use the same conditions as TTree, looking at fOutputFile instead of the data source.
hahnjo
force-pushed
the
df-ntuple-snapshot-mt
branch
from
bd8b5ae to
6fd522f
Compare
Switch the existing code to use the
RNTupleParallelWriterwith oneRNTupleFillContextper slot. For sequential snapshotting, this should be (almost) as efficient as theRNTupleWriter(one additional clonedRNTupleModelfor the only fill context), but save quite a bit of code duplication and in testing effort.