`exp show`: lockless issues

[mattseddon]

Bug Report

Description

After the recent changes to make exp show lockless I am seeing intermittent issues with the data returned during experiment runs.

The issues that I have seen so far are as follows:

Running checkpoint experiments from the queue

1. exp show fails when running experiments from the queue with dulwich.porcelain.DivergedBranches errors.
2. @pmrowla tried to patch the above but the change led to this behaviour.

Please take a look at the above behaviour and LMK what you think. I do not anticipate there being an easy fix. IMO we should consider "dropping support" for live tracking experiments run from the queue until the DVC mechanics have been updated.

Running checkpoint experiment in the workspace

1. exp show returns a single dict for a set of checkpoints during an experiment. This happens intermittently and breaks our "live experiment" tracking.
2. exp show shows running experiments as not running mid-run.

Reproduce

Run a checkpoint experiment and monitor the output of exp show.

Expected

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.9.9 on macOS-12.3.1-x86_64-i386-64bit
Supports:
        webhdfs (fsspec = 2022.3.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc (subdir), git

Additional Information (if any):

I'll continue to add issues here as I find them. I have discussed with @pmrowla already. Wanted to raise the visibility by raising an issue to discuss possible mitigation and the priority for fixes.

Thanks

[mattseddon]

I managed to catch the behaviour of a checkpoint experiment being collapsed to a single record without a name (and then returning to all checkpoint):

Screen.Recording.2022-05-05.at.8.35.24.am.mov

[karajan1001]

For the problem in Running checkpoint experiments from the queue

#7684 (comment)

Unexpected behaviour:

1. Only a couple of checkpoints are shown for each experiment until they are both finished and all are then copied to the workspace.
2. The running experiment record is shown in a different record to the rest of the experiment.
3. The running experiment record returns to queued.

Problems 2and3` had already been solved in #8369

For the problems in Running checkpoint experiment in the workspace

1. exp show returns a single dict for a set of checkpoints during an experiment. This happens intermittently and breaks our "live experiment" tracking.
2. exp show shows running experiments as not running mid-run.

Problem 2` also had already been solved in #8369 and for problem 1, I tried and it still exists.

So we can focus on problem 1 in both sections. BTW, the epochs finished so quickly in the demo (less than 1S) and that makes the live tracking can not be shown one by one.

[karajan1001]

For the problem.

1. Only a couple of checkpoints are shown for each experiment until they are both finished and all are then copied to the workspace.

I do some investigation on this problem yesterday. The problem comes from the refs downloaded from

https://github.com/iterative/scmrepo/blob/d175b923c76494c9023ee3581b349191ea2c8a6f/src/scmrepo/git/backend/dulwich/__init__.py#L658-L668

        try:
            fetch_result = client.fetch(
                path,
                self.repo,
                progress=DulwichProgressReporter(progress)
                if progress
                else None,
                determine_wants=determine_wants,
            )
        except NotGitRepository as exc:
            raise SCMError(f"Git failed to fetch ref from '{url}'") from exc

will cause

https://github.com/iterative/scmrepo/blob/d175b923c76494c9023ee3581b349191ea2c8a6f/src/scmrepo/git/backend/dulwich/__init__.py#L671-L681

        for (lh, rh, _) in fetch_refs:
            refname = os.fsdecode(rh)
            if rh in self.repo.refs:
                if self.repo.refs[rh] == fetch_result.refs[lh]:
                    result[refname] = SyncStatus.UP_TO_DATE
                    continue
                try:
                    check_diverged(
                        self.repo, self.repo.refs[rh], fetch_result.refs[lh]
                    )

raise DivergedBranches, and the problem behind this error is that the sha value fetch_result.refs[lh] does not exist in the repo and will cause KeyError. This will cause any updates (fetching) fails, only after the training progress is finished can the fetching succeed. This might come from the new commits coming to fast in the temp workspace because if we add a time gap into the training progress, for example, adding time.sleep(5) in each training epoch, the fetching can success and the training can be shown in living progress.

So if we want to solve this problem completely we might need to go deep into the dulwich's fetching progress and make sure the revision returned is also properly downloaded and exists. And if I use force update here,

dvc/dvc/repo/experiments/queue/utils.py

Lines 41 to 42 in 43a8eab

	def on_diverged(_ref: str, _checkpoint: bool):
	return False

It will raise Invalid commit Exception.

[karajan1001]

might be solved after #8477 if we use pygit as new backends for fetch.

[karajan1001]

I did some research today, and found that the problem I force it to update comes from we are using different backends, the scm.resolve_commit is pygit backend and it can not properly resolve the force updated commit fetched by from dulwich .

Confirmed that the problem comes from Dulwich. because I saw the ref fetched by Dulwich invalid.

git commit                                                                                                                                           
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
error: refs/exps/2a/1c3a54209af04dda29df9fc9309d8e93fcaf45/exp-710c6 does not point to a valid object!