UI progress cleanup

[casperdcl]

• remove <00:00/<??:?? from completed bars
• don't use SI prefixes for small totals < 999 (i.e. 1.00/32.0 files => 1/32 files)
• don't increment bar when a target fails (backtrack bar)
• make sure we don't show an extra progress bar when there is only one target (majority of cases, see below)
• large dirs progress bar hanging when collecting files before computing checksums progress when collecting directories with giant number of files #2770 add: cleanup and improve output #2699 (comment)
• cleanup output when if fails add: cleanup and improve output #2699 (comment)
• add before/after recordings
• fixes add: cleanup and improve output #2699
• continued in UI progress #3060

Extra progress bar concern:

We started using an extra progress bar that is counting targets, as described and shown here: #2658 . In most cases though there is a single target - dvc add file or dvc add directory. That extra progress bar is not needed in those cases. It's needed in case of multiple targets or if -R option is specified.

most issues fixed:

large dir fix, no outer bar for single target, more progress output rather than hanging:

recursive:

[casperdcl]

@efiop could you list scenarios that you refer to for bad (too little or too much verbosity)? e.g.:

• dvc add single_file_over_10_gb
• dvc add single_dir_containing_one_file_over_10_gb_and_lots_of_small_files_under_1kb
• dvc add single_dir_containing_many_small_files_total_over_10_gb

Also you mentioned RemoteBASE so I presume fetch/push/pull are issues too?

[efiop]

@casperdcl Correct, I think you've listed them all. Same applies to push/pull/fetch/checkout too.

[casperdcl]

@efiop should we also consider these despite not being recommended?

• dvc add -R single_dir_containing_many_small_files_total_over_10_gb
• dvc add single_dir_containing_many_small_files_total_over_10_gb/*.*

[efiop]

@casperdcl Good point! What are your thoughts on that? 🙂

[casperdcl]

er... just wondering whether I should bother profiling them too to find out where more progress/feedback is required for users

[efiop]

@casperdcl I guess both of those cases are dvc add small_file times N. dvc add small_file is a base scenario and is handled already and times N part is handled by the progress bar that you've introduced earlier. So unless I'm missing something, we are already done there (or will be done indirectly).

[casperdcl]

@efiop there is much fun to be had with that last commit (a941967371b4839f629e6a253792178624c016bb), I added progress everywhere except check_modified_graph - should actually catch all cases I think. Might not even want to revert any of that additional feedback.

[casperdcl]

So stage.commit() is a big time-consuming culprit which we hadn't added any progress for. Turns out copying files takes as long as reading & computing md5sums 🙃

[shcheklein]

A bit offtopic, sorry about that. Just to share some possible solutions to the UI when there are a few stages (top level progress, below done with [1/4], [2/4] .. ), each of stages includes a progress bar (name of the stage + remove the bar when stage is done), multiple threads progress (in this case show them all, but they are very different - probably application specific):

https://asciinema.org/a/rUwhTMR06Rz7MIFEeFxFsjrHd

[casperdcl]

@shcheklein

cleanup output when if fails #2699 (comment)

when does this happen?

make sure that we show HINT message to enable cache optimization properly when copy is taking too long already

in what case(s) does the hint (I assume it's a message recommending reflink caches) fail to show?

[shcheklein]

@casperdcl

when does this happen?

Try just some random path:

$ dvc add asedfsdfsd

in what case(s) does the hint (I assume it's a message recommending reflink caches) fail to show?

yes, this is the "reflink" message. cache.slow_link_warning here https://dvc.org/doc/command-reference/config#cache

@pared has more context about the limits we have now. You can try some huge file with a cache set to "copy". The purpose I created that checkbox for is to reconsider this policy, the way the message is written in the context of recent changes with progress bars, etc. It should be clearly visible if we hit a threshold (we can keep it the same for now?) along with progress bar (below them?).

[casperdcl]

@shcheklein right.

For dvc add nonexistent_file, what should the output be? 1/1 is bad imho. Should we go for 0/0 or 0/1 though?

About the reflink message, it uses logger.warning so plays fine with the progressbars (

dvc/dvc/logger.py

Lines 197 to 198 in e031366

	"class": "dvc.logger.LoggerHandler",
	"level": "WARNING",

, yes will appear above the bars).

About changing the default thresholds, I think that should be a separate issue.

[codecov]

Codecov Report

Merging #2846 into master will decrease coverage by 0.92%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2846      +/-   ##
==========================================
- Coverage   92.66%   91.73%   -0.93%     
==========================================
  Files         139      138       -1     
  Lines        8726     8704      -22     
==========================================
- Hits         8086     7985     -101     
- Misses        640      719      +79

Impacted Files	Coverage Δ
dvc/remote/gdrive.py	30.53% <0%> (-62.69%)	⬇️
dvc/lock.py	58.66% <0%> (-9.39%)	⬇️
dvc/dependency/repo.py	88.88% <0%> (-9.3%)	⬇️
dvc/istextfile.py	94.11% <0%> (-5.89%)	⬇️
dvc/command/get.py	87.5% <0%> (-4.4%)	⬇️
dvc/scm/tree.py	93.1% <0%> (-3.57%)	⬇️
dvc/version.py	83.33% <0%> (-2.78%)	⬇️
dvc/remote/ssh/connection.py	84.23% <0%> (-2.77%)	⬇️
dvc/repo/get.py	97.82% <0%> (-2.18%)	⬇️
dvc/utils/fs.py	94.84% <0%> (-2.08%)	⬇️
... and 109 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a42d4f2...ab115df. Read the comment docs.

[casperdcl]

rebased, fixed conflicts; updated description with before & after recordings.

[efiop]

Looks pretty good, thanks @casperdcl ! Let's merge and iterate over.