ref: document dvc queue

content/docs/command-reference/exp/remove.md

@@ -27,8 +27,8 @@ With `--queue`, the list of experiments awaiting execution is cleared instead.
 - `--queue` - remove all experiments that haven't been run yet (defined via
   `dvc exp run --queue`).
 
-- `-A`, `--all` - remove all experiments that have been run. Use `--queue` to
-  remove queued ones.
+- `-A`, `--all` - remove all experiments that have been run. Use
+  `dvc queue remove` to remove queued experiment tasks.
 
 - `--rev <commit>` - remove experiments derived from the specified `<commit>` as
   baseline.

content/docs/command-reference/exp/run.md

@@ -34,13 +34,9 @@ Use the `--set-param` (`-S`) option as a shortcut to change
 <abbr>parameter</abbr> values [on-the-fly] before running the experiment.
 
 It's possible to [queue experiments] for later execution with the `--queue`
-flag. To actually run them, use `dvc exp run --run-all`. Queued experiments are
-run sequentially by default, but can be run in parallel using the `--jobs`
-option.
-
-> ⚠️ Parallel runs are experimental and may be unstable. Make sure you're using
-> a number of jobs that your environment can handle (no more than the CPU
-> cores).
+flag. Queued experiments can be run using `dvc queue start`, refer to the
+`dvc queue` documentation for more information on managing the experiment task
+queue.
 
 It's also possible to run special [checkpoint experiments] that log the
 execution progress (useful for deep learning ML). The `--rev` and `--reset`
@@ -82,8 +78,7 @@ committing them to the Git repo. Unnecessary ones can be [cleared] with
   runs.
 
 - `--queue` - place this experiment at the end of a line for future execution,
-  but don't actually run it yet. Use `dvc exp run --run-all` to process the
-  queue.
+  but don't actually run it yet. Use `dvc queue start` to process the queue.
 
   > For checkpoint experiments, this implies `--reset` unless a `--rev` is
   > provided.
@@ -96,9 +91,13 @@ committing them to the Git repo. Unnecessary ones can be [cleared] with
   parallel. Only has an effect along with `--run-all`. Defaults to 1 (the queue
   is processed serially).
 
-  > Note that since queued experiments are run isolated from each other, common
-  > stages may sometimes be executed several times depending on the state of the
-  > [run-cache] at that time.
+  <admon type="warn">
+
+  `dvc exp run --run-all [--jobs]` is now a shortcut for
+  `dvc queue start [--jobs]` followed by `dvc queue logs -f`. The `--run-all`
+  and `--jobs` options will be deprecated in a future DVC release.
+
+  </admon>
 
 - `-r <commit>`, `--rev <commit>` - resume an experiment from a specific
   checkpoint name or hash (`commit`) in `--queue` or `--temp` runs.

content/docs/command-reference/queue/index.md

@@ -0,0 +1,39 @@
+# queue
+
+A set of commands to manage the
+[DVC experiments](/doc/user-guide/experiment-management/experiments-overview)
+task queue: [start](/doc/command-reference/queue/start),
+[stop](/doc/command-reference/queue/stop),
+[status](/doc/command-reference/queue/status),
+[logs](/doc/command-reference/queue/logs),
+[remove](/doc/command-reference/queue/remove),
+[kill](/doc/command-reference/queue/kill)
+
+## Synopsis
+
+```usage
+usage: dvc queue [-h] [-q | -v]
+                 {start,stop,status,logs,remove,kill} ...
+
+positional arguments:
+  COMMAND
+    start       Start experiments queue workers.
+    stop        Stop experiments queue workers.
+    status      List the status of the queue tasks and workers.
+    logs        Show output logs for a task in the experiments queue.
+    remove      Remove tasks in experiments queue.
+    kill        Kill tasks in experiments queue.
+```
+
+## Description
+
+`dvc queue` subcommands provide specialized ways to manage queued experiment
+tasks.
+
+## Options
+
+- `-h`, `--help` - prints the usage/help message, and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output.
+
+- `-v`, `--verbose` - displays detailed tracing information.

content/docs/command-reference/queue/kill.md

@@ -0,0 +1,41 @@
+## queue kill
+
+Kill actively running
+[DVC Experiment](/doc/user-guide/experiment-management/experiments-overview)
+tasks.
+
+## Synopsis
+
+```usage
+usage: dvc queue kill [-h] [-q | -v] [<task> ...]
+
+positional arguments:
+  <task>         Tasks in queue to kill.
+```
+
+## Description
+
+Forcefully stops execution of the specified (running) experiment tasks. Killed
+tasks will be considered as failed runs.
+
+This command does not stop the queue worker process. After the specified task
+has been killed, the worker process will consume and execute the next experiment
+task in the queue.
+
+To kill all running experiment tasks and also stop queue processing, you can use
+`dvc queue stop --kill`.
+
+<admon type="warn">
+
+Note that killed experiment tasks will be considered failed runs and will not be
+re-added to the queue for future execution.
+
+</admon>
+
+## Options
+
+- `-h`, `--help` - prints the usage/help message, and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output.
+
+- `-v`, `--verbose` - displays detailed tracing information.

content/docs/command-reference/queue/logs.md

@@ -0,0 +1,130 @@
+## queue logs
+
+Show output logs for running and completed tasks in the
+[DVC Experiment](/doc/user-guide/experiment-management/experiments-overview)
+task queue.
+
+## Synopsis
+
+```usage
+usage: dvc queue logs [-h] [-q | -v] [-e <encoding>] [-f] <task>
+
+positional arguments:
+  <task>                Task to show.
+```
+
+## Description
+
+Shows output logs for the specified running or completed experiment task.
+
+By default, this command will show any available log data and then exit. For
+tasks which are still running, the `--follow` option can be used to attach to
+the task and continuously show live log output, until the task has completed.
+
+When using the `--follow` option, it is safe to stop following output using
+`Ctrl+C` (or `SIGINT`). This will only cause the logs command to exit, and the
+experiment task will continue to be run in the background.
+
+## Options
+
+- `-e <encoding>`, `--encoding <encoding>` - text encoding for log output.
+  Defaults to the system locale encoding.
+
+  <admon type="warn">
+
+  Note that this option is used to specify the encoding of the experiment task
+  output (i.e. the output of pipeline stage commands), which may not always
+  match the encoding of your system terminal.
+
+  </admon>
+
+- `-f`, `--follow` - attach to task and follow additional live output. Only
+  applicable if the task is still running.
+
+- `-h`, `--help` - prints the usage/help message, and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output.
+
+- `-v`, `--verbose` - displays detailed tracing information.
+
+## Examples
+
+## Example: View logs for completed experiment tasks
+
+Let's say we have previously run some queued experiment tasks:
+
+```dvc
+$ dvc queue status
+Task     Name    Created    Status
+192a13c          04:15 PM   Failed
+753b005          04:01 PM   Success
+0bbb118          04:01 PM   Success
+1ae8b65          04:01 PM   Success
+
+Worker status: 0 active, 0 idle
+```
+
+We can view the output for both failed and successfully completed experiment
+tasks:
+
+```dvc
+$ dvc queue logs 192a13c
+'data/data.xml.dvc' didn't change, skipping
+Running stage 'prepare':
+> python src/prepare.py data/data.xml
+Traceback (most recent call last):
+  File "/Users/pmrowla/git/example-get-started/.dvc/tmp/exps/tmp217n0tjv/src/prepare.py", line 10, in <module>
+    raise AssertionError
+AssertionError
+ERROR: failed to reproduce 'prepare': failed to run: python src/prepare.py data/data.xml, exited with 1
+```
+
+```dvc
+$ dvc queue logs 0bbb118
+'data/data.xml.dvc' didn't change, skipping
+Stage 'prepare' is cached - skipping run, checking out outputs
+Updating lock file 'dvc.lock'
+
+Stage 'featurize' is cached - skipping run, checking out outputs
+Updating lock file 'dvc.lock'
+
+Stage 'train' is cached - skipping run, checking out outputs
+Updating lock file 'dvc.lock'
+
+Stage 'evaluate' is cached - skipping run, checking out outputs
+Updating lock file 'dvc.lock'
+
+To track the changes with git, run:
+
+    git add dvc.yaml scores.json roc.json params.yaml data/prepared data/data.xml prc.json src/featurization.py data/features src/evaluate.py model.pkl dvc.lock src/train.py src/prepare.py
+
+To enable auto staging, run:
+
+        dvc config core.autostage true
+```
+
+## Example: View logs for running experiment tasks
+
+Let's queue a new experiment and view the output while it is running:
+
+```dvc
+$ dvc exp run --queue -S prepare.split=0.40 -S featurize.max_features=4000
+Queued experiment '93cfa70' for future execution.
+$ dvc queue start
+Started '1' new experiments task queue worker.
+$ dvc queue logs 93cfa70
+'data/data.xml.dvc' didn't change, skipping
+Running stage 'prepare':
+> python src/prepare.py data/data.xml
+Updating lock file 'dvc.lock'
+
+Running stage 'featurize':
+> python src/featurization.py data/prepared data/features
+```
+
+We can see that by default, `dvc queue logs` displays any available output and
+then exits. In this case, our `featurize` stage is still running, so no
+additional output is available at this time.
+
+If we wanted to continuously view live output from the running task (until it
+completes) we also could have used the `--follow` option.

content/docs/command-reference/queue/remove.md

@@ -0,0 +1,48 @@
+## queue remove
+
+Remove queued and completed tasks from the
+[DVC Experiment](/doc/user-guide/experiment-management/experiments-overview)
+task queue.
+
+## Synopsis
+
+```usage
+usage: dvc queue remove [-h] [-q | -v]
+                        [--all] [--queued] [--success] [--failed]
+                        [<task> ...]
+
+positional arguments:
+  <task>         Tasks in queue to remove.
+```
+
+## Description
+
+Removes the specified queued or completed experiment tasks from the queue. For
+completed tasks, this will also remove any associated output logs.
+
+<admon type="warn">
+
+Note that for successfully completed tasks, this command is not the same as
+`dvc exp remove`. `dvc queue remove` does not remove any Git or DVC data
+associated with a successful DVC experiment. It only removes the task queue
+entry and any associated output logs for that task.
+
+</admon>
+
+## Options
+
+- `--all` - remove all (queued and completed) experiment tasks from the queue.
+
+- `--queued` - remove all queued experiment tasks from the queue.
+
+- `--success` - remove all successfully completed tasks (and associated output
+  logs) from the queue.
+
+- `--failed` - remove all failed tasks (and associated output logs) from the
+  queue.
+
+- `-h`, `--help` - prints the usage/help message, and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output.
+
+- `-v`, `--verbose` - displays detailed tracing information.

content/docs/command-reference/queue/start.md

@@ -0,0 +1,67 @@
+## queue start
+
+Start the
+[DVC experiments](/doc/user-guide/experiment-management/experiments-overview)
+task queue worker process.
+
+## Synopsis
+
+```usage
+usage: dvc queue start [-h] [-q | -v] [-j <number>]
+```
+
+## Description
+
+Starts one or more task queue worker processes. Each worker process will consume
+and execute one queued experiment task at a time in the background, until either
+`dvc queue stop` is used or the queue is empty.
+
+<admon type="info">
+
+Due to [internal limitations], when the queue is empty a worker may be idle for
+up to 10 seconds before exiting. If new experiment tasks are added to the queue
+during this time, the idle worker will resume processing them instead.
+
+[implementation limitations]:
+  /doc/user-guide/experiment-management/running-experiments#how-are-experiments-queued
+
+</admon>
+
+Queued experiment tasks are run sequentially by default, but can be run in
+parallel by using the `--jobs` option to start more than one worker.
+
+<admon type="warn">
+
+Parallel runs are experimental and may be unstable. Make sure you're using
+number of jobs that your environment can handle (no more than the CPU cores).
+
+Note that since queued experiments are run isolated from each other, common
+stages may sometimes be executed several times depending on the state of the
+[run-cache] at that time.
+
+</admon>
+
+## Options
+
+- `-j <number>`, `--<number>` - start up to this number of workers in parallel.
+  Defaults to 1 (the task queue is processed serially).
+
+  <admon type="info">
+
+  Note that if any queue worker processes have already been started, this
+  command will not start additional processes unless `number` is greater than
+  the number of existing workers (`number` is treated as the maximum allowed
+  concurrency value).
+
+  If `number` is less than the number of existing worker processes, this command
+  will not stop any existing worker processes. To reduce worker concurrency,
+  `dvc queue stop` must first be used to stop queue processing, before running
+  `dvc queue run` with the desired number of `--jobs`.
+
+  </admon>
+
+- `-h`, `--help` - prints the usage/help message, and exit.
+
+- `-q`, `--quiet` - do not write anything to standard output.
+
+- `-v`, `--verbose` - displays detailed tracing information.