Nested Looping in Parameterization

[bharathc346]

This feature request is in regards to the experimental parameterization feature here: https://github.com/iterative/dvc/wiki/Parametrization

Currently if in params.yaml if one has

  models:
    a:
      features: 1
    a:
      features: 2
    b:
      features: 1
    b:
      features: 2

we can have dvc.yaml as so

stages:
  build:
    foreach: ${models}
    do:
      cmd: >- 
          python script.py
          --model ${item}
          --features ${item.features}

Currently dvc.yaml can only loop over one thing e.g models in this case. I am wondering if something like this would be possible:

# params.yaml

models: 
  - a
  - b
features: 
  - 1
  - 2

and

# dvc.yaml

stages:
  build:
    foreach: ${models}
    foreach: ${features}
    do:
      cmd: >- 
          python script.py
          --model ${item_0}
          --features ${item_1}

This would be great as one wouldn't have to specify all possible configurations in params.yaml.
@skshetry tagging you as I believe you have been working on parameterization.

[skshetry]

@bharathc346, we cannot have two foreach as it's not supported in YAML format.
We could support nested foreach as following:

stages:
  build:
    foreach: ${models}
    do:
      foreach: ${features}
      do:
        cmd: >- 
            python script.py
            --model ${item_0}
            --features ${item_1}

But, as you can see, this requires too much indentation. Though, this is what we have in mind right now.

@johnnychen94 suggested introducing something similar to Github's matrix or travis's env in #3633 (comment). But, we are not sure if matrix is more intuitive to our users than the loops. With matrix, it'd look something like following:

stages:
  build:
    matrix:
      - ${models}
      - ${features}
    do:
      cmd: >- 
          python script.py
          --model ${item_0}
          --features ${item_1}

Regarding item_0/item_1 naming, I think it might be better to do similar to what Ansible does.
So, item_0 will be accessible as item[0] and item_1 as item[1]. item will just be a list. Similarly, key[0], key[1], etc. This might make it confusing with item and key as they are now a list, but I don't have any good ideas for now.

Anyway, @bharathc346 thanks for creating the issue. Let's hear some feedback for 2.0 change and given enough interests, we'll come to it. Thanks.

[bharathc346]

I see. I like @johnnychen94 comment about introducing the github matrix. But I think any solution for nested interpolation would be great. Thanks for the detailed comment

[skshetry]

@bharathc346, ahh, looks like I used the wrong term. It should be nested foreach.

[courentin]

Upvoted, that definitely something that would be nice to have.

Intuitively I was doing something like that:

stages:
  build:
    foreach:
      - fr:
        - a
        - b
      - en:
        - a
        - b
    do:
    ...

However, I feel the matrix is more readable. But I'd be ok with a nested foreach too.

[ClementWalter]

I am definitely interested in such a feature. If I were to vote on the naming convention, I would go for matrix as foreach means to me some sort of a parallelizable loop (for - each).

For the keys and items, I would stick with what is well done with the current foreach loop, meaning:

• if a list is provided, then the var is item and since there is a matrix, item is a list:

stages:
  build:
    matrix:
      - ["us", "fr", "gb"]
      - [128, 256, 512]
    do:
      cmd: >-
          python script.py
          --model ${item[0]}
          --features ${item[1]}

• if a dict is provided, item is a dict to access the values by their names (this second option improves readability and I could even vote to enforce it)

stages:
  build:
    matrix:
      model: ["us", "fr", "gb"]
      features: [128, 256, 512]
    do:
      cmd: >-
          python script.py
          --model ${item.model}
          --features ${item.features}

[kiedanski]

Hi, any updates on this feature? I would be really interested in this

[ClementWalter]

I am glad to work on it as well if we have a go from the dvc team

[johnnychen94]

The matrix loop is a very trivial case in that it introduces a dense parameter grid. There will always be a need to run a "filter" on this grid to reduce unnecessary tasks.

It would be nice if DVC defines an interchangeable protocol for multiple parameterization tasks so that people can write their own task generation script when needed. The "params.yaml" example I provide in #3633 (comment) is a very naive version of the interchangeable protocol. It would be very trivial work to dump "matrix" definition into something like this, as long as the protocol is well-defined.

#5795 should be also fixed to enable wide usage of this requested feature.

[dberenbaum]

Overlap with dvc exp run

There is some overlap between this kind of parametrized loop and dvc exp run --set-param. The current foreach approach creates many separate stages, while dvc exp run creates many variations of the same stage. @johnnychen94 mentioned concurrent execution in another thread, which dvc exp run is already starting to tackle. If it were possible to programmatically define the set of experiments to run with dvc exp run, would people still need matrix?

Nested loop syntax

There seems to be consensus that the matrix syntax makes sense. Personally, I agree with @ClementWalter that enforcing a dict structure is most readable.

Adding matrix might make it challenging to explain the difference between foreach and matrix. This also relates generally to how to balance complexity and flexibility in defining DAGs (see #5646 (comment)).

[karajan1001]

@dberenbaum, For myself, the foreach is a useful functionality when I'm doing some text classifications. In this work, two billion items need to be classified into about 5000 child categories inside 50 parent categories. The project has two levels, in the first one I trained a model to classify items into parent categories, and in the second I use foreach to train a separate model for the classification of the child categories for each parent category. It could not be replaced by exp or other functionalities.

So, If we are facing a project with an even more level of structure, then a nested loop might be an unavoidable solution.