Sort dvc pipeline list #2669
Sort dvc pipeline list #2669
Conversation
| stages = nx.get_node_attributes(p, "stage") | ||
| for node in p: | ||
| stage = stages[node] | ||
| # if not stage.locked: |
There was a problem hiding this comment.
Please don't leave commented-out code like that. Looks like you don't need it, right? If so - remove it.
| for stage in stages: | ||
| logger.info(stage) | ||
| stages = nx.get_node_attributes(p, "stage") | ||
| for node in p: |
There was a problem hiding this comment.
Please correct me if I'm wrong, but what you are doing here is you are walking through all nodes in the pipeline in the random order, then you do a dfs_postorder for each one of those and then just print it. This is basically the same as what it was before but now with a redundant step of dfs_postorder_nodes. 🙂 Looks like we can do better than that 🙂
There was a problem hiding this comment.
Good way to think about it is to imagine how the pipeline looks. So it has a N start nodes (p.out_degree(node) == 0), M end nodes (p.in_degree(node) == 0) and some nodes between them. The way dvc repro --pipeline works, is it finds all end nodes and for each of them does dfs_postorder_nodes and just runs those. But obviously, that real order of execution is not going to look good here 🙁 Could we somehow order them better here? Maybe nx.topological_sort(p)?
There was a problem hiding this comment.
Thank you for the explanation. I'll work on it.
There was a problem hiding this comment.
p.out_degree(node) == 0 for end nodes and p.in_degree(node) == 0 for start nodes right?
There was a problem hiding this comment.
@SrividyaKK Actually no, our DAG has a reversed shape right now because we put edges in the direction from dependent stage to the one it depends on. So say your pipeline is cleanup step and then process step, then DAG would look like cleanup <- process. Just a caveat of the current design, we might change it in the future 😉
|
@SrividyaKK Need any help? 🙂 |
|
Closing due to inactivity. |
postorder traversal for issue #2588