Weekly Digest (28 June, 2020 - 5 July, 2020)

[weekly-digest]

Here's the Weekly Digest for iterative/dvc:

---

ISSUES

Last week 40 issues were created.
Of these, 25 issues have been closed and 15 issues are still open.

OPEN ISSUES

💚 #4167 ssh: support scp-like relpaths in urls, by Kuluum
💚 #4166 Add more tests according to .gitignore, by karajan1001
💚 #4165 daemon: launch cmd is broken when running main() from elsewhere, by efiop
💚 #4164 dvc push doesn't recognise that files are missing in remote storage, by dldx
💚 #4162 Developping a workflow for simultaneous work, by bjonnh
💚 #4160 'dvc add dir' fails when 'dir' contains a file and a relative symlink on this file , by CRodVig
💚 #4159 dvc status reports changed checksum when it should not, by jnd77
💚 #4153 keep track of CVE score for our dependencies, by efiop
💚 #4146 hooks: pylint: fix pre-commit, by casperdcl
💚 #4145 hooks: pylint: pre-commit failure, by casperdcl
💚 #4144 get-url and import-url doesn't seem to work with S3 buckets anymore., by anotherbugmaster
💚 #4143 Artifact does not get added to .gitignore on failed multi-add, by pared
💚 #4139 push: OOM when uploading 1.5TB(3.5Mfiles) dataset to local remote, by lenrys29
💚 #4136 conf: credentialpath is not recalculated when it's a relpath, by Suor
💚 #4131 Issues with dvc get and dvc get-url after 1.0, by honnibal

CLOSED ISSUES

❤️ #4163 tests: create remote fixture, by efiop
❤️ #4161 jsonschema: generate from schema, by skshetry
❤️ #4158 dvc plots diff: use a descriptive filename instead of plots.html by default, by xnutsive
❤️ #4157 gdrive: pop prefix kwarg, by efiop
❤️ #4156 status: pass tree as a kwarg, by efiop
❤️ #4155 [RFC] tests: make read_text() work with directories, by efiop
❤️ #4154 dvc add --external fails on files created by gsutil compose, by haishiro
❤️ #4152 [WIP] subrepo: get/import/list support, by skshetry
❤️ #4151 Endpoint URL is not taken into account when adding an external file from Minio, by lucasmaheo
❤️ #4150 DVC status fails after using add -R, by moshe-rl
❤️ #4149 s3,gs: don't append "/" to prefixed path when listing objects, by pmrowla
❤️ #4148 completion: update dependency, by casperdcl
❤️ #4147 azure: use knack directly instead of azure-cli-core, by efiop
❤️ #4142 GC encounters unexpected error, by WestXu
❤️ #4141 dvc pull does not seem to work for files whose md5 starts with 00, by jnd77
❤️ #4140 tests: remotes: use TmpDir-like fixtures, by efiop
❤️ #4138 Reorder package widgets, by dmpetrov
❤️ #4137 azure: use CLI config, by efiop
❤️ #4135 remote: repo.cloud.pull() errors out with "'NoneType' object has no attribute 'execute'", by Suor
❤️ #4134 Consider using an alternative for ruamel.yaml due to critical CVE, by stefanvangastel
❤️ #4133 RepoTree: preserve relative paths in copytree(), by pmrowla
❤️ #4132 rename .dvc file to .lock file, by skshetry
❤️ #4130 metrics: skip missing metrics silently, by Suor
❤️ #4129 azure: use ~/.azure/config, by dmpetrov
❤️ #4128 hdfs: implement makedirs separately, by efiop

LIKED ISSUE

👍 #4140 tests: remotes: use TmpDir-like fixtures, by efiop
It received 👍 x1, 😄 x0, 🎉 x1 and ❤️ x1.

NOISY ISSUE

🔈 #4134 Consider using an alternative for ruamel.yaml due to critical CVE, by stefanvangastel
It received 15 comments.

---

PULL REQUESTS

Last week, 17 pull requests were created, updated or merged.

UPDATED PULL REQUEST

Last week, 3 pull requests were updated.
💛 #4146 hooks: pylint: fix pre-commit, by casperdcl
💛 #4120 Optimize ignore performance, by karajan1001
💛 #4114 version: redesign output format for the dvc version command #3499, by sahilbhosale63

MERGED PULL REQUEST

Last week, 14 pull requests were merged.
💜 #4163 tests: create remote fixture, by efiop
💜 #4157 gdrive: pop prefix kwarg, by efiop
💜 #4156 status: pass tree as a kwarg, by efiop
💜 #4155 [RFC] tests: make read_text() work with directories, by efiop
💜 #4149 s3,gs: don't append "/" to prefixed path when listing objects, by pmrowla
💜 #4148 completion: update dependency, by casperdcl
💜 #4147 azure: use knack directly instead of azure-cli-core, by efiop
💜 #4140 tests: remotes: use TmpDir-like fixtures, by efiop
💜 #4138 Reorder package widgets, by dmpetrov
💜 #4137 azure: use CLI config, by efiop
💜 #4133 RepoTree: preserve relative paths in copytree(), by pmrowla
💜 #4130 metrics: skip missing metrics silently, by Suor
💜 #4128 hdfs: implement makedirs separately, by efiop
💜 #4122 pylint: enable additional checks, by skshetry

---

COMMITS

Last week there were 21 commits.
🛠️ dvc: bump to 1.1.7 by efiop
🛠️ tests: create remote fixture (#4163) Usually we need it to parametrize tests with, so there is no
reason to duplicate the code. I've left local_remote fixture
just because it is so convenient and often used in a lot of tests. by efiop
🛠️ [status: pass tree as a kwarg (#4156) * [RFC] tests: make read_text() work with directories

This PR is just a showcase to see how we like it. I've converted a few
tests to it, but there will be much more to come.

assert (tmp_dir / "file") == "contents"
assert (tmp_dir / "dir").read_text() == {"subdir": {"file": "hello"}}

This approach is nicer because it allows us to use a declaratory
asserts instead of relying on fs files. The problem becomes much more
apparent once you get into remote workspaces, where we trees_equal
doesn't work.

• status: pass tree as a kwarg

Fixes #4150

Depends on https://github.com/iterative/dvc/pull/4155](https://github.com/iterative/dvc/commit/8dd264c025663a357c7b837c4542a3522d3cc6f1) by efiop
🛠️ [[RFC] tests: make read_text() work with directories (#4155) This PR is just a showcase to see how we like it. I've converted a few
tests to it, but there will be much more to come.

assert (tmp_dir / "file") == "contents"
assert (tmp_dir / "dir").read_text() == {"subdir": {"file": "hello"}}

This approach is nicer because it allows us to use a declaratory
asserts instead of relying on fs files. The problem becomes much more
apparent once you get into remote workspaces, where we trees_equal
doesn't work.](ec88fa8) by efiop
🛠️ dvc: bump to 1.1.6 by efiop
🛠️ gdrive: pop prefix kwarg (#4157) https://travis-ci.com/github/iterative/dvc/jobs/357157580#L8485 by efiop
🛠️ dvc: bump to 1.1.5 by efiop
🛠️ [s3,gs: don't append "/" to prefixed path when listing objects (#4149) * s3,gs: don't append "/" to prefixed path when listing objects

• remote: use optional prefix kwarg for walk_files

• if prefix is True, path will be walked as a prefix

• if prefix is False, path will be treated as a directory (trailing
  slash will be appended for S3-like remotes)

• tests: add unit test for BaseRemoteTree.list_paths](fa16b56) by pmrowla
  🛠️ tests: remotes: use TmpDir-like fixtures (#4140) by efiop
  🛠️ [completion: update dependency (completion: update dependency #4148) * completion: use new simpler shtab API

• completion: use FILE|DIR](5d2ac9f) by casperdcl
  🛠️ azure: use knack directly instead of azure-cli-core (#4147) by efiop
  🛠️ dvc: bump to 1.1.4 by efiop
  🛠️ dvc: bump to 1.1.3 by efiop
  🛠️ [setup: limit flufl version flufl.lock released 4.0 in which there is no finalize method anymore, hence

dvc/lock.py:143:16: E1101: Instance of 'HardlinkLock' has no 'finalize' member (no-member)
```](https://github.com/iterative/dvc/commit/00c135f7d5b55e839354da1e47dc51baa6c64d1a) by [efiop](https://github.com/efiop)
:hammer_and_wrench: [dvc: bump to 1.1.2](https://github.com/iterative/dvc/commit/8cbefaaa88f5a5e3d5e6324600cfd035548242f6) by [efiop](https://github.com/efiop)
:hammer_and_wrench: [azure: use CLI config (#4137)  Fixes #4129](https://github.com/iterative/dvc/commit/0275a0d8d876e230e4b287b52cead53b5b492005) by [efiop](https://github.com/efiop)
:hammer_and_wrench: [Reorder package widgets (#4138)](https://github.com/iterative/dvc/commit/b51ee18fc0df839ad8255d54437c0fed80825dd9) by [dmpetrov](https://github.com/dmpetrov)
:hammer_and_wrench: [RepoTree: preserve relative paths in copytree() (#4133)](https://github.com/iterative/dvc/commit/c6a024f83ba2de2d6852222421f2bcad0618bb06) by [pmrowla](https://github.com/pmrowla)
:hammer_and_wrench: [pylint: enable additional checks (#4122)  Created a separate pylint plugin to disable some checks for tests.](https://github.com/iterative/dvc/commit/1c0f5cdf4ba6188238c60990e96b94402faaaae0) by [skshetry](https://github.com/skshetry)
:hammer_and_wrench: [metrics: skip missing metrics silently (#4130)  Reverts an issue caused by #4005](https://github.com/iterative/dvc/commit/a7e825a0386b6aa08a33eac5262246c86b868588) by [Suor](https://github.com/Suor)
:hammer_and_wrench: [hdfs: implement makedirs separately (#4128)  Clouds that have a non-noop `makedirs` should implement it in their own
method. Hdfs somehow slipped under the radar and had `mkdir` used
explicitly in `copy` and `upload` methods.](https://github.com/iterative/dvc/commit/7b97d7249d7db58de93bb3659c9b599396076ca4) by [efiop](https://github.com/efiop)

- - - 
# CONTRIBUTORS
Last week there were 6 contributors.
:bust_in_silhouette: [efiop](https://github.com/efiop)
:bust_in_silhouette: [pmrowla](https://github.com/pmrowla)
:bust_in_silhouette: [casperdcl](https://github.com/casperdcl)
:bust_in_silhouette: [dmpetrov](https://github.com/dmpetrov)
:bust_in_silhouette: [skshetry](https://github.com/skshetry)
:bust_in_silhouette: [Suor](https://github.com/Suor)

- - - 
# STARGAZERS
Last week there were 45 stagazers.
:star: [LiamLiang](https://github.com/LiamLiang)
:star: [nervegas](https://github.com/nervegas)
:star: [aaravindan101](https://github.com/aaravindan101)
:star: [ffgama](https://github.com/ffgama)
:star: [gurpreet-ai](https://github.com/gurpreet-ai)
:star: [nguyentp](https://github.com/nguyentp)
:star: [mmkassem](https://github.com/mmkassem)
:star: [FaisalSalem](https://github.com/FaisalSalem)
:star: [bitnot](https://github.com/bitnot)
:star: [kemokemo](https://github.com/kemokemo)
:star: [Nokvel](https://github.com/Nokvel)
:star: [0vermind](https://github.com/0vermind)
:star: [maqianlidy](https://github.com/maqianlidy)
:star: [marvinbuss](https://github.com/marvinbuss)
:star: [tsella](https://github.com/tsella)
:star: [stjordanis](https://github.com/stjordanis)
:star: [lordlinus](https://github.com/lordlinus)
:star: [Jinksi](https://github.com/Jinksi)
:star: [fjp](https://github.com/fjp)
:star: [hsinhoyeh](https://github.com/hsinhoyeh)
:star: [zhan-ge](https://github.com/zhan-ge)
:star: [setu4993](https://github.com/setu4993)
:star: [vinogradovkonst](https://github.com/vinogradovkonst)
:star: [joivo](https://github.com/joivo)
:star: [odellus](https://github.com/odellus)
:star: [petoknm](https://github.com/petoknm)
:star: [NaziGanster](https://github.com/NaziGanster)
:star: [PhilipMachineLearning](https://github.com/PhilipMachineLearning)
:star: [rlindskog](https://github.com/rlindskog)
:star: [LumaKernel](https://github.com/LumaKernel)
:star: [Ran-An](https://github.com/Ran-An)
:star: [ojcobb](https://github.com/ojcobb)
:star: [Villux](https://github.com/Villux)
:star: [mysticaltech](https://github.com/mysticaltech)
:star: [dakebl](https://github.com/dakebl)
:star: [faizankshaikh](https://github.com/faizankshaikh)
:star: [justachetan](https://github.com/justachetan)
:star: [wikiped](https://github.com/wikiped)
:star: [shrebox](https://github.com/shrebox)
:star: [narain1](https://github.com/narain1)
:star: [infokiller](https://github.com/infokiller)
:star: [murthy95](https://github.com/murthy95)
:star: [clementabary](https://github.com/clementabary)
:star: [NeerajSarwan](https://github.com/NeerajSarwan)
:star: [kifish](https://github.com/kifish)
You all are the stars! :star2:

- - - 
# RELEASES
Last week there were 6 releases.
:rocket: [1.1.7 1.1.7](https://github.com/iterative/dvc/releases/tag/1.1.7)
:rocket: [1.1.6 1.1.6](https://github.com/iterative/dvc/releases/tag/1.1.6)
:rocket: [1.1.5 1.1.5](https://github.com/iterative/dvc/releases/tag/1.1.5)
:rocket: [1.1.4 1.1.4](https://github.com/iterative/dvc/releases/tag/1.1.4)
:rocket: [1.1.3 1.1.3](https://github.com/iterative/dvc/releases/tag/1.1.3)
:rocket: [1.1.2 1.1.2](https://github.com/iterative/dvc/releases/tag/1.1.2)

- - - 

That's all for last week, please <kbd>:eyes: **Watch**</kbd> and <kbd>:star: **Star**</kbd> the repository [*iterative/dvc*](https://github.com/iterative/dvc) to receive next weekly updates. :smiley:

*You can also [view all Weekly Digests by clicking here](https://github.com/iterative/dvc/issues?q=is:open+is:issue+label:weekly-digest).* 

> Your [**Weekly Digest**](https://github.com/apps/weekly-digest) bot. :calendar:

[efiop]

Whoa, the formatting broke 🙁 And seems like we didn't see this report a week ago. Will consider disabling if it continues this way.

[skshetry]

@efiop, you broke the Weekly Digest with your commit message. 😄