Upgrade DVC on Ubuntu 18.04 Deb Repository Install #4193

melkiga · 2020-07-11T20:11:21Z

Bug Report (Short Story)

I've installed DVC on my work server using the deb package from here, but want to be able to upgrade when a new version is released. I followed the instructions here in the repo section, and when I run sudo apt update I receive the following.

W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Sat, 11 Jul 2020 19:55:43 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive     [IP: 10.224.0.67 8000]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Behavior

When I run sudo apt-get install --only-upgrade dvc, I receive a response that I already have the latest version of DVC, when in fact a new version has been released (1.1.8):

$ sudo apt-get install --only-upgrade dvc
Reading package lists... Done
Building dependency tree       
Reading state information... Done
dvc is already the newest version (1.1.7).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I then tried following your instructions here for how to install from repo, but when I run sudo apt update I receive the following error:

$ sudo apt update
.....
Reading package lists... Done                     
W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Sat, 11 Jul 2020 19:55:43 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  ...
W: Some index files failed to download. They have been ignored, or old ones used instead.

Please provide information about your setup

Output of dvc version:

$ dvc version
1.1.7

OS cat /etc/os-release

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

The text was updated successfully, but these errors were encountered:

efiop · 2020-07-11T20:59:25Z

Hi @melkiga !

Thank you so much for the feedback! Looks like we had some issues with our repo permissions. We've adjusted that and it should be working fine now. Could you please give it a try and let us know how it goes?

melkiga · 2020-07-11T21:01:29Z

Thank you for the quick response!
Unfortunately I still get the following:

W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Sat, 11 Jul 2020 21:00:35 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive

efiop · 2020-07-11T21:02:04Z

Oops, sorry, still not quite resolved. Looking into it...

melkiga · 2020-07-11T21:04:12Z

Not sure if this helps:
If I open this link https://dvc.org/deb/dists/stable/InRelease in my browser, I get redirected to this https://s3-us-east-2.amazonaws.com/dvc-s3-repo/deb/dists/stable/InRelease with this as the page contents:

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>deb/dists/stable/InRelease</Key>
<RequestId>BAFC7EB1008F87B5</RequestId>
<HostId>rHkvp+fPWpByruL4IQHXdrROVDcAPT1tn2PpIlRTagOyjkXbk7Q248WE2ekcik+FCmMcm8zHlMY=</HostId>
</Error>

efiop · 2020-07-11T21:07:20Z

@melkiga Yeah, InRelease file doesn't exist (and it never did, not sure why apt is trying to access it at all), investigating...

efiop · 2020-07-11T21:16:34Z

Ok, looks like this might be the cause krobertson/deb-s3#137 . Investigating why it is not generated already...

efiop · 2020-07-12T00:33:54Z

@melkiga Sorry for the delay. I've added a dummy signature to make it comply for now. Could you please give it another shot? It could complain about it not being able to verify gpt signature, but it should work fine for now.

melkiga · 2020-07-12T00:37:07Z

No issues about the delay, this isn't a blocker for me :-)
OK so I ran the following:

$ sudo wget \
        https://dvc.org/deb/dvc.list \
        -O /etc/apt/sources.list.d/dvc.list
$ sudo apt update

and got the same result:

W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Sun, 12 Jul 2020 00:35:36 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive    
W: Some index files failed to download. They have been ignored, or old ones used instead.

Could this be an issue from my end?

efiop · 2020-07-12T00:42:30Z

@melkiga Hm, that's strange. That file is available right now. Maybe it didn't yet update in your region, let's wait a few minutes and then try once more. Thank you for the feedback! 🙏

melkiga · 2020-07-12T00:45:59Z

No issues! Thanks for the quick response. I'll wait about 20 more minutes and try again. Worst case we can leave it till tomorrow and I can try then and report back.

efiop · 2020-07-14T09:07:47Z

@melkiga Any luck? 🙂

melkiga · 2020-07-17T18:56:55Z

Giving it a try now. Sorry for the late reply!

melkiga · 2020-07-17T19:01:27Z

Still no luck, here's what I've done:

Ran the following

$ sudo wget \
       https://dvc.org/deb/dvc.list \
       -O /etc/apt/sources.list.d/dvc.list
$ sudo apt update

Here is the response of sudo apt update:

$ sudo apt update
...
Err:3 https://dvc.org/deb stable InRelease                                                                                                                                                                    
  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Fri, 17 Jul 2020 18:58:35 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive     [IP: 10.224.0.67 8000]
......
W: Failed to fetch https://dvc.org/deb/dists/stable/InRelease  Invalid response from proxy: HTTP/1.1 403 Forbidden  Server: squid/3.5.27  Mime-Version: 1.0  Date: Fri, 17 Jul 2020 18:58:35 GMT  Content-Type: text/html;charset=utf-8  Content-Length: 3506  X-Squid-Error: ERR_ACCESS_DENIED 0  Vary: Accept-Language  Content-Language: en  X-Cache: MISS from squid-deb-proxy  X-Cache-Lookup: NONE from squid-deb-proxy:8000  Via: 1.1 squid-deb-proxy (squid/3.5.27)  Connection: keep-alive     [IP: 10.224.0.67 8000]
W: Some index files failed to download. They have been ignored, or old ones used instead.

efiop · 2020-07-18T04:25:23Z

@melkiga Interesting 🙁 Are you on company's vpn/proxy or something?

melkiga · 2020-07-18T12:05:04Z

Yes, I'm on a company VPN. Do you think that's the cause? Is anyone else able to update replicate the issue?

🧐

efiop · 2020-07-18T20:16:41Z

@melkiga Might be, actually. Could you try this on your personal machine? I'm not able to reproduce myself nor on our test machines :(

melkiga · 2020-07-19T00:50:44Z

Going to try that now :-) I'll report back in a few.

melkiga · 2020-07-19T01:28:14Z

Here's what I got on my personal machine:

Reading state information... Done
All packages are up to date.
W: GPG error: https://s3-us-east-2.amazonaws.com/dvc-s3-repo/deb stable InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8F02DCF1DB7C115F

Machine Info:

~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

efiop · 2020-07-19T10:19:23Z

@melkiga GPG warning is expected, we are temporarily using one-off keys to sign InRelease iterative/dvc-s3-repo#3 . So that warning could be ignored for now. I suppose other than that warning, the command worked fine, right?

I'm no longer seeing Forbidden errors in your log, so your company's VPN is indeed blocking access to our repo 🙁

melkiga · 2020-07-19T15:03:03Z

Whoops, I mistook the warning for an error. I just completed my install so yes, it seems that the company VPN is the culprit.
Thanks again for helping out with this!

efiop · 2020-07-19T20:12:51Z

@melkiga Thank you so much for the feedback! So looks like you'll need to ping your admins to whitelist that url or, you could consider installing using a different package. E.g. snap or pip. Would the alternative packages work for you?

melkiga · 2020-07-21T19:40:54Z

I'm currently just doing deb installs manually. I'll ping our admin to whitelist the url, just because of the way our pip/python stuff is set up on our servers.

Thanks a ton for the help!

efiop · 2021-03-03T00:18:33Z

Hi @melkiga ! Just a heads up that proper signatures are now available iterative/dvc.org#2252 . Thanks again for the feedback! 🙏

ghost added the triage Needs to be triaged label Jul 11, 2020

efiop added bug Did we break something? p0-critical labels Jul 11, 2020

ghost removed the triage Needs to be triaged label Jul 11, 2020

efiop self-assigned this Jul 11, 2020

efiop added the awaiting response we are waiting for your reply, please respond! :) label Jul 11, 2020

weekly-digest bot mentioned this issue Jul 12, 2020

Weekly Digest (5 July, 2020 - 12 July, 2020) #4195

Closed

efiop mentioned this issue Jul 15, 2020

properly sign our packages with our own gpg signature iterative/dvc-s3-repo#3

Closed

melkiga closed this as completed Jul 21, 2020

Upgrade DVC on Ubuntu 18.04 Deb Repository Install #4193

Upgrade DVC on Ubuntu 18.04 Deb Repository Install #4193

Comments

melkiga commented Jul 11, 2020

Bug Report (Short Story)

Behavior

Please provide information about your setup

efiop commented Jul 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

melkiga commented Jul 11, 2020

Uh oh!

efiop commented Jul 11, 2020

Uh oh!

melkiga commented Jul 11, 2020

Uh oh!

efiop commented Jul 11, 2020

Uh oh!

efiop commented Jul 11, 2020

Uh oh!

efiop commented Jul 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

melkiga commented Jul 12, 2020

Uh oh!

efiop commented Jul 12, 2020

Uh oh!

melkiga commented Jul 12, 2020

Uh oh!

efiop commented Jul 14, 2020

Uh oh!

melkiga commented Jul 17, 2020

Uh oh!

melkiga commented Jul 17, 2020

Uh oh!

efiop commented Jul 18, 2020

Uh oh!

melkiga commented Jul 18, 2020

Uh oh!

efiop commented Jul 18, 2020

Uh oh!

melkiga commented Jul 19, 2020

Uh oh!

melkiga commented Jul 19, 2020

Uh oh!

efiop commented Jul 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

melkiga commented Jul 19, 2020

Uh oh!

efiop commented Jul 19, 2020

Uh oh!

melkiga commented Jul 21, 2020

Uh oh!

efiop commented Mar 3, 2021

Uh oh!

efiop commented Jul 11, 2020 •

edited

Loading

efiop commented Jul 12, 2020 •

edited

Loading

efiop commented Jul 19, 2020 •

edited

Loading