Skip to content

Support uploading over http(s)://  #7

New issue
New issue
Not planned
Not planned
Support uploading over http(s):// #7
@kousu

Description

@kousu
kousu
opened on May 8, 2022 · edited by kousu

This is a follow up to #3: filling in the blanks from https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/, I thought that only git annex get worked with http remotes. But now I think that was premature.

Investigate and make git-annex uploading work over http remotes.

Specifically this means these commands should work:

  • git annex sync --content
  • git annex copy --to origin

Activity

kousu

kousu commented on May 8, 2022

@kousu
kousu
on May 8, 2022 · edited by kousu
Author

Using a test repo at https://data.praxisinstitute.org.dev.neuropoly.org/uofc/CANDICE-fMRI-, which is currently running the patch for #3, download anonymously:

git clone ... && git annex get 
[kousu@nigiri down]$ git clone https://data.praxisinstitute.org.dev.neuropoly.org/UofC/CANDICE-fMRI-.git
Cloning into 'CANDICE-fMRI-'...
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 71 (delta 21), reused 0 (delta 0)
Receiving objects: 100% (71/71), 9.72 KiB | 4.86 MiB/s, done.
Resolving deltas: 100% (21/21), done.
[kousu@nigiri down]$ cd CANDICE-fMRI-/
[kousu@nigiri CANDICE-fMRI-]$ git annex get
get sub-BAN01/anat/sub-BAN01_T1w.nii.gz (from origin...) 
47%   2.87 MiB        895 KiB/s 3s    
ok                                    
get sub-BAN02/anat/sub-BAN02_T1w.nii.gz (from origin...) 
52%   3.31 MiB          2 MiB/s 1s
ok                                
get sub-BAN03/anat/sub-BAN03_T1w.nii.gz (from origin...) 
ok                                
(recording state in git...)

then construct an new annexed file:

cp -r sub-BAN02/ sub-BAN04
dd if=/dev/urandom of=sub-BAN04/anat/sub-BAN02_T1w.nii.gz count=512
git add sub-BAN04/
git commit -m "Subject 04"

prove it's distinct (it's the last one, the one with 1 copy):

git annex whereis 
[kousu@nigiri CANDICE-fMRI-]$ git annex whereis
whereis sub-BAN01/anat/sub-BAN01_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN02/anat/sub-BAN02_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN03/anat/sub-BAN03_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN04/anat/sub-BAN02_T1w.nii.gz (1 copy) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
ok

and now see what happens if we try to upload it; with debugging turned on:

git annex copy --to origin 
[kousu@nigiri CANDICE-fMRI-]$ git config annex.debug true
[kousu@nigiri CANDICE-fMRI-]$ git annex copy --to origin
[2022-05-08 02:52:23.217458919] (Utility.Process) process [28036] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2022-05-08 02:52:23.219000083] (Utility.Process) process [28036] done ExitSuccess
[2022-05-08 02:52:23.219516464] (Utility.Process) process [28037] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2022-05-08 02:52:23.221135085] (Utility.Process) process [28037] done ExitSuccess
[2022-05-08 02:52:23.22179462] (Utility.Process) process [28038] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..6c3b23ed5b89e73fdb170e4dfb3dc4f7324acd87","--pretty=%H","-n1"]
[2022-05-08 02:52:23.224259745] (Utility.Process) process [28038] done ExitSuccess
[2022-05-08 02:52:23.225976282] (Utility.Process) process [28039] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2022-05-08 02:52:23.228332152] (Utility.Process) process [28040] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--stage","-z","--error-unmatch","--"]
[2022-05-08 02:52:23.229159541] (Utility.Process) process [28041] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2022-05-08 02:52:23.22958377] (Utility.Process) process [28042] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2022-05-08 02:52:23.230213916] (Utility.Process) process [28039] done ExitSuccess
[2022-05-08 02:52:23.231313697] (Utility.Process) process [28043] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
copy sub-BAN01/anat/sub-BAN01_T1w.nii.gz [2022-05-08 02:52:23.269482287] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/c34/98e/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:23.715269089] (Utility.Process) process [28045] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:33.439560188] (Utility.Process) process [28045] done ExitSuccess
[2022-05-08 02:52:33.439918575] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/c34/98e/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:33.531585766] (Utility.Process) process [28081] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:33.533291906] (Utility.Process) process [28081] done ExitSuccess
[2022-05-08 02:52:33.53378081] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/73/34/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:33.622360074] (Utility.Process) process [28082] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:38.54396353] (Utility.Process) process [28082] done ExitSuccess
[2022-05-08 02:52:38.544127686] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/73/34/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:38.629070613] (Utility.Process) process [28086] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:38.630412108] (Utility.Process) process [28086] done ExitSuccess
(not found) failed
copy sub-BAN02/anat/sub-BAN02_T1w.nii.gz [2022-05-08 02:52:38.63124213] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:38.717445234] (Utility.Process) process [28087] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:43.084445916] (Utility.Process) process [28087] done ExitSuccess
[2022-05-08 02:52:43.084688615] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:43.170468809] (Utility.Process) process [28088] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:43.172434906] (Utility.Process) process [28088] done ExitSuccess
[2022-05-08 02:52:43.172943361] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:43.258412042] (Utility.Process) process [28089] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:46.248660233] (Utility.Process) process [28089] done ExitSuccess
[2022-05-08 02:52:46.249043634] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:46.335692255] (Utility.Process) process [28090] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:46.337206776] (Utility.Process) process [28090] done ExitSuccess
(not found) failed
copy sub-BAN03/anat/sub-BAN03_T1w.nii.gz [2022-05-08 02:52:46.337740833] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/58d/8ea/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:46.423388602] (Utility.Process) process [28091] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:49.262936219] (Utility.Process) process [28091] done ExitSuccess
[2022-05-08 02:52:49.263330345] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/58d/8ea/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:49.349186911] (Utility.Process) process [28092] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:49.350967438] (Utility.Process) process [28092] done ExitSuccess
[2022-05-08 02:52:49.35138871] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/1X/Mx/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:49.435139518] (Utility.Process) process [28093] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:52.986881594] (Utility.Process) process [28093] done ExitSuccess
[2022-05-08 02:52:52.987023514] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/1X/Mx/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:53.076025925] (Utility.Process) process [28096] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:53.077770146] (Utility.Process) process [28096] done ExitSuccess
(not found) failed
copy sub-BAN04/anat/sub-BAN02_T1w.nii.gz [2022-05-08 02:52:53.078448893] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:53.16577961] (Utility.Process) process [28097] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:56.868778349] (Utility.Process) process [28097] done ExitSuccess
[2022-05-08 02:52:56.869119387] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:56.954725712] (Utility.Process) process [28098] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:56.956656836] (Utility.Process) process [28098] done ExitSuccess
[2022-05-08 02:52:56.957140664] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:57.041148655] (Utility.Process) process [28099] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:53:00.095495453] (Utility.Process) process [28099] done ExitSuccess
[2022-05-08 02:53:00.095774849] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:53:00.183376947] (Utility.Process) process [28100] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:53:00.18501078] (Utility.Process) process [28100] done ExitSuccess
(not found) failed
[2022-05-08 02:53:00.185253039] (Utility.Process) process [28043] done ExitSuccess
[2022-05-08 02:53:00.185356286] (Utility.Process) process [28042] done ExitSuccess
[2022-05-08 02:53:00.185460462] (Utility.Process) process [28041] done ExitSuccess
[2022-05-08 02:53:00.185552806] (Utility.Process) process [28040] done ExitSuccess
copy: 4 failed

Interestingly, git-annex is sending an Authorization header, and it looks like it's leaning on git credential so so it's git doing the prompting, which is nice and elegant. So maybe uploading over HTTP does work afterall.

But this part of the protocol isn't documented on the wiki, or at least I don't want to waste my time trying to find my way around that place. But I bet we have enough here to reverse engineer it; it looks like the first thing it needs is a response to a HEAD request, presumably checking if the file already exists or not; so we'll have to add another route for that, and once we do that maybe it'll send a POST or a PUT request to actually upload the file, and we'll have to add a route for that too. And then it should be done 🤞

kousu
mentioned this on May 8, 2022
kousu

kousu commented on Sep 20, 2022

@kousu
kousu
on Sep 20, 2022 · edited by kousu
Author

I tried again to get this working for a few hours but no matter what I tried I couldn't convince git-annex to do it.

I learnt it issues a HEAD request for each file. If that returns 200, and the Content-Length matches the expected size, it says "ok". Otherwise, it says "not found". It never tries to issue a PUT or a POST. So I doubt this is supported.

I suspect it's a result of there being no server in the world that it could possibly be tested against. GitHub doesn't support it, Apache doesn't support PUTs, GitLab probably doesn't either. If we added annex HTTP uploading support we'd be the first, probably.

I'll put in a feature request and see what joeyh says.

kousu
changed the title [-]Support `git annex sync --content`/`git annex copy --to` over http(s):// [/-] [+]Support uploading over http(s):// [/+] on Sep 20, 2022
kousu

kousu commented on Sep 20, 2022

@kousu
kousu
on Sep 20, 2022
Author

Feature request/request for clarification: https://git-annex.branchable.com/forum/HTTP_uploads/?updated

kousu

kousu commented on Sep 23, 2022

@kousu
kousu
on Sep 23, 2022
Author

This is a wontfix:

There is no "one true way" to upload a file to a HTTP server.

I don't want to add some additional git-annex specific HTTP-based protocol.

(I lightly disagree: the One True Way is PUT + an Authorization: header, which is just what S3 uses, and what Gitea could easily support. But if the code doesn't exist it doesn't exist.)

kousu
closed this as not plannedon Sep 23, 2022

179 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @kousu

        Issue actions
          Support uploading over http(s):// · Issue #7 · neuropoly/gitea