Closed
Description
Is your feature request related to a problem? Please describe
Once disk usage in /workspace
approaches or hits a limit (30G), our ability to restore workspaces becomes unstable and files can no longer be added to /workspace
.
Describe the behaviour you'd like
- How reliably do we restore
/workspace
once "almost full", say 29.5GB? With one large file? With many small files? How long do the backup and restore take? - Run a test, with a
/workspace
that has its limit set to 50GB in a XL cluster. With one large file? With many small files? How long do the backup and restore take?
Additional context
Given the below test, we were able to backup the workspace, but not restore it.
gitpod /workspace/template-python-flask $ dd if=/dev/zero of=upload_test bs=1M count=32000
dd: error writing 'upload_test': No space left on device
30535+0 records in
30534+0 records out
32017219584 bytes (32 GB, 30 GiB) copied, 19.5889 s, 1.6 GB/s
gitpod /workspace/template-python-flask $ touch foo
touch: cannot touch 'foo': No space left on device
gitpod /workspace/template-python-flask $ du -hs
30G
gitpod /workspace/template-python-flask $ ls -ltrh
total 30G
-rw-r--r-- 1 gitpod gitpod 950 Mar 2 14:27 README.md
-rw-r--r-- 1 gitpod gitpod 122 Mar 2 14:27 MANIFEST.in
-rw-r--r-- 1 gitpod gitpod 1.5K Mar 2 14:27 LICENSE.rst
drwxr-xr-x 2 gitpod gitpod 122 Mar 2 14:27 tests
-rw-r--r-- 1 gitpod gitpod 38 Mar 2 14:27 setup.py
-rw-r--r-- 1 gitpod gitpod 535 Mar 2 14:27 setup.cfg
drwxr-xr-x 2 gitpod gitpod 110 Mar 2 14:27 flaskr.egg-info
drwxr-xr-x 5 gitpod gitpod 136 Mar 2 14:27 flaskr
drwxr-xr-x 2 gitpod gitpod 27 Mar 2 14:27 instance
-rw-r--r-- 1 gitpod gitpod 30G Mar 2 14:28 upload_test
On restore, the error we received was cannot initialize workspace: cannot write workspace ready file content: open /dst/.gitpod/ready.tmp: no space left on device
.