Skip to content

Workspace cannot be restarted when /worksapce is full. #8671

Closed
@kylos101

Description

@kylos101

Is your feature request related to a problem? Please describe

Once disk usage in /workspace approaches or hits a limit (30G), our ability to restore workspaces becomes unstable and files can no longer be added to /workspace.

Describe the behaviour you'd like

  1. How reliably do we restore /workspace once "almost full", say 29.5GB? With one large file? With many small files? How long do the backup and restore take?
  2. Run a test, with a /workspace that has its limit set to 50GB in a XL cluster. With one large file? With many small files? How long do the backup and restore take?

Additional context

Given the below test, we were able to backup the workspace, but not restore it.

gitpod /workspace/template-python-flask $ dd if=/dev/zero of=upload_test bs=1M count=32000
dd: error writing 'upload_test': No space left on device
30535+0 records in
30534+0 records out
32017219584 bytes (32 GB, 30 GiB) copied, 19.5889 s, 1.6 GB/s

gitpod /workspace/template-python-flask $ touch foo
touch: cannot touch 'foo': No space left on device

gitpod /workspace/template-python-flask $ du -hs
30G  

gitpod /workspace/template-python-flask $ ls -ltrh
total 30G
-rw-r--r-- 1 gitpod gitpod  950 Mar  2 14:27 README.md
-rw-r--r-- 1 gitpod gitpod  122 Mar  2 14:27 MANIFEST.in
-rw-r--r-- 1 gitpod gitpod 1.5K Mar  2 14:27 LICENSE.rst
drwxr-xr-x 2 gitpod gitpod  122 Mar  2 14:27 tests
-rw-r--r-- 1 gitpod gitpod   38 Mar  2 14:27 setup.py
-rw-r--r-- 1 gitpod gitpod  535 Mar  2 14:27 setup.cfg
drwxr-xr-x 2 gitpod gitpod  110 Mar  2 14:27 flaskr.egg-info
drwxr-xr-x 5 gitpod gitpod  136 Mar  2 14:27 flaskr
drwxr-xr-x 2 gitpod gitpod   27 Mar  2 14:27 instance
-rw-r--r-- 1 gitpod gitpod  30G Mar  2 14:28 upload_test

On restore, the error we received was cannot initialize workspace: cannot write workspace ready file content: open /dst/.gitpod/ready.tmp: no space left on device.

Metadata

Metadata

Assignees

No one assigned

    Labels

    team: workspaceIssue belongs to the Workspace team

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions