Problem accessing cache.lock due to permission error

[weitjong]

In our use case we install/activate the EMSDK with root user but we use the EMSDK to build app with normal non-root user. This worked before as the .emscripten_cache and .emscripten_cache.lock were created in the home-dir of the normal user. The recent EMSDK has broken this use case because it expects the normal user to have write permission to the EMSDK or at least to the parent dir where the cache.lock resides.

As I understood from WebAssembly/waterfall#644, the "libs" are now pre-built by upstream and as such it is not so much a build-on-demand-and-cache-for-future-use anymore. Do you really still need the "locking" mechanism to prevent concurrent on-demand build? It seems we can bypass this mechanism and allow the non-root user to read from the cache dir directly.

[trzecieu]

Hi @weitjong, sorry for the inconvenience. We're aware about this problem and it will be fixed as a part of #533

[sbc100]

@trzecieu I'm not sure this issue relates to the docker image does it?

@weitjong I'm interested to learn more about your use case. We currently --no-embedded to revert to the previous behaviour or using the user's home directory. However I'm hoping to remove that option ASAP (#510).

Can you explain why you include/activate as the root user? Is there some reason why you can't install/activate as the specific user? What OS are you using? Do you actually have more than one user what needs to share a single EMSDK install?

I will look into ensuring that the cache is never actually written to in the emsdk case.. you are correct that it should be fully populated already at install time.

[weitjong]

Our use case is to prepare dockerized build environment for building our cross-platform project. We have docker image for each target platform that we support and the Web platform with EMSDK as compiler toolchain is just one of them. Basically the docker images are prepared by installing the compiler toolchain as root during docker image building time, however, during runtime we just use a non-root user (or in rootless mode when podman is being used) to invoke the compiler toolchain to build the app. Back to EMSDK, it means we install and activate the SDK as root like all other platforms, nothing special here. The activated SDK is then persisted in our docker image ready to be run by our CI or devs. The idea is, we get a reproducible "dockerized" build environment each time we build our project. We have even managed to use docker volume to persist the content of the home directory between runs, meaning all the good bits in the caches created by EMSDK, or ccache, or gradle, etc, are reused when the docker container is spawned with the volume reattached back.

[sbc100]

Ah, OK, I didn't realize you were talking about a docker image.

Do you know we have semi-offiical docker images that we publish at: https://hub.docker.com/r/emscripten/emsdk

You can see how we solve the permissions issue here:

emsdk/docker/Dockerfile

Line 37 in 7f77b61

&& chmod -R 777 ${EMSDK}/upstream/emscripten/cache \

As @trzecieu points out it looks like we need to change the permissions of the upstream/emscripten directory too (not recursively, just the directory). See #533. We will try fix this in the future so its is no longer necessary.

Do you have your own cusom Dockerfile or are you shared/inheritiing from our one?

[weitjong]

Actually after reporting the issue here, I have also quickly worked around the permission issue so it does not block me further. We have now this extra line in our "entrypoint" shell script.

# Temporary workaround to let 'urho3d' user to lock the cache, see https://github.com/emscripten-core/emsdk/issues/535
sudo chmod o+w $(dirname $EM_CACHE) && sudo rm -f $EM_CACHE.lock

It does not need to be recursive. Still, to me this is just a temporary workaround. No, we are not inheriting from Emscripten docker image. All our docker images are inherited from our own common base image.

[sbc100]

Rather than a custom entry point, could you not just run chmod o+w $(dirname $EM_CACHE) && rm -f $EM_CACHE.lock as the last step in image creation? (no sudo needed here).

[weitjong]

Yes, it could be and it should work the same. But since I see it as a temporary workaround (and that we have an entrypoint shell script anyway in our Dockerfile), I opted the former so that I can preserve the large part of our docker layer cache. That is, I still hope for a more permanent solution upstream later and I can quickly remove the extra chmod line without invalidating the docker layer cache again. BTW, my original question about the need for locking has not been answered yet. With your changes to have all the libs prebuilt upstream, do you really still need to consider the “cache” dir as cache per se. if it contains all the libs already prebuilt then why not just call it as “libs” and allows the libs dir to be read without any locking mechanism. I believe the existing permission (755) on the directory is actually fine, if we don’t have this locking mechanism.

…

Sent from my iPad

On 30 Jun 2020, at 11:44 PM, Sam Clegg ***@***.***> wrote:  Rather than a custom entry point, could you not just run chmod o+w $(dirname $EM_CACHE) && rm -f $EM_CACHE.lock as the last step in image creation? (no sudo needed here). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[sbc100]

You are correct that ideally the cache when completely populated would not need write access and could be seen a read-only resource. We are not quite there yet though. For now the cache and lock still need write access.

In the future I do hope to rename it to something like sysroot. In this case it is still useful if the user/developer can write to this location though. For example if as part of the build they want to install sysroot/local/lib/libfoo.a or something like that. So I think it fine if the user always has write access to this directory, even in the docker case. Do you have some reason not to give users write access? With docker, any changes will not persist across runs anyway right?

[weitjong]

You are correct that ideally the cache when completely populated would not need write access and could be seen a read-only resource. We are not quite there yet though. For now the cache and lock still need write access.

I see. But if that is the case then we need more "chmod" to workaround the non-root user to write into the current cache dir. I just tried to issue the command emcc --clear-cache as non-root user to emulate writing into the cache dir. It failed because:

1. The non-root user does not have write permission to create this file $EMSDK/.emscripten_sanity.
2. After fixing the above by chmod o+w $EMSDK, the non-root user does not have write permission on $EM_CACHE dir itself and also on all the files and subdirs in it.

Setting world writable files is not a good practice. So, I don't think you should do that in the EMSDK and may be also not in the docker image as permanent solution.

Do you have some reason not to give users write access? With docker, any changes will not persist across runs anyway right?

Our docker images are built using Ubuntu as base OS. As you know in Linux only root is allowed to install and setup things. Non-root user only allowed to read those things that have been installed and to write into their own home-dir or other dirs that the non-root user actually owns. Even when inside a container, it is easier to reason about when we know the non-root user does not have free reign on the filesystem.

Also I have mentioned before that previously we have managed to persist our non-root user home-dir between docker runs using docker volume. So, in the worse case scenario where we have to build the lib from scratch, we only build it once and cache it in the ~/.emscripten_cache. With the EM_CACHE now points to somewhere else, even with the write permission issue resolved, we still lose in performance because the lib will be rebuilt again in the next run, as you pointed out.

[sbc100]

As a short term solution you can give all users write access to that cache directory and the containing emsdk directory:

chmod go+w -R $EMSDK/cache && chmod go+w $EMSDK

(note the first command is recursive and the second is not).

In theory the cache should not be written too since it comes repopulated and should be shareable by all users. I thought the problem was solved by pre-generating the sanity file here:

emsdk/docker/Dockerfile

Line 58 in 7f77b61

RUN echo "## Generate sanity" \

But it seems we need to better solution we we will continue to work on this.

Are you OK with the short tem fix of giving more permissions to the top level directory + the cache?

[trzecieu]

@sbc100 I've seen that there was a Pull Request addressing that issue emscripten-core/emscripten#11301
So the problem as you've pointed could be to make emscripten root writeable.

A workaround could be tackled from a different angle tho:

As Emscripten has a predefined cache folder os.path.expanduser('~/.emscripten_cache') used if the cache folder is not writeable (don't confuse with EM_CACHE).
And docker container has been called with user that's UID:GID is not reflected in docker image, so that $HOME is empty, which leads to os.path.expanduser('~/.emscripten_cache') gets resolved to /.emscripten_cache and /.emscripten_cache.lock and eventually an error because of lack permission of modifying root / folder.

By simple adding -e HOME=/tmp should also solve the problem that you're facing, drawback is that cache will not be shared for consequential builds, but that could be fixed by mounting cache folder.

In the meantime we'll try to fix that in emcc / emsdk

[weitjong]

But it seems we need to better solution we we will continue to work on this.

Are you OK with the short tem fix of giving more permissions to the top level directory + the cache?

@sbc100 and @trzecieu Thanks for your time looking at this issue.

For now we have identified a few possible short term workaround, each with its own drawback:

1. give write-access to parent dir of $EM_CACHE (so non-root can create $EM_CACHE.lock). Drawback: it only allows non-root to access the prebuilt libraries but not creating new one.
2. to solve the first drawback, after 1. above then continue to give write-access recursively to $EM_CACHE dir too, possibly also need to give write-access to EMSDK (so non-root can create $EMSDK/.emscripten_sanity). Drawback: prebuilt libraries with world-writable permission, and newly built libraries (if any) are not persistent (in docker container context).
3. instead of having world-writable permission, don't do 1 & 2 above but simply change the ownership of the $EMSDK recursively to this non-root user on the fly in the entrypoint shell script. Drawback: newly built libraries (if any) are not persistent still. I actually haven't really tested 3 yet but it should work as it is almost like it was the non-root who had installed the EMSDK in the first place.
4. revert to previous behavior by having $EM_CACHE set to ~/.emscripten_cache in the entrypoint shell script. Any library built will be cached in between runs. Except, now the drawback is, we could not take advantage of upstream prebuilt libraries. So it also means slow initial run when the initial cache is still empty.

After considering all these are just workaround and that assuming a better and more permanent solution will be available later, I decided to still stick to our initial workaround (1). We can live with its drawback at the moment.

As for the permanent solution, have you guys considered the possibility of having two different locations for libraries? One global write-protected (owned by root) containing all the prebuilt libs. One local owned by each non-root user on home-dir containing newly built libraries that not yet available on global. No locking mechanism required on global.

[kleisauke]

PR #578 should resolve this by giving write-access to the $EMSDK/upstream/emscripten directory (this is described as point 1 within the above post).

[weitjong]

Just for the completeness sake, we have adapted our workaround in our own Docker image to let the "other" user to lock the file only, just in order to allow this user to read from the cache dirs (since we cannot request to get the lock mechanism removed).

# Temporary workaround to let 'urho3d' user to lock the cache, see https://github.com/emscripten-core/emsdk/issues/535
sudo find $EM_CACHE -type f -name \*.lock -exec sudo chmod o+w {} \;

[sbc100]

Sadly that is not enough. Users will need full write access to the whole cache in case they want to build a library that doesn't exist yet. They even might want to rebuild the cache using emcc --clear-cache for some reason.

For many users this won't be a problem, but (in the general case) emscripten doesn't' currently support read-only caches because it expects to be able to write files there.