Intermittent errors with CWL Viewer

[kinow]

Description

Last week @mr-c reported CWL Viewer was not working, showing an error message when trying to import a new workflow. We fixed it after cleaning the temporary directory and restarting some services in AWS.

Today I was reviewing the user guide (for common-workflow-language/user_guide#219) and in the chapter 22 there are links to some CWL Viewer workflows. Both were broken.

I opened view.commonwl.org, clicked "Explore" and then clicked on the first workflow I found. This was broken as well.

Expected Behavior

Either a more useful error message (e.g. no disk space, DB instance is busy, etc), or no error.

Actual Behavior

Intermittent errors in the CWL Viewer UI.

Possible Fix

1. Review if we are not doing something inefficient in CWL Viewer regarding DB transactions (I noticed in the logs numerous transactions for each step or input, I think)
2. Check if we can add a finally call that removes any unnecessary files from the temporary directories
3. Add some indicator of what's wrong with the server (something like a Prometheus endpoint?)

Steps to Reproduce

1. Open view.commonwl.org and click on Explore
2. Choose a workflow
3. You may have to wait a while, or import some Workflows… it's not clear what is causing this issue exactly

Context

Possibly related to the PostgreSQL migration? Not sure if it happened before, if we got new users, or some new workflow that triggered it. I'll work with the hypothesis, at first, that it is somehow related to either PostgreSQL or to the dependencies upgraded.

Your Environment

• Version used: 1.4.5
• Environment name and version (e.g. Chrome 39, Java 17): Firefox Ubuntu LTS

[kinow]

• Figure out if we can improve the transactions in the DB, and if that's impacting the performance and intermittent errors
• Fix issues with the tmp dir
  • /tmp fills up unecessarily #200
  • out of disk space due to repeated user requests for the same workflow #279
• Final check to confirm it looks like this issue is fixed

[kinow]

Reading @stain's comment in the linked issue above gave me a clue where to start looking at these temporary files.

Looking at the source code of JGit, the Git instance that we create in multiple parts of the code is a Java Autocloseable.

Which means that we can call .close() on it, or even better, with a try-with-resources that auto-closes it.

That tells JGit to free up the resources of that repository. So eventually the JVM must garbage-collect the unused objects/closed objects, which should result in file descriptors being freed too, and files deleted from the file system.

[kinow]

Created a draft pull request to start adding the fixes for this issue. Not sure if I will disable the cron job that clears directories. To be honest, I might just update the pull request #344 since the current job is cheap and could work as a fallback option (i.e. doesn't hurt to have it running, IMO).

[kinow]

Note too, that closing a Git repository should free up the resources in the JVM. But as with anything related to garbage collection in Java, there are no 100% promises that the objects will be freed, nor that the files will be deleted, nor that it will work with every JVM implementation. So, again, doesn't hurt to have a cron job deleting older files, that may not have been deleted for some reason 🤷‍♂️

[swzCuroverse]

@kinow you should talk to @cure as well since I am pretty sure he knows about these issues and has been periodically triaging them.

[kinow]

Already did so a couple days ago, but will mention again in today's meeting. Not sure if I will have a working branch this week, so the new devops will probably have to deploy this new version as one of their first tasks I think. Thanks @swzCuroverse !

[kinow]

Note to self: a good way to test the temporary files issues is to start Jena + PostgreSQL, and then start the Java application using a different temporary directory.

mkdir -pv /tmp/cwlviewer/
java -Djava.io.tmpdir=/tmp/cwlviewer/ ....

The tomcat folders are used while the Spring application is running (at least one is left behind, but empty). Then import a workflow, and observe the temporary files and folders created. Stop the server, start again, and keep analyzing the behavior of the system and comparing with the source code…

[kinow]

And using the technique above I can confirm the pull request linked, even though it is closing the JGit repository, is still leaving the git repository behind 🤔

[kinow]

For the compile.cwl workflow linked in the main page, here's what is created in the temporary directory (ignore the two tomcat folders).

2 folders, 4 files 🤓

[kinow]

Huh, interesting. I modified the CWLToolRunner, to delete the Git repository directory once it has finished running (success or not).

Then I did a test where I imported compile.cwl. Success. Then I related the RO bundle, image files, and refreshed the page. The Git repository directory was there again now 😥

[kinow]

The ROBundleService cloned the repository again if the RO Bundle file had been deleted (e.g. cron job, manually), but also didn't delete the repository once it was done.

It looks like the RO bundle ZIP file is streamed to the user in a download, but then left behind in the server. I will take a look if there is a way to delete it (.deleteOnExit?). But there is also a RO Bundle empty directory left behind that can be deleted (just need to find out where it's created). Getting there…

[kinow]

What's left behind now in the temporary directory: