Skip to content

Parallelise our CI/CD jobs #250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
CMCDragonkai opened this issue Oct 19, 2021 · 6 comments · Fixed by #292
Closed
3 tasks done

Parallelise our CI/CD jobs #250

CMCDragonkai opened this issue Oct 19, 2021 · 6 comments · Fixed by #292
Labels
procedure Action that must be executed

Comments

@CMCDragonkai
Copy link
Member

CMCDragonkai commented Oct 19, 2021

Details from https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/184#note_668294739

Using an ever longer delay is not a good way of doing these tests, they don't scale. Ideally we fix this up before going into prod.

I also noticed our tests taking longer now. Up to 30 minutes.

To speed up our testing, we can parallelise our testing. Now our jest tests are naturally parallelised by checking the number of cores the host environment has. But the CI/CD runners all have 1 CPU core:

If you’re using the GitLab.com 32 autoscaling shared runners, I believe the runners have 1CPU, 4GB RAM, and 22Gb of storage (16Gb available).

There should be a way for us to configure with more cores or other wise increase the number of parallel jobs by splitting the test job, which would end up using multiple runners (somewhat inefficient) due to duplication of resources, and then adding our own custom runner with more cores.

A quick way to do this would to be create separate CI/CD jobs for each domain run. So npm test tests/X and npm test tests/Y might allow us to take advantage of gitlab CI/CD's splitting of each job in parallel. Only issue is the sharing of the Nix environment.

Tasks

  1. - Investigate dynamic parent child pipelines
  2. - Test splitting can be automatic by splitting a pipeline for each subdirectory under tests/*
  3. - For all the immediate files in tests/* they can run under a single test pipeline
@CMCDragonkai CMCDragonkai added the procedure Action that must be executed label Oct 19, 2021
@CMCDragonkai CMCDragonkai changed the title Parallelise our CI/CD Parallelise our CI/CD jobs Oct 19, 2021
@tegefaulkes
Copy link
Contributor

It would be pretty simple to make multiple jobs for running tests. the question is how granular would we want the tests to be? I assume there is some cost to using more runners at a time.

I can do 1 test job per domain but really we just need to split the expensive ones up and keep or the trivial tests together.

@CMCDragonkai
Copy link
Member Author

Because it is open source, it costs free!

I suppose this has a relationship with optimising testing. So for tests that have complicated side effects that are centralised then it can be done separately. More info in #264.

@CMCDragonkai
Copy link
Member Author

It is being attempted here: #292

Although I think better grouping than 1/16. some way of tagging the job with the domain name so we can see it here.

image.png

@CMCDragonkai
Copy link
Member Author

For test splitting, we can use dynamic parent child pipelines in gitlab. This is rather primitive, as we would then probably split on the top level directories, but that's better than nothing and better than hardcoding all the test splits atm. It can even split by directories and all test files at root can be tested directly by themselves too.

Being done in #292.

@CMCDragonkai
Copy link
Member Author

We are now automatically splitting all tests jobs by subdirectories. Any subdirectory in tests ends up as their own test job. This allows us future flexibility for further subdivision by just creating more subdirectories.

Note that this works in a hierarchical way. Test modules in a parent directory get executed as their own job, but all subdirectories are separate jobs.

@CMCDragonkai
Copy link
Member Author

Doing this led to some discoveries regarding how gitlab CI/CD variables are used across pipelines. Recommend reading #292 (comment) for any future work/maintenance on this feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
procedure Action that must be executed
Development

Successfully merging a pull request may close this issue.

2 participants