Impossible to add some GitLab repositories as a Project when you're a maintainer of more than 100 repositories

[jankeromnes]

Bug description

When you want to add a repository as a new Project in Gitpod, Gitpod first gets the list of all the possible repositories you could add (grouped by account), and then lets you find the repository you want to add via scrolling or searching/filtering.

Unfortunately, in the case of GitLab, we only fetch the first 100 repositories that you could add:

gitpod/components/server/ee/src/gitlab/gitlab-app-support.ts

Line 54 in 1d52870

const projectsWithAccess = await api.Projects.all({ min_access_level: "40", perPage: 100 });

This is a problem, because if you want to add a specific repository that is not present in the first 100 repositories returned by GitLab, you will never be able to see it or find it in the Gitpod UI -- thus adding it as a project is totally impossible.

Steps to reproduce

1. Be a maintainer of more than 100 GitLab repositories
2. Try to create a new project for a repository that is not listed in the first 100 results returned by GitLab

Workspace affected

No response

Expected behavior

• You should always be able to create a new project for a GitLab repository that you're a maintainer of (even if it's not in the first 100 results from GitLab)

Example repository

No response

Anything else?

Possible solutions:

• Paginate the GitLab API call in order to fetch more than just the first 100 repositories
• Allow creating projects by manually entering a repository URL

[gtsiolis]

Cross-posting in case it helps, there’s support for offset-based pagination which we could use to increase the page size here. But maybe this is not available through the API? 🤷

For example:

https://gitlab.com/api/v4/projects?min_access_level=40
https://gitlab.com/api/v4/projects?min_access_level=40&per_page=2000

See relevant discussion (internal).

[jankeromnes]

There is actually one workaround: Using the browser console to create a Project manually.

Say I want to add my repository https://gitlab.com/jankeromnes/test-before as a Project to one of my Teams, but for some reason, Gitpod's UI won't show it to me. Then, I can still do this:

1. Run await window._gp.gitpodService.server.getTeams() in order to find the team ID under which I want to create the Project
2. Then, run the following (just replace the name, slug, cloneUrl and teamId -- appInstallationId is useless but the API requires it):

await window._gp.gitpodService.server.createProject({
    name: "test-before",
    slug: "test-before",
    cloneUrl: "https://gitlab.com/jankeromnes/test-before.git",
    teamId: "5db3013f-31d1-4d8e-9445-0a746b43edde",
    appInstallationId: "",
})

[jankeromnes]

@gtsiolis Good catch!

https://gitlab.com/api/v4/projects?min_access_level=40&per_page=2000

Unfortunately, this isn't quite as simple, because the maximum allowed page length is 100:

The proper solution to get all results would be to implement API pagination, i.e.:

1. Query the first 100 results (i.e. page 1)
2. If the response provides a hint that there is a next page (typically via a response header), we do step 1. again but for the next page (e.g. we fetch page 2, then 3, then 4... until there is no next page anymore and we have all the results)

[jankeromnes]

Apparently, Gitbeaker (the library we use to fetch GitLab data) is already supposed to handle pagination and return all results (even if more than 100): https://github.com/jdalrymple/gitbeaker#offset-pagination 🤔

But maybe that feature or the version of Gitbeaker we use has a bug with pagination? Or we're not using it properly? 🤷

EDIT: Confirmed by @mrzarquon in Slack (internal):

pagination was broken until 2021 - jdalrymple/gitbeaker#1880 - we're using "@gitbeaker/node": "^25.6.0" which was shipped in 2020

So, the fix could be as "simple" as updating our @gitbeaker/node library to a version that is not super old. 😇

[mrzarquon]

These are the core changes in the pagination handling with what we're using and what is the latest version:

https://github.com/jdalrymple/gitbeaker/blob/25.6.0/packages/gitbeaker-core/src/infrastructure/RequestHelper.ts#L89-L103

vs

https://github.com/jdalrymple/gitbeaker/blob/35.7.0/packages/core/src/infrastructure/RequestHelper.ts#L129-L143

[AlexTugarev]

@jankeromnes, are you planing to work on this? If not, I'd start working on it.

[AlexTugarev]

First step: bumping GL library in #13001