Rethink CRUD with large tables (e.g., server side rendering or graphQL)

[philippotto]

Tables with many entries (e.g., annotations or tasks) are currently a bit buggy and have code that is difficult to reason about.

The main problem is that we don't fetch all items for performance reasons and rely on pagination which is controlled by the front-end. From that, multiple problems follow:

• searching/filtering all items reliably is only possible by fetching all pages first (the user has to click on "load next page" until none are there anymore) and then using the front-end search
• tracking which pages were already loaded does not work properly when using the search (since that likely changes the amount of pages)
• the detection whether the front-end has loaded all pages is only a heuristic right now (but could be improved in some way of course)
  • this should probably make use of the includeTotalCount parameter
• other bugs can easily arise (see Merge "Shared Annotations" with "My annotations" #6230 (review) as an example)

The above bugs might be fixable while keeping the current architecture, but I think it might be better to go another way. For example: To improve this situation, we could make the integration with the back-end tighter by:

• using server side rendering for the table or
• using something like graphql and/or relay to piggyback on their robustness
• other alternatives?

These options probably require some research and discussion.

[philippotto]

react-query was quite useful for implementing #6367. it could be useful here, too, but I think there are still open questions (e.g., how to do the pagination while data is being filtered).

[fm3]

What lists suffer from these problems? We changed the dataset list so that it is filtered in the backend. I think we could do the same for other lists as well, e.g. annotations. I’m not sure I understand all of your points above, do you think that more is needed?

[philippotto]

I think the main culprit of the problems listed above was the front-end side searching for datasets. Since this isn't the case, anymore, the main problems should be gone now. In theory, a back-end search for the annotation table might also be a good idea, but I think the amount of annotations for a single user isn't that big (compared to the amount of datasets).

Therefore, I'd close this issue for now. We can reconsider it, if we notice problems for the annotation table. We have a good blueprint from the datasets table now, I think.