Calling opengrok-projadm from opengrok-sync fails when multiple works are used

[ohm314]

Describe the bug

I've been trying to pack as much of the nightly sync workflow into the opengrok-sync configuration. That includes opengrok-projadm, which is run per project. This tool uses a a file lock, though, so when running the sync tool with multiple works we end up in the situation where a bunch of opengrok-projadm calls fail because they cannot acquire a lock. Instead they should run one after the other (or have a generous timeout set).

opengrok version: 1.7.18

To Reproduce

1. Setup an opengrok with multiple projects
2. create a sync.conf along these lines:

commands:
- command: [opengrok-projadm, --jar, /opengrok/dist/lib/opengrok.jar, -l, DEBUG, -b, /opengrok, -a, '%PROJECT%']
- command: [opengrok-mirror, -c, /opengrok/etc/mirror_config.yaml, -l, DEBUG, -U, http://localhost:8080/source]
- command: [opengrok-reindex-project, bunch, of, arguments]

Expected behavior
I would expect opengrok-projadm to be executed per project, but if locking is required that other instances wait their turn. Checking projadm code does not reveal an option being passed to FileLock for a timeout. I am not sure about opengrok-mirror but I think the way I am running it there will be one call that will itself iterate over all projects, so that's fine.

[vladak]

I'd like to understand the use case first. How do you run opengrok-sync and why do you need opengrok-projadm as part of the workflow ?

[ohm314]

My use-case is that we have many repositories organized into a number of opengrok projects. Developers create new repos when they start a new project, and sometimes a repo is deleted/archived. I would like our source search to reflect this.

A nightly cronjob first runs a script that queries our gitlab to check for new or removed repositories and clones/deletes them in /opengrok/src/[project].

then I run opengrok-sync in the cron job like this:

opengrok-sync -l INFO -w 2 -c /opengrok/etc/sync.conf  \
        -U http://localhost:8080/source -d /opengrok/src/

As we discussed a while back in #3421 the way to make sure the changed repo lists are reflected in opengrok is to first readd the projects, which is what I do first with opengrok-projadm inside sync.conf but these calls unfortunately cannot be run in parallel.
As a workaround I have moved this command out of sync.conf which seems fine, but I was getting failures for the longest time and it was not at all obvious why.

[vladak]

Before I take a look at the lock file names used by the various Python tools let me muse a bit about the use case:

The use case makes sense. I'd be careful with deleting large amount of files under a project, though. When the files for the deleted repositories are gone from the file system and a search request finds a hit in one of them before reindex is done, this will lead to errors because the Lucene documents for these files are still present in the index. So, when the results are displayed, this will make it impossible for the web app to present the context for these files. A "big hammer" way to approach this would be to:

1. mark the project as not indexed (using RESTful API) so that it is not displayed by the web application anymore
2. reindex the project

This will avoid the inconsistency window.

Alternatively, it might make sense to kick off per project reindex right after the file system change is done. However, one needs to be aware of this: when opengrok-sync is run for single project (-P), it will use dedicated lock file so it will not clash with the global opengrok-sync run. In this case this could be a problem because the indexer itself does not synchronize the running instances I think.

There are some techniques for avoiding the inconsistency window altogether by using file system level snapshots and clones. I will not go into the detail now as I don't know the limitations in your environment.

Another consideration is to make data deletion more granular. OpenGrok can only delete (using the RESTful API) with project granularity currently. That would be an enhancement, though.

[vladak]

A workaround would be to use direct API call instead of invoking opengrok-projadm. After all, the tool is just a wrapper to the RESTful API calls. The commands specification in the opengrok-sync configuration should be able to do that. Something like this should work for project addition:

- command:
  - 'http://localhost:8080/source/api/v1/projects'
  - POST
  - '%PROJECT'
  - 'Content-type': 'text/plain'

[ohm314]

Thanks for the comments!

I'd be careful with deleting large amount of files under a project, though. When the files for the deleted repositories are gone from the file system and a search request finds a hit in one of them before reindex is done, this will lead to errors because the Lucene documents for these files are still present in the index. So, when the results are displayed, this will make it impossible for the web app to present the context for these files.

Yes, I was actually thinking about that issue, but couldn't find a good solution so far.

1. mark the project as not indexed (using RESTful API) so that it is not displayed by the web application anymore
2. reindex the project

You mean though to do the file operations (deleting/adding repos) in between 1 and 2, right? Also, if you mark a project as not indexed, will it reindex later "from scratch" or will it take the previous index into account?

I am checking apiary, to me it looks there is no API for marking a project as not indexed. Deleting the project will drop the index.

Leaving the issue of inconsistency window aside, your idea of using the RESTful API to directly "re-add" the projects, rather than using the opengrok-projadm tool seems actually really nice and should be a good workaround at least (I still think something needs to be done about concurrent opengrok-projadm calls failing - at least some documentation).

One last point: my opengrok-reindex-project call in the sync.conf is also taking the -W argument to write the configuration once we're done reindexing a project (it's not necessary to do this also after the project add, right?). If multiple instances of opengrok-reindex-project are running from opengrok-sync, won't that lead to a race condition in writing the configuration.xml ?

[vladak]

1. mark the project as not indexed (using RESTful API) so that it is not displayed by the web application anymore
2. reindex the project

You mean though to do the file operations (deleting/adding repos) in between 1 and 2, right? Also, if you mark a project as not

yes.

indexed, will it reindex later "from scratch" or will it take the previous index into account?

As long as the index is still there, there will be incremental reindex performed.

I am checking apiary, to me it looks there is no API for marking a project as not indexed. Deleting the project will drop the index.

It is a per project property, documented on https://github.com/oracle/opengrok/wiki/Per-project-configuration#properties so this works:

curl -d "False" -X PUT http://localhost:8080/source/api/v1/projects/<INSERT_PROJECT_NAME_HERE>/property/indexed

except it currently throws an error (filed PR #3731 to fix it) that can be ignored.

Leaving the issue of inconsistency window aside, your idea of using the RESTful API to directly "re-add" the projects, rather than using the opengrok-projadm tool seems actually really nice and should be a good workaround at least (I still think something needs to be done about concurrent opengrok-projadm calls failing - at least some documentation).

Yep, will look into that.

One last point: my opengrok-reindex-project call in the sync.conf is also taking the -W argument to write the configuration once we're done reindexing a project (it's not necessary to do this also after the project add, right?). If multiple instances of opengrok-reindex-project are running from opengrok-sync, won't that lead to a race condition in writing the configuration.xml ?

Definitely opengrok-reindex-project should not be run with -W. The configuration changes should happen using the process described on https://github.com/oracle/opengrok/wiki/Per-project-management-and-workflow

[ohm314]

Hello!
Hope it's OK if I simply continue in this thread - since it's a bit of a side project for me - I continue on and off working on this. I've been trying to implement your suggestions in my setup and have a couple follow-up questions:

https://github.com/oracle/opengrok/wiki/Per-project-management-and-workflow
The example sync.conf shown here mentions a %PROJ% placeholder, I suppose that's an error? (this gets us back to our discussion in #3730 to move documentation out of wiki such that interested users can help improve the documentation, btw.)

The thing that is unclear from the per project management workflow is what happens after the opengrok-sync. Once the opengrok-reindex-project returns for all the projects, shoudl there not be one command issuesd that would write the final configuration to disk? I see in the document somewhere pengrok-projadm -b /opengrok -r described, but is it necessary to call this per project? is it not enough to write the configuration once?

So my current (improved) workflow is:

• disable index for all projects
• add/remove repositories
• call opengrok-sync which will
  • do a curl to re-add projects
  • run opengrok-mirror
  • run opengrok-reindex-project
• reenable all project indices

I think after that I shoudl still write the configuration.xml merging it with the read-only config. Is that correct?

[vladak]

Hi! we can hijack this issue for the discussion even though the newly introduced Discussions might be better place.

The %PROJ% used in the example configuration is intentional. This is because %PROJECT% in opengrok-sync configuration is replaced by project name that is currently being processed and the -p option value is a template that is processed by opengrok-reindex-project to expand the process name in the logging template. If -p was used with %PROJECT%, the option argument would be replaced with project name and the logging template expansion would not happen.

Strictly speaking, it is not absolutely necessary to retrieve the configuration from the web app and write it to disk after opengrok-sync is done, at least in the usual case when project/repository structure has not changed. It is good idea though, because it will make the fresh RepositoryInfo objects persistent. Otherwise the web app would display old data in the 'Current version' column on the index page if it was redeployed, until new reindex would take place.

As for the documentation, I discussed this with @ahornace and the folks responsible for Oracle's Github organization and concluded we do not have the will/capacity to move the wiki to a repo that would accept external contributions more easily at this moment.

[ohm314]

Hi! we can hijack this issue for the discussion even though the newly introduced Discussions might be better place.

Cool! Great, you enabled github Discussions, that will be helpful in the future!

The %PROJ% used in the example configuration is intentional. This is because %PROJECT% in opengrok-sync configuration is replaced by project name that is currently being processed and the -p option value is a template that is processed by opengrok-reindex-project to expand the process name in the logging template. If -p was used with %PROJECT%, the option argument would be replaced with project name and the logging template expansion would not happen.

Ah, I missed that - sorry. Maybe it could be useful to mention this in the workflow project page since it appears there in the sync.conf example.

Strictly speaking, it is not absolutely necessary to retrieve the configuration from the web app and write it to disk after opengrok-sync is done, at least in the usual case when project/repository structure has not changed. It is good idea though, because it will make the fresh RepositoryInfo objects persistent. Otherwise the web app would display old data in the 'Current version' column on the index page if it was redeployed, until new reindex would take place.

OK, that makes sense. I do now a final configuration retrieval via curl and the REST API and that works nicely.

As for the documentation, I discussed this with @ahornace and the folks responsible for Oracle's Github organization and concluded we do not have the will/capacity to move the wiki to a repo that would accept external contributions more easily at this moment.

That's a pity, but OK. I hope you'll reconsider this in the future. FWIW, moving the individual wiki .md files into a docs/ subdirectory of this repo and updating the links seems to me a rather minor change (from the technical perspective, but certainly there are also other aspects you have in mind.).

Thanks for taking the time to help me with all my detail questions!