History is enabled in index.sh, but no git history in webapp when running indexer without -S

[Ymoise]

I'm running in a container based on a docker image built from opengrok/docker, and an altered index.sh that currently looks like this:

#!/bin/bash

LOCKFILE=/var/run/opengrok-indexer
URI="http://localhost:8080"
OPS=${INDEXER_FLAGS:='-H -P -G'}

if [ -f "$LOCKFILE" ]; then
        date +"%F %T Indexer still locked, skipping indexing"
        exit 1
fi

touch $LOCKFILE

date +"%F %T Indexing starting"
opengrok-indexer \
    -J=-Xmx8g \
    -J=-server \
    -a /opengrok/lib/opengrok.jar -- \
    -i hugeMatlabFile.mat \
    -m 256 \
    -v \
    --repository /opengrok/src/master \
    -s /opengrok/src \
    -d /opengrok/data \
    --remote on \
    -W /opengrok/etc/configuration.xml \
    -U "$URI" \
    $OPS \
    $INDEXER_OPT "$@"
date +"%F %T Indexing finished"

---

The index forms ok, and seems fine to a cursory examination, but the history link doesn't work, and there's no list of "current version" with last commit on the web app front page, which I get with the 1.5 year old version, if I run it on the same files.

---

The only severe I found in the log is a handful of these:

WARNING: CTags parsing problem:
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at org.opengrok.indexer.analysis.Ctags.readTags(Ctags.java:547)
at org.opengrok.indexer.analysis.Ctags.lambda$doCtags$2(Ctags.java:455)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Mar 16, 2020 11:31:08 PM org.opengrok.indexer.analysis.Ctags readTags
SEVERE: CTag reader cycle was interrupted!

---

Is this a settings issue? Did I put in the wrong flag somewhere? I mean, as far as I understand it, the -H means history is on, no?

Or is it the severe?

Either way, please advise... and please excuse my being a pain.

[Ymoise]

On a side question - it currently takes each container ~25 minutes to re-index after a git pull.

If I fix the history issue, am I risking that number increasing in the extreme?

Re-indexing, using the 1.5 year old version consistently took about 7hrs, even if only one file had been changed.

At the cost of having to tell the developers "If you want a history - go to the repo and look it up", I'd keep the history off if it'll let me keep the short update time.

25 minutes is doable. 7hrs is not.

Will this put me back in 7hrs territory?

[vladak]

You do not need to alter index.sh in order to insert indexer options, there is a env variable for that, see https://github.com/oracle/opengrok/tree/master/docker#environment-variables

[vladak]

Any Ctags problem cannot be related to history generation because it happens in second stage of indexing after the history was already generated (if enabled).

Check the indexer logs (see https://github.com/oracle/opengrok/tree/master/docker#indexer-logs) to see what happened with the history. It should say something like this near the beginning of the indexing when run with -H -S:

Mar 17, 2020 10:30:09 AM org.opengrok.indexer.util.Statistics report
INFO: Done scanning for repositories, found 7 repositories (took 555 ms)
Mar 17, 2020 10:30:09 AM org.opengrok.indexer.index.Indexer prepareIndexer
INFO: Generating history cache for all repositories ...
Mar 17, 2020 10:30:09 AM org.opengrok.indexer.history.HistoryGuru createCacheReal
INFO: Creating historycache for 7 repositories
Mar 17, 2020 10:30:09 AM org.opengrok.indexer.history.HistoryGuru createCache
INFO: Creating historycache for /var/opengrok/src/foo (MercurialRepository) without renamed file handling
Mar 17, 2020 10:30:09 AM org.opengrok.indexer.history.HistoryGuru createCache
INFO: Creating historycache for /var/opengrok/src/sudo (GitRepository) without renamed file handling
...
Mar 17, 2020 10:30:09 AM org.opengrok.indexer.util.Statistics report
INFO: Done historycache for /var/opengrok/src/sudo (took 20 ms)
Mar 17, 2020 10:30:10 AM org.opengrok.indexer.util.Statistics report
INFO: Done historycache for /var/opengrok/src/foo (took 194 ms)
INFO: Done historycache for all repositories (took 2.795 seconds)
Mar 17, 2020 10:30:12 AM org.opengrok.indexer.index.Indexer prepareIndexer
INFO: Done...

If I run the indexer just with -S and without -H the logs will look like this:

Mar 17, 2020 10:31:40 AM org.opengrok.indexer.index.Indexer prepareIndexer
INFO: Generating history cache for all repositories ...
Mar 17, 2020 10:31:40 AM org.opengrok.indexer.history.HistoryGuru createCacheReal
INFO: Creating historycache for 7 repositories
Mar 17, 2020 10:31:40 AM org.opengrok.indexer.history.HistoryGuru createCache
INFO: Skipping history cache creation of MercurialRepository repository in /var/opengrok/src/foo and its subdirectories
Mar 17, 2020 10:31:40 AM org.opengrok.indexer.history.HistoryGuru createCache
INFO: Skipping history cache creation of GitRepository repository in /var/opengrok/src/opengrok and its subdirectories
Mar 17, 2020 10:31:40 AM org.opengrok.indexer.history.HistoryGuru createCache
...
Mar 17, 2020 10:31:40 AM org.opengrok.indexer.util.Statistics report
INFO: Done historycache for all repositories (took 9 ms)
Mar 17, 2020 10:31:40 AM org.opengrok.indexer.index.Indexer prepareIndexer
INFO: Done...

If I run the indexer without -S and -H it will say just:

INFO: Generating history cache for all repositories ...
Mar 17, 2020 10:33:35 AM org.opengrok.indexer.history.HistoryGuru createCacheReal
INFO: Creating historycache for 0 repositories
Mar 17, 2020 10:33:35 AM org.opengrok.indexer.util.Statistics report
INFO: Done historycache for all repositories (took 5 ms)
Mar 17, 2020 10:33:35 AM org.opengrok.indexer.index.Indexer prepareIndexer
INFO: Done...

I.e. without scanning for repositories (or getting them from read-only configuration) there is no way how history cache can be generated.

A for the Ctags failure, without seeing more context of the log it is hard to tell what happened. Possibly the ctags process went away for some reason.

[vladak]

On a side question - it currently takes each container ~25 minutes to re-index after a git pull.

If I fix the history issue, am I risking that number increasing in the extreme?

Re-indexing, using the 1.5 year old version consistently took about 7hrs, even if only one file had been changed.

At the cost of having to tell the developers "If you want a history - go to the repo and look it up", I'd keep the history off if it'll let me keep the short update time.

25 minutes is doable. 7hrs is not.

Will this put me back in 7hrs territory?

The simplest way how to find out is to try. With containers this is particularly easy.

The history indexing for most of the modern SCMs happens incrementally so it should not add too much time. Depends on how large is each increment of course.

Again, the indexer log contains some statistics to see how long each step took (indeed, the log messages say e.g. something like Done indexing of directory /opengrok (took 3 ms)).

[vladak]

On a side question - it currently takes each container ~25 minutes to re-index after a git pull.

The long time might be due to I/O (again assuming that the updates are on the small side) as the indexer traverses the whole source directory tree. (be it source root or project root in case of per project indexing. In Docker environment this is the former case.)

You can see the time spent by directory traversal by subtracting the times between Starting traversal of directory and Starting indexing of directory. Also, you can run the indexer with --progress to see the file counts.

#3049 might actually track similar problem even though it focuses on other metrics.

Now, for SCMs based on changesets the indexer could somehow take the information about changed files from the mirror script (by looking into the "stats" of the incoming changesets) and process just these files.

[Ymoise]

So I have to have -S on to have the history on?

I have one repository in most containers... and 124 in one of them.

I took out the -S flag and added the repositories to index.sh with the --repository flag so it could skip the "searching for repositories" step which I have seen take up a whole hour - At least in the older version of opengrok.

In the above-quoted index.sh I have do one repository;

--repository /opengrok/src/master \

Would that not be enough to trigger history? It has to actually search, first?

---

As to the mirroring script, I'm running the indexer via "docker exec /scripts/index.sh". I'm not using the scripts. Are they a must now?

[vladak]

The --repository should be sufficient. The history generation just needs to have some knowledge of the repostories.

[vladak]

As to the mirroring script, I'm running the indexer via "docker exec /scripts/index.sh". I'm not using the scripts. Are they a must now?

In the current docker image version the mirroring is done using the mirror script:

opengrok/docker/index.sh

Line 17 in fd17fd1

opengrok-mirror --all --uri "$URI"

The opengrok-mirror script queries the web app for list of projects with repositories to sync. The indexer updates this list at the end of indexing.

[Ymoise]

I see. Mystery solved, then.

Thank you.

[vladak]

Can you see the history link now ? What was the problem then ?