WARNING: Error from ctags: ctags: Warning: cannot open input file: No such file or directory

[totoroliu]

On Ubuntu 18.04 machine, I just upgrade:
Opengrok: from v1.3.11 to v1.5.5
ctags: from /usr/local/bin/ctags v5.8 (e87283aa) Mar 30 2020 to /snap/bin/universal-ctags v5.8 (e852ee0e) Oct 7 2020
java: from java-1.8.0-openjdk-amd64 to java-1.11.0-openjdk-amd64
Tomcat: 9.0.16-3ubuntu0.18.04.1
Git: 2.17.1

Then,
when I run index update:
#!/bin/bash -x
time java -server -Xms20G -Xmx20G \
-Djava.util.logging.config.file=/opengrok/etc/logging.properties \
-jar /opengrok/dist/lib/opengrok.jar \
-c /snap/bin/universal-ctags \
-s /opengrok/src -d /opengrok/data -H -P -S -G \
-v --progress \
--threads 6 --historyThreads 6 -m 15360 \
-W /opengrok/etc/configuration.xml -U http://localhost:8080/source

I got tons of WARNINGs (probably one WARNING per one source file under /opengrok/src)
about ctags cannot open input file (No such file or directory):
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_efuse.h" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/Kconfig.series" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Rega-Sense" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/Kconfig.soc" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_event.h" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/base/component.c" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/linker.ld" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Video2000" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/soc.h" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/sound/firewire/oxfw/oxfw.c" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_fifoctrl_bitdef.h" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/fs/hfs/mdb.c" : No such file or directory
11:27:14 WARNING: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Zuerich-upc-cablecom" : No such file or directory

but when I check the file path, the file does exist there:
-rw-rw-r-- 1 devops devops 21092 Jun 2 20:05 /opengrok/src/Yocto-Valkyrie/kernel/drivers/base/component.c
-rw-rw-r-- 1 devops devops 1614 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_efuse.h
-rw-rw-r-- 1 devops devops 2359 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_event.h
-rw-rw-r-- 1 devops devops 3269 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/kernel/drivers/staging/rtl8712/rtl8712_fifoctrl_bitdef.h
-rw-rw-r-- 1 devops devops 10371 Aug 13 20:05 /opengrok/src/Yocto-Valkyrie/kernel/fs/hfs/mdb.c
-rw-rw-r-- 1 devops devops 10303 Apr 8 2020 /opengrok/src/Yocto-Valkyrie/kernel/sound/firewire/oxfw/oxfw.c
-rwxrwxr-x 1 devops devops 3733 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Rega-Sense
-rwxrwxr-x 1 devops devops 218 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Video2000
-rwxrwxr-x 1 devops devops 201 Mar 30 2020 /opengrok/src/Yocto-Valkyrie/meta-openembedded/meta-multimedia/recipes-multimedia/dvb-apps/files/dvb-scan-table/dvb-c/ch-Zuerich-upc-cablecom
-rw-rw-r-- 1 devops devops 257 May 8 18:53 /opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/Kconfig.series
-rw-rw-r-- 1 devops devops 295 May 8 18:53 /opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/Kconfig.soc
-rw-rw-r-- 1 devops devops 141 May 8 18:53 /opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/linker.ld
-rw-rw-r-- 1 devops devops 3010 May 8 18:53 /opengrok/src/Yocto-Valkyrie/zephyr/arch/riscv32/soc/riscv-privilege/miv/soc.h

Are these ctags ERROR (opengrok WARNING) correct?
How to debug or fix these these ERROR/WARNING?
Does opengrok index update break?

[vladak]

Is there anything interesting in indexer logs ? (other than the errors from ctags)

Firstly, upgrade only one thing at a time. Try upgrading just Java and OpenGrok first. See if it breaks the indexer. Then upgrade Universal ctags.

[vladak]

To get more info of what is happening run the indexer with --threads 1 so that only single ctags process is created. Then strace both the ctags process and the indexer process to see what is going on w.r.t. I/O syscalls.

Also, what is /opengrok/src exactly ?

[vladak]

Also, does this happen with all projects under source root or just some of them ?

[totoroliu]

opengrok18.0.log.gz
2020-10-30 11:27:14.918-0700 WARNING t10324 Ctags.lambda$run$0: Error from ctags: ctags: Warning: cannot open input file "/opengrok/src/Yocto-Valkyrie/kernel/drivers/base/component.c" : No such file or directory
2020-10-30 11:27:14.945-0700 FINEST t10314 Statistics.logIt: file '/opengrok/src/Yocto-Valkyrie/kernel/drivers/base/component.c' indexed (took 33 ms)

$ tree -L 2 /opengrok/src/
/opengrok/src/
├── Creole
│   └── creole
├── NetXtreme
│   └── netxtreme
├── Trunk:MxC
│   ├── brcm
│   ├── ing
│   ├── map
│   ├── meego
│   ├── meego.git
│   ├── mirrors
│   ├── opensource
│   ├── os
│   ├── qa
│   ├── repo_list
│   └── tizen
└── Yocto-Valkyrie
├── arm-trusted-firmware
├── bcm-secimage
├── ffmpeg
├── halapis
├── kernel
├── manifest
├── meta-brcm
├── meta-openembedded
├── meta-raspberrypi
├── poky
├── qemu
├── sbi
├── sec-keys
├── vkcmodel
├── vkfirmware
├── vkh264dec
├── vkil
├── vkipc
├── vkservices
├── vksim
├── vktest
├── vkutils
└── zephyr

[totoroliu]

it looks like happen to all the files under source root for all projects
I got 19 log file after running the re-indexing for all of 4 projects
The gzip tarball of all of 19 log files is 58MB.

[totoroliu]

opengrok19.0.log.gz
this log has ERROR for "/opengrok/src/Creole/creole/ project:

[totoroliu]

opengrok7.0.log.gz
This log has Error for /opengrok/src/Trunk:MxC/ and /opengrok/src/NetXtreme/netxtreme/

[totoroliu]

I'll try --threads 1 now.

[vladak]

$ tree -L 2 /opengrok/src/
/opengrok/src/
├── Creole

I meant - is there something special about the file system holding this data ?

[totoroliu]

The root / filesystem is on LVM
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/lvm-root 148G 83G 59G 59% /

devops@vpx-opengrok:/opengrok$ ls -l /opengrok/src
total 16
drwxrwxr-x 3 devops devops 4096 Mar 30 2020 Creole
drwxrwxr-x 3 devops devops 4096 Oct 7 11:16 NetXtreme
drwxrwxr-x 12 devops devops 4096 Mar 30 2020 Trunk:MxC
drwxrwxr-x 25 devops devops 4096 Mar 30 2020 Yocto-Valkyrie
devops@vpx-opengrok:/opengrok$ ls -l /opengrok/src/*
/opengrok/src/Creole:
total 4
drwxrwxr-x 18 devops devops 4096 Oct 30 16:17 creole

/opengrok/src/NetXtreme:
total 4
drwxrwxr-x 5 devops devops 4096 Oct 22 00:00 netxtreme

'/opengrok/src/Trunk:MxC':
total 108
drwxrwxr-x 3 devops devops 4096 Mar 30 2020 brcm
drwxrwxr-x 3 devops devops 4096 Mar 30 2020 ing
drwxrwxr-x 120 devops devops 4096 Mar 30 2020 map
drwxrwxr-x 453 devops devops 16384 Mar 30 2020 meego
drwxrwxr-x 3 devops devops 4096 Mar 30 2020 meego.git
drwxrwxr-x 4 devops devops 4096 Mar 30 2020 mirrors
drwxrwxr-x 101 devops devops 4096 Mar 30 2020 opensource
drwxrwxr-x 4 devops devops 4096 Mar 30 2020 os
drwxrwxr-x 17 devops devops 4096 Mar 30 2020 qa
-rw-rw-r-- 1 devops devops 53876 Mar 30 2020 repo_list
drwxrwxr-x 4 devops devops 4096 Mar 30 2020 tizen

/opengrok/src/Yocto-Valkyrie:
total 92
drwxrwxr-x 20 devops devops 4096 Oct 14 10:05 arm-trusted-firmware
drwxrwxr-x 6 devops devops 4096 Apr 18 2020 bcm-secimage
drwxrwxr-x 19 devops devops 4096 Sep 25 04:05 ffmpeg
drwxrwxr-x 28 devops devops 4096 Aug 21 20:05 halapis
drwxrwxr-x 26 devops devops 4096 Aug 13 20:05 kernel
drwxrwxr-x 3 devops devops 4096 Sep 24 20:05 manifest
drwxrwxr-x 15 devops devops 4096 Apr 2 2020 meta-brcm
drwxrwxr-x 14 devops devops 4096 Apr 30 2020 meta-openembedded
drwxrwxr-x 18 devops devops 4096 Mar 30 2020 meta-raspberrypi
drwxrwxr-x 12 devops devops 4096 Apr 3 2020 poky
drwxrwxr-x 45 devops devops 4096 Mar 30 2020 qemu
drwxrwxr-x 11 devops devops 4096 Aug 7 20:05 sbi
drwxrwxr-x 5 devops devops 4096 Sep 17 20:05 sec-keys
drwxrwxr-x 4 devops devops 4096 Mar 30 2020 vkcmodel
drwxrwxr-x 8 devops devops 4096 Aug 4 20:05 vkfirmware
drwxrwxr-x 7 devops devops 4096 Jul 1 04:05 vkh264dec
drwxrwxr-x 8 devops devops 4096 Mar 30 2020 vkil
drwxrwxr-x 9 devops devops 4096 Jul 24 20:05 vkipc
drwxrwxr-x 6 devops devops 4096 Mar 30 2020 vkservices
drwxrwxr-x 8 devops devops 4096 Sep 30 04:05 vksim
drwxrwxr-x 9 devops devops 4096 Oct 8 04:05 vktest
drwxrwxr-x 11 devops devops 4096 Oct 29 04:05 vkutils
drwxrwxr-x 19 devops devops 4096 May 8 18:53 zephyr

[totoroliu]

opengrok0.0.log.gz
this is the log for --threads 1:

#!/bin/bash -x
time java -server -Xms20G -Xmx20G \
    -Djava.util.logging.config.file=/opengrok/etc/logging.properties \
    -jar /opengrok/dist/lib/opengrok.jar \
    -c /snap/bin/universal-ctags \
    -s /opengrok/src -d /opengrok/data -H -P -S -G \
    -v --progress \
    --threads 1 --historyThreads 6 -m 15360 \
    -W /opengrok/etc/configuration.xml -U http://localhost:8080/source

[vladak]

There's nothing interesting in the log from what I can tell. You need to run strace in the ctags program while it is being executed by the indexer to see more.

[totoroliu]

how to run strace in ctags program while it's been executed by indexer?
do you have instruction?