Skipped file because the analyzer didn't understand it

[thales92P]

Hi,

I don't really know if it is a bug report actually but I don't know how to call this since it is described as an information instead of a warning

During the indexation, I meet a lot of Skipped file because the analyzer did not understand it and I just would like to know why this message because I have a lot of them and most of them from c++ files.

The thing is, I would like these files to be analyzed ... because if not, it is impossible to find them with indexation (logic)

Any idea to change something ?

[vladak]

Need more context + reproducible test case.

[thales92P]

oups, my bad,

command launche on a linux virtual machine (RH 7.6) with opengrok-1.3.4 and ctags 5.8. files are on a network.

Is it enough for you ? I don't know what to tell more

Thanks in advance

[vladak]

Not really, more info is needed. Ideally, the content of the files that exhibit the problem. If you cannot provide the content then some isolated/redacted version that still reproduces the problem.

Also, could you try with latest OpenGrok and latest Universal ctags ?

[thales92P]

hum ok thanks for your reply, in that case I will try latests versions and tell you further the result.

[thales92P]

Hello, So, I can't provide the content but we are using opengrok 1.3.4 ctags 5.8. I tried with the latest versions of opengrok 1.5.5 but I got an error due to java version.

About the skipping, here is the exact message I get : INFO: Skipped file '...file' because the analyzer didn't understand it.

On the interface, this result to a file only accessible by clicking "raw".

Do you think this will disappear with new versions of opengrok ? I'd like to be sur of this before upgrading our use of java.

[vladak]

The recent OpenGrok versions require Java 11. You can try this in the official OpenGrok Docker container.

In order to do anything meaningful we need to have reproducible case, i.e. you need to strip down the file so that it does not include any proprietary/closed information and yet still reproduce the problem.

[debjyoti-samanta]

Facing similar issue, files are getting skipped. used latest official docker image latest/master.

Ex:- 2021-07-14T21:49:10.148960021+00:00 stdout F INFO: Skipped file '/-/-/-/-/CODEOWNERS' because the analyzer didn't understand it. 2021-07-14T21:49:10.149506633+00:00 stdout F INFO: Skipped file '/-/-/-/-/-/-/-/-/-/-/reset_details.py' because the analyzer didn't understand it. 2021-07-14T21:49:10.149539252+00:00 stdout F INFO: Skipped file '/-/-/-/-/-/-/-/-/-/lib.c' because the analyzer didn't understand it.

Docker run command:- "docker run -d --name opengrok-latest -p :/tcp -e SYNC_PERIOD_MINUTES="1440" -e NOMIRROR="non-empty" -e WORKERS="1" -e INDEXER_OPT="-P -S -H -r on --progress" -v //src/:/opengrok/src/ -v //etc/:/opengrok/etc/ -v /****/data/:/opengrok/data/ opengrok/docker:master"

Docker Image:- docker.io/opengrok/docker master 9e******* 13 days ago 1.27 GB

Please let me know if i am missing something here, wants index a git repo. is this because of ctag?

Please do let me know if you need any other info.

[vladak]

A reproducible case is needed to do anything here. This means the actual contents of the files.

[debjyoti-samanta]

Hi @vladak Thank you for quick reply. Yes able to reproduced the same and looks to found the issue, it observed with submodule directory.

Steps to reproduce:-

1. create src dir

2. clone :- https://github.com/debjyoti-samanta/main_repo.git

3. cd main_repo; git submodule update --init .

4. execute:- docker run -d --name opengrok-latest -p 8080:8080/tcp -e SYNC_PERIOD_MINUTES="0" -e NOMIRROR="non-empty" -e WORKERS="1" -v /home/ubuntu/src/:/opengrok/src/ -v /home/ubuntu/etc/:/opengrok/etc/ -v /home/ubuntu/data/:/opengrok/data/ opengrok/docker:master

5. {"log":"INFO: Skipped file '/main_repo/submd_repo/submd.py' because the analyzer didn't understand it.\n","stream":"stdout","time":"2021-07-15T11:24:46.344134935Z"} {"log":"INFO: Skipped file '/main_repo/submd_repo/README.md' because the analyzer didn't understand it.\n","stream":"stdout","time":"2021-07-15T11:24:46.344143563Z"}

Note:- in older version we never face this issue.

Please let me know if you required any other info. Thanks in advance.

[vladak]

I checked https://github.com/debjyoti-samanta/main_repo.git and the analyzer error is just a manifestation of a problem with getting history for given file - IndexDatabase#addFile() calls populateDocument() where NPE is produced in GitRepository (because it failed to generate history cache in the first phase of indexing it tries to retrieve the history per file in the second phase of indexing which is a bug on its own) and given it catches Exception this error will appear. Created #3666 to track this as I believe this is different issue than the one originally reported here.

[debjyoti-samanta]

Thank you @vladak. Was facing this issue since couple of days.

[vladak]

This would not happen after #4456, hence closing.