AI chatting functionality (the right branch)

[InAnYan]

Mandatory checks

• [ ] Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
• [ ] Tests created for changes (if applicable)
• [ ] Manually tested changed features in running JabRef (always required)
• [ ] Screenshots added in PR description (for UI changes)
• [ ] Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
• [ ] Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

[InAnYan]

On my machine, when I run "checkstyleMain" from Gradle commands, I have no such errors. It has successfully generated for me a list of violations

[Siedlerchr]

Then it seems reviewdog seems to have an outdated version of checkstyle

[Siedlerchr]

I updated the checkstyle reviewdog action to a newer version so the checkstyle works now again and can parse your code :)

[koppor]

• [x] Blocked by https://github.com/InAnYan/jabref/issues/5
• [x] Blocked by https://github.com/InAnYan/jabref/issues/6

[koppor]

The embeddings generation makes my machine unsuable. Please try to use lower priority for the indexing threads:

---

Without embeddings generation:

[ThiloteE]

Use as many threads as there are physical cores. Using LLMs, that's the most performant in my personal tests.

[koppor]

If embedding generation is cancleded, JabRef does not shut down cleanly:

Meaning: It does not shut down at all.

[koppor]

• [ ] If no libraries are opened (e..,g because they were closed), no indexing should happen

[koppor]

Please also test with a shared SQL database. See https://docs.jabref.org/collaborative-work/sqldatabase for details.

Ensure that autosave to a local .tex file is activated.

i assume that either a check for database location == LOCAL -has to be added -- or another storage path for the chat history used.

[koppor]

Good thing: seems to count files right.

@Article{test,
  file = {:test.pdf},
}
@Comment{jabref-meta: databaseType:bibtex;}

[koppor]

Bad thing: if test.pdf was made availble using the file system (and automatic relinking by JAbRef), it is not indexed:

To reproduce: Use BibTeX from above. Open JabRef. Check indexing. Then copy test.pdf to a folder where JabRef can find the PDF.

[koppor]

Use as many threads as there are physical cores. Using LLMs, that's the most performant in my personal tests.

Maybe, we need a setting. If the thing takes 1 hour on a "normal" machine to create embeddings and the user cannot work in parallel, this is a no-go in my oppinion. The PC should be usable without any signifcant lags.

I know that Linux is a better in multi tasking. However, there are still persons using Windows. Thus, JabRef should also be useable for Windows users.

[koppor]

• [x] Depends on https://github.com/InAnYan/jabref/pull/67

[InAnYan]

https://github.com/JabRef/jabref/pull/11430#issuecomment-2209569043

@koppor, you mean JabRef can detect if new files are added from the filesystem and then link it?

[koppor]

https://github.com/JabRef/jabref/pull/11430#issuecomment-2209569043

@koppor, you mean JabRef can detect if new files are added from the filesystem and then link it?

Link doesn't work in mobile... 🙈 Maybe cite the text? Use >.

But I didn't mean that. One can trigger it from the menu. Lookup "Search for unlinked files".

[koppor]

#11430 (comment) @koppor, you mean JabRef can detect if new files are added from the filesystem and then link it?

Text on the issue:

To reproduce: Use BibTeX from above. Open JabRef. Check indexing. Then copy test.pdf to a folder where JabRef can find the PDF.

@InAnYan Yes. Please check for yourself. 😅

Preferences are

Field in the entry editor:

Update There is even documentation for that: https://docs.jabref.org/finding-sorting-and-cleaning-entries/filelinks#auto-linking-files

[InAnYan]

I'm asking because I'm impressed of that feature :)

[koppor]

Wrong message parameters can lead to IOB exceptions at LangChain4J - see https://github.com/langchain4j/langchain4j/pull/1436 and my quick fix 861da12 (#11430).

[Siedlerchr]

Please fix the checkstyle issues!

[InAnYan]

@koppor did something change with GitHub, that when I'm pushing commits to ai-pr-1, they don't appear here?

Or, probably, it's a problem from my side? I've set up wrong ai-pr-1?

[koppor]

@koppor did something change with GitHub, that when I'm pushing commits to ai-pr-1, they don't appear here?

Or, probably, it's a problem from my side? I've set up wrong ai-pr-1?

You should push to upstream, not to main. 😅

I was synching now and then upstream/ai-pr-1 with origin/ai-pr-1 to keep you concentrated

Google "How to change the upstream repositgory for my branch" --> git branch -u upstream/ai-pr-1 should do the trick. Then do a normal push!

Self-driven learning: Open "gitk --all&" an try to read it. More info on gitk: https://lostechies.com/joshuaflanagan/2010/09/03/use-gitk-to-understand-git/

[InAnYan]

Hi @Siedlerchr, @calixtus! Does anyone want to delve into code review for AI functionality, while Oliver is on holidays?

You can start from AiChatTab in entry editor. Then, you can go to AiChatHistory, AiChatLogic. Also you can see how embeddings are managed (probably the ugliest part of the PR). And then the AI preferences, etc.

[InAnYan]

@koppor here is the question I posted on MVStore github: https://github.com/h2database/h2database/issues/4090

MVStore really does not support mutable values.

MVStore runs auto-commit on background, it means it will periodically check if any changes were made. It knows when values were added or removed, but it doesn't understand if values such as List or HashMap were mutated.

Your examples were working because auto-commit probably was turned on at the end of you program, so it means at the beginning of your test there was no saved changes, then you modified MVStore, and at the end there was a commit (either auto-commit or when you call close() method).

When I used List<ChatMessage>, they were saved only partially

[InAnYan]

@koppor actually I understood:

• We can't store chat history in MVStore using Lists, because Lists will be mutated, and MVStore doesn't support that
• But, we can store embeddings using records, because they will be never mutated :)

[InAnYan]

@koppor no, only now I understand: I can use also records with chat history. I just don't have to store them in List, just Map<Integer, ChatMessage> and that's all

[koppor]

@koppor actually I understood:

• We can't store chat history in MVStore using Lists, because Lists will be mutated, and MVStore doesn't support that

Please add an MWE to https://github.com/koppor/mvstore-mwe/blob/main/src/test/java/org/example/MainTest.java

[InAnYan]

Haha, know what I did?

I merged week 2 there :)

Now, I can finally continue to work on week 3 (just a little left for summarization)

[koppor]

On 0a33b34 (#11430), I radically simplified BibChatHistoryManager. Also added a test case BibChatHistoryManagerTest. Feel free to refine the test.

[InAnYan]

:flushed:

[koppor]

I tried to revert "Add workaround to get FileHistoryMenuTest running again" (commit 8ec3876c737f0144e337c397037290104f3701ed), but did not succeed - still the mockito error.