Apex Language Server - Error during startup

[brandbarr]

Summary

Consistently getting failures to start the apex language server. Logs suggest corrupted database. Processes are also being orphaned.

This is happening during the following:

1. Closing VScode and reopening.
2. Uninstalling the extension, reloading vscode and reinstalling.
3. After VSCode update restarts
4. Rolling back to previous versions of the apex extension.

Steps To Reproduce:

1. Start vscode. Fails during startup and plugin is completely unusable.

Expected result

Language server should start without issue.

Actual result

Language server fails.

Additional information

Log output:

11:37:05.242 [Apex Prelude Service STARTING] INFO  a.j.l.i.s.ApexPreludeManagedService - Scanning built-in and system Apex types.
11:37:05.251 [Apex Prelude Service STARTING] INFO  a.j.l.i.s.ApexPreludeManagedService - Scanning built-in Apex types.
11:37:05.252 [Apex Prelude Service STARTING] INFO  a.j.l.i.s.ApexPreludeManagedService - Collecting built-in standard types from jar
11:37:05.402 [Apex Prelude Service STARTING] INFO  a.j.l.i.s.ApexPreludeManagedService - Done collecting built-in standard types from jar
11:37:05.402 [Apex Prelude Service STARTING] INFO  a.j.l.i.s.ApexPreludeManagedService - Scanning Apex Standard Library.
11:37:05.494 [Apex Prelude Service STARTING] INFO  a.j.l.i.s.ApexPreludeManagedService - Done scanning Apex Standard Library.
11:37:05.495 [Apex Prelude Service STARTING] INFO  a.j.l.i.s.ApexPreludeManagedService - Starting ApexIndexer.
11:37:05.495 [ApexIndexer STARTING] INFO  a.jorje.lsp.impl.index.ApexIndexer - ApexIndexer: Start indexer.
11:37:05.497 [ApexIndexer STARTING] INFO  a.jorje.lsp.impl.index.ApexIndexer - ApexIndexer: Scanning user-defined types.
11:37:05.498 [ApexIndexer STARTING] INFO  a.jorje.lsp.impl.index.ApexIndexer - ApexIndexer: Scanning workspace source.
11:37:05.574 [ApexIndexer STARTING] INFO  a.jorje.lsp.impl.index.ApexIndexer - ApexIndexer: Loading existing resource files from apex.db.
11:37:05.648 [ApexIndexer STARTING] INFO  a.jorje.lsp.impl.index.ApexIndexer - ApexIndexer: Building change list.
11:37:05.769 [ApexIndexer STARTING] INFO  a.jorje.lsp.impl.index.ApexIndexer - ApexIndexer: Applying found changes
11:37:05.769 [ApexIndexer STARTING] INFO  a.jorje.lsp.impl.index.ApexIndexer - ApexIndexer: Updating 1132 Apex artifacts in Apex DB
11:37:05.774 [ApexIndexer STARTING] INFO  a.jorje.lsp.impl.index.ApexIndexer - ApexIndexer: Error during startup
org.eclipse.jdt.internal.core.nd.db.IndexException: Corrupted database: apex.db
	at org.eclipse.jdt.internal.core.nd.db.Database.databaseCorruptionDetected(Database.java:435)
	at org.eclipse.jdt.internal.core.nd.db.Database.getChunk(Database.java:410)
	at org.eclipse.jdt.internal.core.nd.db.Database.getInt(Database.java:1108)
	at org.eclipse.jdt.internal.core.nd.db.Database.getString(Database.java:1226)
	at org.eclipse.jdt.internal.core.nd.field.FieldSearchKey.get(FieldSearchKey.java:76)
	at org.eclipse.jdt.internal.core.nd.field.FieldSearchIndex$2.compare(FieldSearchIndex.java:170)
	at org.eclipse.jdt.internal.core.nd.db.BTree.deleteImp(BTree.java:421)
	at org.eclipse.jdt.internal.core.nd.db.BTree.deleteImp(BTree.java:437)
	at org.eclipse.jdt.internal.core.nd.db.BTree.delete(BTree.java:299)
	at org.eclipse.jdt.internal.core.nd.field.FieldSearchKey.cleanup(FieldSearchKey.java:89)
	at org.eclipse.jdt.internal.core.nd.field.FieldSearchKey.destruct(FieldSearchKey.java:81)
	at org.eclipse.jdt.internal.core.nd.field.StructDef.destructFields(StructDef.java:389)
	at org.eclipse.jdt.internal.core.nd.field.StructDef$1.destructFields(StructDef.java:139)
	at org.eclipse.jdt.internal.core.nd.field.StructDef$1.destruct(StructDef.java:134)
	at org.eclipse.jdt.internal.core.nd.Nd.deleteIfUnreferenced(Nd.java:636)
	at org.eclipse.jdt.internal.core.nd.Nd.processDeletions(Nd.java:164)
	at org.eclipse.jdt.internal.core.nd.Nd.releaseWriteLock(Nd.java:357)
	at org.eclipse.jdt.internal.core.nd.Nd.releaseWriteLock(Nd.java:342)
	at apex.jorje.lsp.impl.index.ApexIndexer.deleteTypeInfo(ApexIndexer.java:1024)
	at apex.jorje.lsp.impl.index.ApexIndexer.buildCache(ApexIndexer.java:246)
	at apex.jorje.lsp.impl.index.ApexIndexer.lambda$startUp$1(ApexIndexer.java:803)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)
	at apex.jorje.lsp.impl.index.ApexIndexer.startUp(ApexIndexer.java:801)
	at com.google.common.util.concurrent.AbstractScheduledService$ServiceDelegate$2.run(AbstractScheduledService.java:230)
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:119)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

I get these notifications as well.

Salesforce Extension Version in VS Code: 59.8.0

SFDX CLI Version: @salesforce/cli/2.20.6 darwin-arm64 node-v20.5.0

OS and version: Sonoma 14.0 (23A344)

VS Code version:

Version: 1.85.0 (Universal) Commit: af28b32d7e553898b2a91af498b1fb666fdebe0c Date: 2023-12-06T18:18:04.614Z (1 day ago) Electron: 25.9.7 ElectronBuildId: 25551756 Chromium: 114.0.5735.289 Node.js: 18.15.0 V8: 11.4.183.29-electron.0

[peternhale]

@brandbarr Try these steps:

• exit vscode
• In a terminal navigate to the project's root dir
• run rm -fr .sfdx/tools/246
• start vsocde

[brandbarr]

@peternhale Thanks for the suggestion. This worked to clear the error but it keeps happening and still getting orphaned apex language server processes because of it.

Screenshot from this morning.

[peternhale]

@brandbarr sorry this keeps happening. Are you terminating the processes using the UI in vscode when the orphans are detected?

[brandbarr]

@brandbarr sorry this keeps happening. Are you terminating the processes using the UI in vscode when the orphans are detected?

Yep. Had 6 orphans this morning when I started vscode but the database was not corrupted which was new. Not sure the corruption is tied to the orphans or if there are two distinct issues here. I killed the 6 through the vscode interface and restarted vscode and it seems to be working OK at the moment.

[peternhale]

@brandbarr One of the issues we thought we solved is making sure the language server has an opportunity to shutdown gracefully when either vscode is exited normally or when the workspace is closed.

Simple experiment, if you don't mind, shutdown vscode and restart. Do you get the message regarding orphaned processes?

[rob-baillie-ortoo]

Is there a way that this could be detected and then the rebuild launched automatically (at least once)?

Obviously it's not a lot of work to perform the steps listed, but we have devs who won't even do that and then revert to Dev Console for running tests because 'VSCode is broken again'

[peternhale]

@rob-baillie-ortoo The apex extension does try to restart the language server if it fails unexpectedly. The current restart logic tries five times within three minutes. Only after the five restart attempts does the extension stop trying to restart.

If your cohorts are having difficulty with the language server starting properly, please file an issue so we can help try to resolve the issue.

[rob-baillie-ortoo]

I myself have definitely had instances where I had to run the above deletion to remove a corrected database - are you saying that instances of that should be reported here?

Sorry, based on this ticket, I thought it was be more a case of "Yeah, we know this sometimes happens, delete the database and it'll rebuild it".

In my experience, doing this resolves it - at least for a while (few weeks).

But if you say I should, I will try to report it in the future - before deleting it...

[peternhale]

@rob-baillie-ortoo For the symptom "org.eclipse.jdt.internal.core.nd.db.IndexException: Corrupted database: apex.db", following the suggestion enumerated above the is recommended solution to recover from the error. Having the apex.db in a corrupt state is definitely a productivity killer and if it is happening consistently for you and other devs in organization, we would like to know the circumstances that led up to the corruption.

When this happens again, I would appreciate you filing a new issue, not just to let us know that it happened again, but with details around what just changed in the workspace; did you switch branches, did you abruptly terminate vscode, or reload the workspace. Details like this might make the difference between "delete the db and restart" or discovering the reason the db is corrupted.

Personally, I would like to hear what you think we should do to remediate the error. Such as remove the apex.db and its associated artifacts and restart the language server, if the error is "corrupt db", or provide you with a new command or commands that place that decision into your hands, etc.

[rob-baillie-ortoo]

If I see it again, I'll be sure to raise a ticket.

Since the issue appears to be recoverable (delete the DB, rebuild), when the issue occurs I'd like the plug-in to:

• Delete the DB
• Rebuild it
• If the rebuild results in a non-corrupt state, dialog info to say a rebuild was required and if it happens frequently, to report it.
• If the rebuild fails and is in a corrupt state, dialog error to state the database is corrupt and not recoverable, report it.

Or, something along those lines.

Basically, since many of my co-workers have never worked outside Salesforce and therefore have no expectations of the IDE, when the capabilities disappear, they often assume that it's just not recoverable, there's nothing they can do. They then revert to really time-consuming work arounds. Whilst I am trying to educate them on not having such low expectations, it would be good if the toolset helped them along that path too.

(and completely understand that you want to know the reason why it's happening!)

[git2gus[bot]]

This issue has been linked to a new work item: W-15658026

[mingxuanzhangsfdx]

Hi @rob-baillie-ortoo , we are just able to accidentally reproduce the error you described. We will try to fix it soon and keep you posted.