Improve Search Performance

[JustinGrote]

https://github.com/shilangyu/scoop-search has changed my experience with scoop, the slow search has always been one of my pain points as a user.

I'd be willing to PR improvement to the search engine to incorporate the lessons learnd here if there is interest.

[Bios-Marcel]

It appears that the problem is that we parse the whole json. Maybe we can try to switch this to an approach where we only parse until we find the bin section?

[rashil2000]

I'd be willing to PR improvement to the search engine to incorporate the lessons learnd here if there is interest.

Yes! Please make a PR when you can.

[bartvanandel]

I was annoyed by this myself, so I've just put some effort in writing a fix for this. Will publish a PR soon.

[burgr033]

I managed to reduce search time by almost 99% Because I wanted to keep the core search intact (and I don't understand a bit what is going on under the hood), I built myself an additional command, that builds a cache file containing names, binaries, buckets and versions of all apps in all local buckets, and added a parameter to scoop search (--offline) that just searches this index file.

╭─sb@tealc ~
╰─$ time scoop search lazygit
Results from local buckets...

Name    Version Source Binaries
----    ------- ------ --------
lazygit 0.38.2  extras
00:00:18.2134557

╭─sb@tealc ~
╰─$ time scoop search lazygit --offline
Results from cached list...

Name    Version Source Binaries
----    ------- ------ --------
lazygit 0.38.2  extras lazygit.exe
00:00:00.1915792

For my use case this really is enough. I do not know if this is a valid approach for any of you, i just wanted to share the idea.

Cheers

[Bios-Marcel]

Awesome idea @cigh033

Sounds like something worth adding, but not invoke by default.

[Yakiyo]

@cigh033 generating a cache file would indeed make subsequent searches faster, but the initial invokation would need the same, if not more amount of time as the original search command right? and wouldn't the cache file need to be updated after certain intervals too?

[rashil2000]

I narrowed down the regression in search times to the cmdlet manifest_path in manifest.ps1. With the current version

function manifest_path($app, $bucket) {
    (Get-ChildItem (Find-BucketDirectory $bucket) -Filter "$(sanitary_path $app).json" -Recurse).FullName
}

I get the 20 second search times but with the previous

function manifest_path($app, $bucket) {
    fullpath "$(Find-BucketDirectory $bucket)\$(sanitary_path $app).json"
}

It goes down to 8 seconds.

It seems like the new version is searching for a file in all subdirectories. Is this really necessary? Don't all manifests have to be always located in bucket/?

Originally posted by @mathijshenquet in https://github.com/ScoopInstaller/Scoop/issues/5615#issuecomment-1694653931

Don't all manifests have to be always located in bucket/?

Historically, manifests were also present in the root directory. However most buckets now follow the convention of having a bucket/ subdirectory, in order to keep the repo cleaner.

[xialvjun]

Just an advice: maybe we can use sqlite or moreover we can use a git like sql databse like https://github.com/dolthub/dolt