perf: optimize listing performance

[knowald]

First, thank you for creating and maintaining this great project! It has been fantastic for us integrating Passbolt into our workflows.

The problem!

When working with several hundred credentials, passbolt list resource became unusable due to request timeouts.

The command would consistently fail with: Error: Get Resource Getting Resource Secret: Doing Request: Request Context: context deadline exceeded

Root Cause

The resource listing implementation suffered from a N+1 query problem:

1. Initial request: Fetch list of all resources (1 API call)
2. Per-resource requests: Call helper.GetResource() individually for each resource to decrypt fields (N API calls)

With 300+ credentials, this resulted in 300+ sequential API calls, exceeding the default context timeout.

The code had a TODO comment acknowledging this performance issue but it had not been addressed yet.

Solution

This PR eliminates the N+1 problem using Passbolt API's existing contain[secret] parameter to fetch all data in bulk, then decrypt locally:

1. Bulk Secret Fetching

• Added ContainSecret: true to GetResourcesOptions when encrypted fields are needed
• Reduces 300+ sequential API calls to 1 bulk request

2. Decryption-need detection

• Only fetches secrets when encrypted columns (Name, Username, URI, Password, Description) are requested via --column flag or CEL filters
• Keeps metadata-only queries lightweight

3. Local Decryption (resource/decrypt.go)

• New centralized decryptResource() helper using helper.GetResourceFromData()
• Decrypts secrets client-side without additional API calls
• Eliminates code duplication across list and filter operations

4. ResourceType Caching

• Caches ResourceType objects to avoid duplicate fetches (typically 3-6 unique types per instance)

5. Additional Improvements

• JSON output now respects --column flag (previously ignored)
• Fixed CEL variable name: "Id" → "ID" for correct ID-based filtering

Backward Compatibility

Tested manually to be compatible with Passbolt v3, v4, and v5

Testing

# Build
go build -o passbolt

# Test with large credential set
./passbolt list resource

# Test column filtering (only fetches what's needed)
./passbolt list resource -c ID,FolderParentID,CreatedTimestamp  # No decryption
./passbolt list resource -c ID,Name,Username                     # With decryption

# Test CEL filters
./passbolt list resource --filter 'Name.startsWith("prod")'

# Test JSON output with column selection
./passbolt list resource --json -c ID,Name,Username

This should make the CLI practical for teams / organizations with loads of credentials. It's especially important for Passbolt v5 where Name, Username, URI, and Description fields are now encrypted in the metadata rather than being available as plaintext.

Looking forward to your feedback!

[CLAassistant]

All committers have signed the CLA.

[speatzle]

Hi, Thank you. go-passbolt and go-passbolt-cli where mostly written in my spare time, i am glad it has helped you.

For the Timeout issue, there is the workaround of specifying the timeout flag --timeout with a bigger value (default is 1 minute).

With the Changes in v5 lots of work needed to be done, i unfortunately haven't been able to put in the time go-passbolt requires, i managed to do the minimum to get it working with v5.

Currently there is no caching for anything and due to the "easy" way i chose for Decrypting Metadata by tying it in the helpers to Decrypting the Secret mainly because the description field can be in the Metadata or in the Secret (called Note now in the UI).

One of the Problems with using ContainSecret is that it massively increases the request size. I have had issues multiple times on "big" Passbolt instances where PHP runs out of memory, the Nginx body size limit is hit or the request timeouts out on Nginx/PHP Another issue with ContainSecret is that all requested resources are marked as "Accessed" by Passbolt which we would not want if we are filtering with CEL by Name or other Metadata. Perhaps we could do a batch request of Secrets by id after filtering?

AFAIK Re-fetching Metadatakeys and ResourceTypes for every Resource are the biggest Problem for the network currently. Another issue is not using Session key decryption for Metadata. The biggest Problem locally is also forcing decryption of the Secret for metadata access.

I Believe generic caching should be done in go-passbolt and not in go-passbolt-cli so it can benefit all go-passbolt consumers in a streamlined way.

On go-passbolt there are the following issues about this:

• https://github.com/passbolt/go-passbolt/issues/23
• https://github.com/passbolt/go-passbolt/issues/30
• https://github.com/passbolt/go-passbolt/issues/64
• https://github.com/passbolt/go-passbolt/issues/65

Maybe you could give some input or even make some changes on https://github.com/passbolt/go-passbolt/issues/64 and https://github.com/passbolt/go-passbolt/issues/65 specifically?

I am open to suggestions.

[knowald]

Hi! Looking at it this new way you're absolutely right about the implementation layer - I did indeed put this together at the CLI level on my fork as a quick fix for this one use case, because it was the only v5 issue I've personally encountered. However, I understand that this isn't the proper approach.

I'll investigate the go-passbolt package and do my best to get back to you with some input on the upstream layer.

Feel free to close this merge request if the band-aid doesn't qualify as a TODO™ flagged improvement for now.

Thank you again for maintaining these projects and taking the time for the feedback!