Catalog - Cannot search on fields with values over 200 characters in length

[ckuijjer]

📜 Issue Labels

• [x] Please familiarize yourself with the issue labels used in this project: LABELS.md

🔎 Search Terms

catalog search field missing values

🗃️ Project Area

Catalog

🔗 External Integration

N/A

📝 Description & Context

When full text searching for entities using api/catalog/entities/by-query and on the field metadata.description, we've noticed a few missing search results. The missing search results had in common that they all have a lot of text in their description.

Debugging this issue we found that for these missing entities the rows in the backstage_plugin_catalog.search with key metadata.description all had value NULL. Explaining why it was impossible to search through their description.

During the stitching process where the search table gets refreshed, a MAX_VALUE_LENGTH of 200 characters is used, and if a value is over 200 characters in length it's stored as a null, see the mapToRows function in buildEntitySearch.ts and its rationale.

👍 Expected Behavior

We would like to be able to query the catalog by searching through fields with large text values, but do understand the database performance implications of simply indexing all values

Would it be an idea to trim these values to 200 characters, instead of setting them to null?

We would love to submit a PR for this.

📦 Reproduction Repo

No response

🥾 Reproduction steps

No response

Have you read the Code of Conduct?

• [x] I have read the Code of Conduct

Are you willing to submit PR?

Yes, and I have enough information to get started

[vinzscam]

Using the search plugin is probably a better option than using the catalog for searching if you have such complex entities

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.