marquez icon indicating copy to clipboard operation
marquez copied to clipboard

WIP: Elasticsearch indexing and queries

Open phixMe opened this issue 1 year ago • 1 comments

Very much a WIP

Problem

Our search right now does not enable nested queries on OpenLineage facets, code, linked entities, and ids. We want to enable our search to be the very best place to absorb and catalog OpenLineage based data.

Checklist

  • [ ] You've signed-off your work
  • [ ] Your changes are accompanied by tests (if relevant)
  • [ ] Your change contains a small diff and is self-contained
  • [ ] You've updated any relevant documentation (if relevant)
  • [ ] You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
  • [ ] You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
  • [ ] You've included a header in any source code files (if relevant)

phixMe avatar Jun 07 '24 21:06 phixMe

Deploy Preview for peppy-sprite-186812 canceled.

Name Link
Latest commit d0961387cb09577936d80aeabb1295d4b4a370e1
Latest deploy log https://app.netlify.com/sites/peppy-sprite-186812/deploys/66bfdfaf894d82000853712d

netlify[bot] avatar Jun 07 '24 21:06 netlify[bot]

Just a suggestion as I did not explore this too deeply. Wouldn't PSQL's text search be enough? I've seen some comparisons, e.g. here and it seems to be pretty viable option.

JDarDagran avatar Jul 31 '24 13:07 JDarDagran

Codecov Report

Attention: Patch coverage is 22.36025% with 125 lines in your changes missing coverage. Please review.

Project coverage is 83.28%. Comparing base (422fd43) to head (d096138). Report is 1 commits behind head on main.

Files Patch % Lines
...i/src/main/java/marquez/service/SearchService.java 8.40% 106 Missing and 3 partials :warning:
...c/main/java/marquez/api/v2beta/SearchResource.java 21.05% 15 Missing :warning:
...src/main/java/marquez/api/OpenLineageResource.java 50.00% 0 Missing and 1 partial :warning:
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2834      +/-   ##
============================================
- Coverage     84.77%   83.28%   -1.49%     
- Complexity     1470     1477       +7     
============================================
  Files           256      259       +3     
  Lines          6626     6785     +159     
  Branches        308      313       +5     
============================================
+ Hits           5617     5651      +34     
- Misses          856      977     +121     
- Partials        153      157       +4     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Aug 09 '24 23:08 codecov[bot]