pdr-backend icon indicating copy to clipboard operation
pdr-backend copied to clipboard

Issue1002 - Create lake & etl readme

Open idiom-bytes opened this issue 1 year ago • 1 comments

Fixes #1002

Changes proposed in this PR:

  • [x] Communicated basics of lake, etl, and overall predictoor job into a readme.
  • [x] Communicated how these components are different from each other, and how they work together.
  • [x] Communicated checkpoint details about [GQLDF CSV + RAW] providing end-to-end requirements for how we keep this updated.
  • [ ] Communicated checkpoint details about [RAW => ETL Tables] providing end-to-end requirements for how we keep this updated.
  • [ ] Communicate a bit about CLI commands, and how they work.
  • [ ] An end-to-end example with lake update raw
  • [ ] An end-to-end example with lake update etl
  • [ ] Update vs. Drop vs. other commands
  • [ ] Create basic visuals for each section, to provide a mental model.

idiom-bytes avatar May 09 '24 18:05 idiom-bytes

Reviewed the README and have some feedback:

  • [ ] Add information about where the data comes from and what kind of data we are ingesting into the lake
  • [ ] There is no information about the data available, tables etc. I would expect to visually see the data that I'm going to get before going trough the lake setup process
  • [ ] The README is long and we don't have any legend on the top to make it easy to see the content and navigate trough different sections. Add a legend on the top of the page, example: Screenshot 2024-05-17 at 12 40 51

KatunaNorbert avatar May 17 '24 10:05 KatunaNorbert

I'm having to draw a bunch of diagrams... targeting to deliver this when @kdetry is further along w/ integration. Currently focusing on #1001 and advancing the ETL.

idiom-bytes avatar Jun 07 '24 23:06 idiom-bytes

README has been concluded inside PR #1001, closing PR and updating ticket

idiom-bytes avatar Jul 11 '24 17:07 idiom-bytes