dagger icon indicating copy to clipboard operation
dagger copied to clipboard

doc: improve quickstart documentation and related setup

Open Meghajit opened this issue 2 years ago • 0 comments

Summary As of now, there is some issues in the local setup of Dagger:

  1. The local.properties assumes there is a TestMessage proto present. However, it is not the case. This causes the dagger not to start properly.
  2. The other test protos which come inbuilt with Dagger are in test module which are not usable in local.properties.
  3. The watermark delay is set to 0 in the local.properties which might cause issues in some cases, such as when events are coming in late due to issues with kafka. Also, this watermark delay needs to be much higher when Parquet daggers are being setup.
  4. With multiple ways people can create daggers like with parquet source, kafka source, with preprocessors, etc, ideally there should be specific examples on how to do each

Proposed solution

These are some suggestions based on the feedback received from internal discussions as well as from users:

  1. Creating a Dockerfile/Docker compose with all dependencies bundled( local kafka cluster, sample protos, etc). User just need to run it and check the logs for output. No manual setup required except Docker.

  2. Organizing the documentation explaining how to do different things with Dagger in a single place. For example,

  3. Create a Dagger with Kafka as a source

  4. Create a Dagger with Parquet Files as a source

  5. Create a Dagger using preprocessors

Currently, the reference is not streamlined and is organised in multiple sections.

What it will achieve

Following benefits are expected:

  1. Faster onboarding of users: People can utilise the examples to try out different features of daggers without help by the committers/maintainers
  2. Wider adoption: Faster ways to try out different features by minimal setup means people can quickly check whether the product satisfies their needs or not. Will help in a faster feedback loop.

Additional context None

Meghajit avatar Sep 15 '22 05:09 Meghajit