lambda-promtail: Add support for Kinesis data stream events

[juissi-t]

What this PR does / why we need it:

This PR adds support for sending Kinesis data stream events to lambda-promtail. One use case would be e.g. to send CloudFront realtime logs to Loki.

Which issue(s) this PR fixes:

Fixes #5978

Special notes for your reviewer:

n/a

Checklist

• [x] Documentation added
• [ ] Tests updated
• [x] Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
• [ ] Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[juissi-t]

@grafana/loki-team: could someone review this PR, please?

[juissi-t]

@KMiller-Grafana this PR has been open for about three weeks now. Any chance you could take a look at it?

[slim-bean]

Apologies @juissi-t, it's been busy trying to finish up projects at the end of our quarter and this week most of the team is at an office.

Three weeks is a long time :( very sorry about that, we will take a look at this as soon as we can, but may not be until t week.

In the meantime if you could fix the changelog conflict that will help us get this merged as soon as we can

[juissi-t]

Thanks, it would be great if you could review this in the near future!

In the meantime if you could fix the changelog conflict that will help us get this merged as soon as we can

Rebased the PR.

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[juissi-t]

@slim-bean any chance you could take a look at this, now that it's been over a month since the last time I reached out. :pray:

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[juissi-t]

@cstyan I rebased the PR. Some test seems to fail, but to me it looks unrelated to this change. Could you check this PR?

[luva03]

@juissi-t @slim-bean @cstyan

Is there any progress on this PR? We are very interested in using this feature. 🙏 TY

[juissi-t]

Is there any progress on this PR? We are very interested in using this feature. pray TY

I have been trying to ping the people who have commented or assigned this, but no luck yet. I'll keep trying.

@slim-bean @cstyan could someone take a look? This has been open for over three months now.

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[juissi-t]

Pinging recent contributors to lambda-promtail: @dannykopping @jonathanmorais @simonswine. Could one of you maybe take a look at this pull request? Or find someone who could do that?

This PR has been open for over four months now, and I find it quite disheartening for any new contributor to an open source project backed by a company to have to wait so long for a review for a fairly trivial new feature. I know Loki team has its priorities and for sure you are busy, but still... If you don't want to have this, that's fine too, but could someone at least say something?

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[juissi-t]

Apologies again for the delays here, I hope it doesn't deter you from contributing again

Thanks a lot for your help in getting this reviewed @dannykopping! I will keep in mind the option of asking in Slack if I find another scratch I need to itch, and the PR review seems to take too long. :smile: