meltano Let buffer size be configured on a per-extractor basis using a `buffer

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/2512

Originally created by @DouweM on 2021-01-12 23:05:56

Follow-up to https://gitlab.com/meltano/meltano/-/issues/2392.

Besides an elt.buffer_size setting, we could add a buffer_size extractor extra so that the value can be overridden on a per-extractor basis, since it's directly tied to the maximum message length, which is a property of an individual extractor.

This would also let us set a higher buffer_size for a specific discoverable extractor if we know the global default of 10MiB doesn't cut it.

When both the setting and extra are set, we could pick the maximum value.

Jan 12 '21 23:01 MeltyBot

View 1 previous comment from the original issue on GitLab

May 30 '22 06:05 MeltyBot

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

Apr 27 '23 17:04 stale[bot]

Covered by SDK features

May 10 '23 20:05 tayloramurphy

I think this unfortunately isn't (and probably can't be?) covered by SDK features, since it's more of an orchestrator/shell feature that fits Meltano.

Slack: https://meltano.slack.com/archives/C069CQNHDNF/p1723494267918129
Linen: https://www.linen.dev/s/meltano/t/22914628/waves-i-m-running-into-this-error-with-a-tap-github-extracto#b07c1f68-71d1-4175-a9e6-bc9f07a1c2ee

Aug 12 '24 20:08 edgarrmondragon

FWIW, the suggested implementation as an extractor extra makes sense to me and is not too big an effort to implement.

Aug 12 '24 20:08 edgarrmondragon

Let buffer size be configured on a per-extractor basis using a `buffer_size` extra