Running multiple copies of PyAirbyte for the same stream will lead to errors

[aaronsteers]

trafficstars

PyAirbyte should be able to be run multiple times simultaneously except for these conditions:

1. Loading the same stream at the same time with two PyAirbyte instances is not supported. We do not block against this happening, but it will lead to race conditions, which could cause failures and/or data corruption.
2. DuckDB and/or MotherDuck may have their own limitations around concurrent writes.

[Udit107710]

If we created a new cache for all the concurrent runs, then this should not be blocked, right? @aaronsteers Do you think there can be other issues also?

[aaronsteers]

@Udit107710 - To your question above, yes, I think this could be a valid workaround. I haven't personally reproed this issue; I've only heard from a user that this was an issue.

If anyone either has a workaround or has reproed the issue, this would be helpful and much appreciated data, either way. 🙏