flink-http-connector
flink-http-connector copied to clipboard
getindata
Http Connector for Apache Flink. Provides sources and sinks for Datastream , Table and SQL APIs.
Currently the code has logic if GET then use query params otherwise use body content to resolve the parameters. We have recently included 2 prs that allow GETs with body...
It would be great for us to be able to use [https://ibm.github.io/java-sdk-core/docs/9.17.3/com/ibm/cloud/sdk/core/security/IamAuthenticator.html](https://ibm.github.io/java-sdk-core/docs/9.17.3/com/ibm/cloud/sdk/core/security/IamAuthenticator.html) which handles token refresh for the IBM cloud. This authenticator is Apache 2 licensed, so could go into...
#### Description Introduce caching similar to the Flink JDBC connector, so long running synchronous Flink jobs can cache content resulting in less API calls. Resolves https://github.com/getindata/flink-http-connector/issues/63 ##### PR Checklist -...
#### Description Lookup queries are now retried in case of IOException up to `gid.connector.http.source.lookup.max-retries` with a delay of `gid.connector.http.source.lookup.request.retry-timeout-ms` between retries. The default values are 3 retries and 1 second...
Currently http lookup source will forward every lookup query to external system. This can be easily a bottle neck for processing pipeline. A remedy for this would be to implement...
In Flink 1.16 a `[Rate limiting API](https://flink.apache.org/2022/11/25/optimising-the-throughput-of-async-sinks-using-a-custom-ratelimitingstrategy/)` was added to Flink Async Writer. This can help to slow down rate of Http calls in case when HTTP endpoint is overwhelmed...
Hello everyone, Thank you very much for this connector. I have started some tests with it, and it works really well. For a project, I need to extend the capabilities...
README and examples still points to deprecated method 'setElementConverter(ElementConverter)' Update README and Examples to include 'setElementConverter(SchemaLifecycleAwareElementConverter)' instead.
Could you provide an example of Http Periodically Scan Source (not lookup source)? Does it support renew access token after expiration?
I'm very glad to have found this project. We are building a Flink-SQL platform and want to use this connector in a production environment. We have about 7 million data...
