database-stream-processor icon indicating copy to clipboard operation
database-stream-processor copied to clipboard

API hardening TODOs

Open lalithsuresh opened this issue 1 year ago • 1 comments

Having attempted to interact with the REST API from the commandline (curl + websocat), here are some rough issues I've ran into that I'll try to coordinate with @gz to fix:

  • Currently, there is no way to ergonomically query objects in the API using information supplied by the user. It often requires users to create objects, then do list queries (read more state than needed), parse the results, and then do follow up queries. For example, when trying to use the HTTP input transport, the endpoint at the pipeline uses a generated name (connector-). To find that endpoint from a commandline, I need toGET /configs, use jq to search for the uuid of the entry that has config="<stream name>", use an out of band process to find the pipeline port, and then use the retrieved UUID and port to reach <pipeline url>:<port>/input_endpoint/connector-<uuid>.
  • The above problem comes down to revisiting object naming and identifiers, uniqueness requirements, search in the API, with the server side generating identifiers. This also affects idempotency expectations (e.g., when rerunning python/test.py, we create projects, pipelines etc with the same shape every time).
  • The server needs to generate UUIDs instead of an incrementing counter.
  • Component and schema evolution (e.g., adding new transport config entries requires rebuilding the python client, and adding new types of connectors requires an enum type to be available at the client).
  • API versioning.

lalithsuresh avatar Apr 09 '23 21:04 lalithsuresh