database-stream-processor
database-stream-processor copied to clipboard
API hardening TODOs
Having attempted to interact with the REST API from the commandline (curl + websocat), here are some rough issues I've ran into that I'll try to coordinate with @gz to fix:
- Currently, there is no way to ergonomically query objects in the API using information supplied by the user. It often requires users to create objects, then do list queries (read more state than needed), parse the results, and then do follow up queries. For example, when trying to use the HTTP input transport, the endpoint at the pipeline uses a generated name (connector-
). To find that endpoint from a commandline, I need to GET /configs
, use jq to search for the uuid of the entry that hasconfig="<stream name>"
, use an out of band process to find the pipeline port, and then use the retrieved UUID and port to reach<pipeline url>:<port>/input_endpoint/connector-<uuid>
. - The above problem comes down to revisiting object naming and identifiers, uniqueness requirements, search in the API, with the server side generating identifiers. This also affects idempotency expectations (e.g., when rerunning
python/test.py
, we create projects, pipelines etc with the same shape every time). - The server needs to generate UUIDs instead of an incrementing counter.
- Component and schema evolution (e.g., adding new transport config entries requires rebuilding the python client, and adding new types of connectors requires an enum type to be available at the client).
- API versioning.
Some other things that came up:
- Need to expose status [stopped, starting, running...] of a pipeline through the API (in PipelineDesr)
- Drop pipeline_id/metadata but return just PipelineDescr+metadata
- Should be able to retrieve a config of the running pipeline
- Should return error codes (number) when throwing an error
- project: rename to program
- pipeline: rename status to stats
- uuid and name are unique for everything, have
/id to retrieve and /?name= for everything - drop ConnectorType enum (KafkaIn, KafkaOut)
- pipeline stop/start/shutdown replace with POST action to <pipeline/id>