garnet icon indicating copy to clipboard operation
garnet copied to clipboard

Support for streams?

Open rhaley-starfish opened this issue 5 months ago • 2 comments

Feature request type

enhancement

Is your feature request related to a problem? Please describe

Hi, I reused the same description from the closed item. That git issue was closed about a year and a half ago. Is there any progress or update on streams support? We are evaluating Garnet as the core of a product but streams are a pretty important feature for us. If you could give us some sort of idea about when and if you are implementing it, we could use that in our evaluation. We really like Garnet so far, so fingers crossed!

Describe the solution you'd like

A road map or ETA on Streams as a feature. Or even better, a release announcement! :-)

Describe alternatives you've considered

We will have to go with Redis.

Additional context

Our testing with Garnet has gone really well. Would really like to see stream support added.

rhaley-starfish avatar Sep 09 '25 23:09 rhaley-starfish

See also #1131 PR.

prvyk avatar Sep 17 '25 00:09 prvyk

The linked PR https://github.com/microsoft/garnet/pull/1131 was our summer intern's prototype implementation of the basic stream commands. We do want to revisit this, fill gaps, test, and get it ready for merging to main, but this has been of lower priority due to more pressing work items. If anyone in open source wants to contribute to it, that would be welcome as well.

badrishc avatar Sep 18 '25 18:09 badrishc

I thought it has it already. That's why I am getting this error when using it for background queues using rq in python.

13:27:11 Subscribing to channel rq:pubsub:575dfc5d0ff141cabf7198fc3d40834a
13:27:11 *** Listening on default...
13:27:11 Cleaning registries for queue: default
StartedJobRegistry cleanup: 964a8908-475f-4b5f-ab87-35121dcad227 Moved to FailedJobRegistry, due to AbandonedJobError, at 2025-11-29 07:57:11.775145+00:00
13:27:11 Worker rq:worker:575dfc5d0ff141cabf7198fc3d40834a: found an unhandled exception, quitting...
Traceback (most recent call last):
  File "D:\whook2\.venv\Lib\site-packages\rq\worker.py", line 609, in work
    self.run_maintenance_tasks()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "D:\whook2\.venv\Lib\site-packages\rq\worker.py", line 989, in run_maintenance_tasks
    self.clean_registries()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "D:\whook2\.venv\Lib\site-packages\rq\worker.py", line 467, in clean_registries
    clean_registries(queue, self._exc_handlers)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\whook2\.venv\Lib\site-packages\rq\registry.py", line 619, in clean_registries
    ).cleanup(exception_handlers=exception_handlers)
      ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\whook2\.venv\Lib\site-packages\rq\registry.py", line 309, in cleanup
    pipeline.execute()
    ~~~~~~~~~~~~~~~~^^
  File "D:\whook2\.venv\Lib\site-packages\redis\client.py", line 1613, in execute
    return conn.retry.call_with_retry(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        lambda: execute(conn, stack, raise_on_error),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        lambda error: self._disconnect_raise_on_watching(conn, error),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "D:\whook2\.venv\Lib\site-packages\redis\retry.py", line 87, in call_with_retry
    return do()
  File "D:\whook2\.venv\Lib\site-packages\redis\client.py", line 1614, in <lambda>
    lambda: execute(conn, stack, raise_on_error),
            ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\whook2\.venv\Lib\site-packages\redis\client.py", line 1504, in _execute_transaction
    self.raise_first_error(commands, response)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "D:\whook2\.venv\Lib\site-packages\redis\client.py", line 1539, in raise_first_error
    raise r
  File "D:\whook2\.venv\Lib\site-packages\redis\client.py", line 1473, in _execute_transaction
    self.parse_response(connection, "_")
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "D:\whook2\.venv\Lib\site-packages\redis\client.py", line 1550, in parse_response
    result = Redis.parse_response(self, connection, command_name, **options)
  File "D:\whook2\.venv\Lib\site-packages\redis\client.py", line 653, in parse_response
    response = connection.read_response()
  File "D:\whook2\.venv\Lib\site-packages\redis\connection.py", line 666, in read_response
    raise response
redis.exceptions.ResponseError: Command # 7 (XADD rq:results:964a8908-475f-4b5f-ab87-35121dcad227 MAXLEN ~ 10 * type 2 exc_string...) of pipeline caused error: Command # 7 (XADD rq:results:964a8908-475f-4b5f-ab87-35121dcad227 MAXLEN ~ 10 * type 2 exc_string...) of pipeline caused error: unknown command
13:27:11 Unsubscribing from channel rq:pubsub:575dfc5d0ff141cabf7198fc3d40834a

ilovefreesw avatar Nov 29 '25 08:11 ilovefreesw