Promptify
Promptify copied to clipboard
promptslab
pipe.fit() freezes and don't return anything
Hi many times while running this model in pycharm the model just freezes and don;t return the output.
It works sometimes after restarting the kernel twice or thrice.
Sometimes doesn't work at all.
Please look into this
On further investigation it seems intiliazing the model pipeline can happen only once per kernel as it opens the llm_response session.
If we want to run this api again with different key ,different config(domain and labels), we need to restart the kernel.
Yes, because changing key and domain etc would initiate the prompter, model etc again. for a single model you can initialize the things once and then call the pipe.fit() multiple times on any number of samples.
Hi many times while running this model in pycharm the model just freezes and don;t return the output. It works sometimes after restarting the kernel twice or thrice. Sometimes doesn't work at all. Please look into this
Can you provide more details on this? your sentence, template etc a minimal code to reproduce the error
Was facing the same issue, setting structured_output=False and parsing the output myself seems to bypass the issue.
Hi many times while running this model in pycharm the model just freezes and don;t return the output. It works sometimes after restarting the kernel twice or thrice. Sometimes doesn't work at all. Please look into this
Can you provide more details on this? your sentence, template etc a minimal code to reproduce the error
Facing same issue. Adding structured_output=False in pipe.fit() as @kansalaman said, unfreezes the frozen state but it fails with :
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/uvicorn/middleware/message_logger.py", line 84, in __call__
raise exc from None
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/uvicorn/middleware/message_logger.py", line 80, in __call__
await self.app(scope, inner_receive, inner_send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/routing.py", line 296, in app
content = await serialize_response(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/routing.py", line 180, in serialize_response
return jsonable_encoder(response_content)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/encoders.py", line 301, in jsonable_encoder
jsonable_encoder(
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/encoders.py", line 287, in jsonable_encoder
encoded_value = jsonable_encoder(
^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/encoders.py", line 287, in jsonable_encoder
encoded_value = jsonable_encoder(
^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/encoders.py", line 331, in jsonable_encoder
return jsonable_encoder(
^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/encoders.py", line 331, in jsonable_encoder
return jsonable_encoder(
^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/encoders.py", line 287, in jsonable_encoder
encoded_value = jsonable_encoder(
^^^^^^^^^^^^^^^^^
File "/home/username/.virtualenvs/promptify/lib/python3.11/site-packages/fastapi/encoders.py", line 330, in jsonable_encoder
raise ValueError(errors) from e
ValueError: [TypeError("'builtin_function_or_method' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]
TRACE: 127.0.0.1:35808 - HTTP connection lost
In my case I'm trying to run the pipeline via Fastapi. But it freezes as reported above. Same thing happens when trying with curl.
Upon further inspection i found that it stalls here:
# promptify/pipelines/__init__.py
[...]
def _get_output_from_cache_or_model(self, template):
output = None
if self.cache_prompt:
output = self.prompt_cache.get(template)
if output is None:
try:
response = self.model.execute_with_retry(prompt=template)
except Exception as e:
print(f"Error in model execution: {e}")
return None
if self.structured_output:
output = self.model.model_output(
response, json_depth_limit=self.json_depth_limit
)
[...]
Especifically in the following part:
output = self.model.model_output(
response, json_depth_limit=self.json_depth_limit
)
It just stalls there.
Found the culprit in promptify/parser/parser.py.
This function goes into a infinite loop in the try/Except block. I only noticed it because i printed the Exception (original code doesn't print the exception):
def get_possible_completions(
self, json_str: str, json_depth_limit: int = 5
) -> Union[Dict[str, Any], List[Any]]:
"""
Returns a list of possible completions for a JSON object string.
Parameters
----------
json_str : str
The JSON object string
json_depth_limit : int, optional
The maximum length of the completion strings to try (default is 5)
Returns
-------
Union[Dict[str, Any], List[Any]]
If the completion strings are objects, returns a dictionary with 'completion' and 'suggestions' keys.
If the completion strings are arrays, returns a list of suggested completions.
"""
candidate_marks = ["}", "]"]
if "[" not in json_str:
candidate_marks.remove("]")
if "{" not in json_str:
candidate_marks.remove("}")
# specify the mark should end with
should_end_mark = "]" if json_str.strip()[0] == "[" else "}"
completions = []
for completion_str in self.get_combinations(
candidate_marks, json_depth_limit, should_end_mark=should_end_mark
):
try:
completed_obj = self.complete_json_object(json_str, completion_str)
completions.append(completed_obj)
except Exception as e:
print(e)
pass
return self.find_max_length(completions)
Results in:
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Couldn't fix JSON
Funny thing there are no problems when one runs it in python. Problem happens as soon as you try doing it via CURL, FastAPI or Flask. It just enter this infinite loop.
Funny thing there are no problems when one runs it in python. Problem happens as soon as you try doing it via CURL, FastAPI or Flask. It just enter this infinite loop.
It just started to fail when run from python shell too. The parser is completely unrelieable when it comes to parsing the json. I'd recommend to update the code to use Pydantic with Instructor.
@roperi Hi, did you solve this problem?
I have neither the structured_output=False attribute nor the global Pipeline pipeline working
Upd. it stucks bot in python use and fastapi
Upd2: Fixed. structured_output it's an Pipeline attribute :)
@roperi Hi, did you solve this problem?
I have neither the structured_output=False attribute nor the global Pipeline pipeline working
Upd. it stucks bot in python use and fastapi
Upd2: Fixed. structured_output it's an Pipeline attribute :)
@Fan4ik20 Unfortunately I didn't. The parser kept failing at parsing the json. I decided to not use Promptify and move on.
@roperi Yes, same for me, so i parsed received JSON by myself. BUT, for some reason we received JSON with extra brackets, so i had to remove them