genkit
genkit copied to clipboard
Published
3 weeks ago
•
firebase
firebase
[JS] Schema validation failed when using a tool
Describe the bug I have a flow with a prompt defined with dotpromt with a json schema as output. When not using any tool the flow works fine, but when I added a tool the flow is failling because the Schema validation fails when the model replies that needs to use a tool.
To Reproduce
- Create a prompt with JSON schema
- Create a tool
- Make a request that forces the model to use the tool.
Expected behavior GenKit handles the tool calling, calls the tool and calls again the model with the result.
Actual behavior The flow fails with:
Candidate[0] Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:
- (root): must be object
Provided data:
null
Required JSON schema:
...
Runtime (please complete the following information): MacOS
** Node version v20.15.0
Additional context
I tried to use the returnToolRequests: true, to handle the reply myself but the same issue happens since the check is on the model reply.
See trace logs: b198423822c2d17f302dbd4674c0b6cc.txt
Thanks for reporting. Can you re-upload the trace as plain text? I am unable to read it.
I saw that it had a weird format, but I just used the "export trace > save" option in the UI. Maybe another issue to report.
In any case, here is the trace.
I think the issue is in the generate.ts at the candidateError check: https://github.com/firebase/genkit/blob/2f6c7d8f3a9b7f0a9466d20598fa2cfb4363956d/js/ai/src/generate.ts#L668
if (c.text() === "" && c.data() === null) return null
c.data() is actually undefined. If I switch to:
if (c.text() === "" && !c.data()) return null
Then it works fine.
{
"traceId": "5569630ab920a351608eb119092ac5a1",
"spans": {
"e7cd70f774571360": {
"spanId": "e7cd70f774571360",
"traceId": "5569630ab920a351608eb119092ac5a1",
"parentSpanId": "bfefc0be23fe76a3",
"startTime": 1726056474355,
"endTime": 1726056474367.9668,
"attributes": {
"genkit:type": "dotprompt",
"genkit:name": "bot",
"genkit:input": "{\"input\":{\"uid\":\"ipSL4LxtW325ME9KHXg02fvgGlGv\",\"text\":\"create a task for fixing a bug in the landing\"},\"tools\":[null],\"returnToolRequests\":true}",
"genkit:path": "/{dev-run-action-wrapper}/{botFlow,t:action}/{botFlow,t:flow}/{bot,t:dotprompt}",
"genkit:metadata:prompt_fingerprint": "dd8287bae646c972996f69604a9c7f6877b4ddf72a79f85924726e65874ef817",
"genkit:output": "{\"model\":\"vertexai/gemini-1.5-pro\",\"config\":{\"temperature\":0.5},\"history\":[{\"role\":\"system\",\"content\":[{\"text\":\"\\nYou are a helpful AI task management assistant that takes the user input (text, voice or image) and\\nperforms actions on behalf of the user to organize and manage tasks in a project or answers questions\\nabout the projects tasks. Identify the type of message, perform the action and/or reply to it using the provided tools\\nIf possible provide 3 to 4 suggestion replies to the user input.\\n\\n\"}]}],\"prompt\":[{\"text\":\"\\nUseUser message: create a task for fixing a bug in the landing\\n\"}],\"candidates\":1,\"output\":{\"format\":\"json\",\"schema\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"},\"text\":{\"type\":\"string\",\"description\":\"The reply to the user message\"},\"transcript\":{\"type\":[\"string\",\"null\"],\"description\":\"The transcript of the user message, if the input was a voice message\"},\"suggestions\":{\"anyOf\":[{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"text\":{\"type\":\"string\",\"description\":\"The text of the suggestion to show to the user\"},\"type\":{\"type\":\"string\",\"enum\":[\"full\",\"partial\",\"action\"],\"description\":\"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"},\"action\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"}},\"required\":[\"text\",\"type\",\"action\"],\"additionalProperties\":true},\"description\":\"A set of flow up suggestions for the reply\"},{\"type\":\"null\"}],\"description\":\"A set of flow up suggestions for the reply\"},\"taskId\":{\"type\":[\"string\",\"null\"],\"description\":\"The task ID the reply type is associated with, or none if not applicable\"}},\"required\":[\"type\",\"text\"],\"additionalProperties\":true,\"$schema\":\"http://json-schema.org/draft-07/schema#\"},\"jsonSchema\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"},\"text\":{\"type\":\"string\",\"description\":\"The reply to the user message\"},\"transcript\":{\"type\":[\"string\",\"null\"],\"description\":\"The transcript of the user message, if the input was a voice message\"},\"suggestions\":{\"anyOf\":[{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"text\":{\"type\":\"string\",\"description\":\"The text of the suggestion to show to the user\"},\"type\":{\"type\":\"string\",\"enum\":[\"full\",\"partial\",\"action\"],\"description\":\"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"},\"action\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"}},\"required\":[\"text\",\"type\",\"action\"],\"additionalProperties\":true},\"description\":\"A set of flow up suggestions for the reply\"},{\"type\":\"null\"}],\"description\":\"A set of flow up suggestions for the reply\"},\"taskId\":{\"type\":[\"string\",\"null\"],\"description\":\"The task ID the reply type is associated with, or none if not applicable\"}},\"required\":[\"type\",\"text\"],\"additionalProperties\":true,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}},\"tools\":[null],\"returnToolRequests\":true}",
"genkit:state": "success"
},
"displayName": "bot",
"links": [],
"instrumentationLibrary": {
"name": "genkit-tracer",
"version": "v1"
},
"spanKind": "INTERNAL",
"sameProcessAsParentSpan": {
"value": true
},
"status": {
"code": 0
},
"timeEvents": {
"timeEvent": []
}
},
"973a66090b6e8dd0": {
"spanId": "973a66090b6e8dd0",
"traceId": "5569630ab920a351608eb119092ac5a1",
"parentSpanId": "bfefc0be23fe76a3",
"startTime": 1726056474400,
"endTime": 1726056476317.246,
"attributes": {
"genkit:type": "action",
"genkit:name": "vertexai/gemini-1.5-pro",
"genkit:path": "/{dev-run-action-wrapper}/{botFlow,t:action}/{botFlow,t:flow}/{vertexai/gemini-1.5-pro,t:action,s:model}",
"genkit:input": "{\"messages\":[{\"role\":\"system\",\"content\":[{\"text\":\"\\nYou are a helpful AI task management assistant that takes the user input (text, voice or image) and\\nperforms actions on behalf of the user to organize and manage tasks in a project or answers questions\\nabout the projects tasks. Identify the type of message, perform the action and/or reply to it using the provided tools\\nIf possible provide 3 to 4 suggestion replies to the user input.\\n\\n\"}]},{\"role\":\"user\",\"content\":[{\"text\":\"\\nUseUser message: create a task for fixing a bug in the landing\\n\"},{\"text\":\"\\n\\nOutput should be in JSON format and conform to the following schema:\\n\\n```\\n{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"type\\\":{\\\"type\\\":\\\"string\\\",\\\"enum\\\":[\\\"question\\\",\\\"update\\\",\\\"add\\\",\\\"remove\\\",\\\"breakdown\\\"],\\\"description\\\":\\\"The type of reply:\\\\n- question: the user asked a question and this is the reply.\\\\n - update: a task was updated.\\\\n - add: a task was added.\\\\n - remove: a task was removed.\\\\n - breakdown: a task was broken down.\\\"},\\\"text\\\":{\\\"type\\\":\\\"string\\\",\\\"description\\\":\\\"The reply to the user message\\\"},\\\"transcript\\\":{\\\"type\\\":[\\\"string\\\",\\\"null\\\"],\\\"description\\\":\\\"The transcript of the user message, if the input was a voice message\\\"},\\\"suggestions\\\":{\\\"anyOf\\\":[{\\\"type\\\":\\\"array\\\",\\\"items\\\":{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"text\\\":{\\\"type\\\":\\\"string\\\",\\\"description\\\":\\\"The text of the suggestion to show to the user\\\"},\\\"type\\\":{\\\"type\\\":\\\"string\\\",\\\"enum\\\":[\\\"full\\\",\\\"partial\\\",\\\"action\\\"],\\\"description\\\":\\\"The type of the suggestion:\\\\n- full: The full text to reply directly\\\\n- partial: The starting of the reply to be completed by the user\\\\n- action: An action to be taken\\\"},\\\"action\\\":{\\\"type\\\":\\\"string\\\",\\\"enum\\\":[\\\"question\\\",\\\"update\\\",\\\"add\\\",\\\"remove\\\",\\\"breakdown\\\"],\\\"description\\\":\\\"The type of reply:\\\\n- question: the user asked a question and this is the reply.\\\\n - update: a task was updated.\\\\n - add: a task was added.\\\\n - remove: a task was removed.\\\\n - breakdown: a task was broken down.\\\"}},\\\"required\\\":[\\\"text\\\",\\\"type\\\",\\\"action\\\"],\\\"additionalProperties\\\":true},\\\"description\\\":\\\"A set of flow up suggestions for the reply\\\"},{\\\"type\\\":\\\"null\\\"}],\\\"description\\\":\\\"A set of flow up suggestions for the reply\\\"},\\\"taskId\\\":{\\\"type\\\":[\\\"string\\\",\\\"null\\\"],\\\"description\\\":\\\"The task ID the reply type is associated with, or none if not applicable\\\"}},\\\"required\\\":[\\\"type\\\",\\\"text\\\"],\\\"additionalProperties\\\":true,\\\"$schema\\\":\\\"http://json-schema.org/draft-07/schema#\\\"}\\n```\\n\",\"metadata\":{\"purpose\":\"output\",\"source\":\"default\"}}]}],\"candidates\":1,\"config\":{\"temperature\":0.5},\"tools\":[{\"name\":\"manageTask\",\"description\":\"Create/Update/Remove a task. To create, provide a the task information (no ID), to update provide the ID and the task fields that needs to be updates, to remove provide only the Id. Returns the ID of the task.\",\"outputSchema\":{\"type\":\"string\",\"$schema\":\"http://json-schema.org/draft-07/schema#\"},\"inputSchema\":{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\",\"enum\":[\"create\",\"update\",\"remove\"]},\"id\":{\"type\":\"string\",\"description\":\"the ID of the task\"},\"task\":{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\",\"description\":\"the new title of the task\"},\"description\":{\"type\":\"string\",\"description\":\"the new description of the task\"},\"type\":{\"type\":\"string\",\"enum\":[\"bug\",\"feature\",\"idea\",\"task\"],\"description\":\"the type of task (if unclear use task)\"},\"status\":{\"type\":\"string\",\"description\":\"the new title of the task\"},\"transcript\":{\"type\":\"string\",\"description\":\"the new title of the task\"}},\"additionalProperties\":true,\"description\":\"A task details to be updated or created\"}},\"required\":[\"action\"],\"additionalProperties\":true,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}}],\"output\":{\"format\":\"json\",\"schema\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"},\"text\":{\"type\":\"string\",\"description\":\"The reply to the user message\"},\"transcript\":{\"type\":[\"string\",\"null\"],\"description\":\"The transcript of the user message, if the input was a voice message\"},\"suggestions\":{\"anyOf\":[{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"text\":{\"type\":\"string\",\"description\":\"The text of the suggestion to show to the user\"},\"type\":{\"type\":\"string\",\"enum\":[\"full\",\"partial\",\"action\"],\"description\":\"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"},\"action\":{\"type\":\"string\",\"enum\":[\"question\",\"update\",\"add\",\"remove\",\"breakdown\"],\"description\":\"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"}},\"required\":[\"text\",\"type\",\"action\"],\"additionalProperties\":true},\"description\":\"A set of flow up suggestions for the reply\"},{\"type\":\"null\"}],\"description\":\"A set of flow up suggestions for the reply\"},\"taskId\":{\"type\":[\"string\",\"null\"],\"description\":\"The task ID the reply type is associated with, or none if not applicable\"}},\"required\":[\"type\",\"text\"],\"additionalProperties\":true,\"$schema\":\"http://json-schema.org/draft-07/schema#\"}}}",
"genkit:metadata:subtype": "model",
"genkit:output": "{\"candidates\":[{\"index\":0,\"message\":{\"role\":\"model\",\"content\":[{\"toolRequest\":{\"name\":\"manageTask\",\"input\":{\"task\":\"unknown\",\"action\":\"create\"}}}]},\"finishReason\":\"stop\",\"custom\":{\"safetyRatings\":[{\"category\":\"HARM_CATEGORY_HATE_SPEECH\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.22363281,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.10253906},{\"category\":\"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.47265625,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.16601563},{\"category\":\"HARM_CATEGORY_HARASSMENT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.29101563,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.12988281},{\"category\":\"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.1640625,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.1484375}]}}],\"custom\":{\"candidates\":[{\"content\":{\"role\":\"model\",\"parts\":[{\"functionCall\":{\"name\":\"manageTask\",\"args\":{\"task\":\"unknown\",\"action\":\"create\"}}}]},\"finishReason\":\"STOP\",\"safetyRatings\":[{\"category\":\"HARM_CATEGORY_HATE_SPEECH\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.22363281,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.10253906},{\"category\":\"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.47265625,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.16601563},{\"category\":\"HARM_CATEGORY_HARASSMENT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.29101563,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.12988281},{\"category\":\"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\":\"NEGLIGIBLE\",\"probabilityScore\":0.1640625,\"severity\":\"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\":0.1484375}],\"avgLogprobs\":-0.034768544137477875,\"index\":0}],\"usageMetadata\":{\"promptTokenCount\":591,\"candidatesTokenCount\":6,\"totalTokenCount\":597}},\"usage\":{\"inputCharacters\":2225,\"inputImages\":0,\"inputVideos\":0,\"inputAudioFiles\":0,\"outputCharacters\":0,\"outputImages\":0,\"outputVideos\":0,\"outputAudioFiles\":0,\"inputTokens\":591,\"outputTokens\":6,\"totalTokens\":597},\"latencyMs\":1913.7462920000107}",
"genkit:state": "success"
},
"displayName": "vertexai/gemini-1.5-pro",
"links": [],
"instrumentationLibrary": {
"name": "genkit-tracer",
"version": "v1"
},
"spanKind": "INTERNAL",
"sameProcessAsParentSpan": {
"value": true
},
"status": {
"code": 0
},
"timeEvents": {
"timeEvent": []
}
},
"bfefc0be23fe76a3": {
"spanId": "bfefc0be23fe76a3",
"traceId": "5569630ab920a351608eb119092ac5a1",
"parentSpanId": "2d6b7e89cdf683d3",
"startTime": 1726056474354,
"endTime": 1726056476361.4963,
"attributes": {
"genkit:type": "flow",
"genkit:name": "botFlow",
"genkit:isRoot": true,
"genkit:path": "/{dev-run-action-wrapper}/{botFlow,t:action}/{botFlow,t:flow}",
"genkit:metadata:flow:execution": "0",
"genkit:metadata:flow:name": "botFlow",
"genkit:metadata:flow:id": "89b51b52-4543-4840-9729-fb7aaf72c64d",
"genkit:metadata:flow:dispatchType": "start",
"genkit:metadata:flow:state": "error",
"genkit:input": "{\"uid\":\"ipSL4LxtW325ME9KHXg02fvgGlGv\",\"text\":\"create a task for fixing a bug in the landing\"}",
"genkit:state": "error"
},
"displayName": "botFlow",
"links": [],
"instrumentationLibrary": {
"name": "genkit-tracer",
"version": "v1"
},
"spanKind": "INTERNAL",
"sameProcessAsParentSpan": {
"value": true
},
"status": {
"code": 2,
"message": "FAILED_PRECONDITION: Generation resulted in no candidates matching provided output schema.\n\nCandidate[0] Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- (root): must be object\n\nProvided data:\n\nnull\n\nRequired JSON schema:\n\n{\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": [\n \"question\",\n \"update\",\n \"add\",\n \"remove\",\n \"breakdown\"\n ],\n \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The reply to the user message\"\n },\n \"transcript\": {\n \"type\": [\n \"string\",\n \"null\"\n ],\n \"description\": \"The transcript of the user message, if the input was a voice message\"\n },\n \"suggestions\": {\n \"anyOf\": [\n {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The text of the suggestion to show to the user\"\n },\n \"type\": {\n \"type\": \"string\",\n \"enum\": [\n \"full\",\n \"partial\",\n \"action\"\n ],\n \"description\": \"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"\n },\n \"action\": {\n \"type\": \"string\",\n \"enum\": [\n \"question\",\n \"update\",\n \"add\",\n \"remove\",\n \"breakdown\"\n ],\n \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n }\n },\n \"required\": [\n \"text\",\n \"type\",\n \"action\"\n ],\n \"additionalProperties\": true\n },\n \"description\": \"A set of flow up suggestions for the reply\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"description\": \"A set of flow up suggestions for the reply\"\n },\n \"taskId\": {\n \"type\": [\n \"string\",\n \"null\"\n ],\n \"description\": \"The task ID the reply type is associated with, or none if not applicable\"\n }\n },\n \"required\": [\n \"type\",\n \"text\"\n ],\n \"additionalProperties\": true,\n \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}"
},
"timeEvents": {
"timeEvent": [
{
"time": 1726056476360.4177,
"annotation": {
"attributes": {
"exception.type": "Error",
"exception.message": "FAILED_PRECONDITION: Generation resulted in no candidates matching provided output schema.\n\nCandidate[0] Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- (root): must be object\n\nProvided data:\n\nnull\n\nRequired JSON schema:\n\n{\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": [\n \"question\",\n \"update\",\n \"add\",\n \"remove\",\n \"breakdown\"\n ],\n \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The reply to the user message\"\n },\n \"transcript\": {\n \"type\": [\n \"string\",\n \"null\"\n ],\n \"description\": \"The transcript of the user message, if the input was a voice message\"\n },\n \"suggestions\": {\n \"anyOf\": [\n {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The text of the suggestion to show to the user\"\n },\n \"type\": {\n \"type\": \"string\",\n \"enum\": [\n \"full\",\n \"partial\",\n \"action\"\n ],\n \"description\": \"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"\n },\n \"action\": {\n \"type\": \"string\",\n \"enum\": [\n \"question\",\n \"update\",\n \"add\",\n \"remove\",\n \"breakdown\"\n ],\n \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n }\n },\n \"required\": [\n \"text\",\n \"type\",\n \"action\"\n ],\n \"additionalProperties\": true\n },\n \"description\": \"A set of flow up suggestions for the reply\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"description\": \"A set of flow up suggestions for the reply\"\n },\n \"taskId\": {\n \"type\": [\n \"string\",\n \"null\"\n ],\n \"description\": \"The task ID the reply type is associated with, or none if not applicable\"\n }\n },\n \"required\": [\n \"type\",\n \"text\"\n ],\n \"additionalProperties\": true,\n \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}",
"exception.stacktrace": "Error: FAILED_PRECONDITION: Generation resulted in no candidates matching provided output schema.\n\nCandidate[0] Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- (root): must be object\n\nProvided data:\n\nnull\n\nRequired JSON schema:\n\n{\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": [\n \"question\",\n \"update\",\n \"add\",\n \"remove\",\n \"breakdown\"\n ],\n \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n },\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The reply to the user message\"\n },\n \"transcript\": {\n \"type\": [\n \"string\",\n \"null\"\n ],\n \"description\": \"The transcript of the user message, if the input was a voice message\"\n },\n \"suggestions\": {\n \"anyOf\": [\n {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"text\": {\n \"type\": \"string\",\n \"description\": \"The text of the suggestion to show to the user\"\n },\n \"type\": {\n \"type\": \"string\",\n \"enum\": [\n \"full\",\n \"partial\",\n \"action\"\n ],\n \"description\": \"The type of the suggestion:\\n- full: The full text to reply directly\\n- partial: The starting of the reply to be completed by the user\\n- action: An action to be taken\"\n },\n \"action\": {\n \"type\": \"string\",\n \"enum\": [\n \"question\",\n \"update\",\n \"add\",\n \"remove\",\n \"breakdown\"\n ],\n \"description\": \"The type of reply:\\n- question: the user asked a question and this is the reply.\\n - update: a task was updated.\\n - add: a task was added.\\n - remove: a task was removed.\\n - breakdown: a task was broken down.\"\n }\n },\n \"required\": [\n \"text\",\n \"type\",\n \"action\"\n ],\n \"additionalProperties\": true\n },\n \"description\": \"A set of flow up suggestions for the reply\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"description\": \"A set of flow up suggestions for the reply\"\n },\n \"taskId\": {\n \"type\": [\n \"string\",\n \"null\"\n ],\n \"description\": \"The task ID the reply type is associated with, or none if not applicable\"\n }\n },\n \"required\": [\n \"type\",\n \"text\"\n ],\n \"additionalProperties\": true,\n \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}\n at /Users/marcel/Dev/GitHub/mumble-sort/functions/node_modules/@genkit-ai/ai/lib/generate.js:563:15\n at Generator.next (<anonymous>)\n at fulfilled (/Users/marcel/Dev/GitHub/mumble-sort/functions/node_modules/@genkit-ai/ai/lib/generate.js:43:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
},
"description": "exception"
}
}
]
}
}
}
}
Jep, I also have this problem. It seems the tool response is validated against the final output schema of the generate call. Tool Requests should be ignored when validating the schema. Or even better: Tool Requests should be validated against the output schema of the tool
Here's a minimal repro:
Code
import { z } from "genkit";
import { googleAI, gemini15Flash } from "@genkit-ai/googleai";
import { genkit } from "genkit";
export const ai = genkit({
model: gemini15Flash,
plugins: [googleAI()],
});
export const schema = z.object({
location: z.string(),
});
export const debugToolFlow = ai.defineFlow(
{
name: "debugToolFlow",
inputSchema: schema,
outputSchema: schema,
},
async (input) => {
const { output } = await ai.generate({
output: {
schema: schema,
},
tools: [debugTool],
prompt: `Use the placesApiTool to return a location nearby of ${input.location}`,
});
if (output == null) {
throw "output was null";
}
return output;
}
);
export const debugTool = ai.defineTool(
{
name: "placesApiTool",
description: "A tool that uses the Places API to find nearby locations.",
inputSchema: schema,
outputSchema: schema,
},
async (_) => {
return { location: "abc" };
}
);
ai.startFlowServer({
flows: [debugToolFlow],
port: 3406,
});
$ curl -X POST "http://localhost:3406/debugToolFlow" -H "Content-Type: application/json" -d '{"data": {"location": "sf"}}'
{"error":{"status":"INTERNAL","message":"INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- (root): must be object\n\nProvided data:\n\nnull\n\nRequired JSON schema:\n\n{\n \"type\": \"object\",\n \"properties\": {\n \"location\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"location\"\n ],\n \"additionalProperties\": true,\n \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}","details":"Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:\n\n- (root): must be object\n\nProvided data:\n\nnull\n\nRequired JSON schema:\n\n{\n \"type\": \"object\",\n \"properties\": {\n \"location\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"location\"\n ],\n \"additionalProperties\": true,\n \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}\n
...
Or the error in full:
INVALID_ARGUMENT: Schema validation failed. Parse Errors: - (root): must be object Provided data: null Required JSON schema: { "type": "object", "properties": { "location": { "type": "string" } }, "required": [ "location" ], "additionalProperties": true, "$schema": "http://json-schema.org/draft-07/schema#" }
Error: INVALID_ARGUMENT: Schema validation failed. Parse Errors:
- (root): must be object
Provided data:
null
Required JSON schema:
{
"type": "object",
"properties": {
"location": {
"type": "string"
}
},
"required": [
"location"
],
"additionalProperties": true,
"$schema": "http://json-schema.org/draft-07/schema#"
}
The issue appears to be that generate currently does not support specifying output: {schema} and tools at the same time....
here:
await ai.generate({
output: {
schema: schema,
},
tools: [debugTool],
prompt: `Use the placesApiTool to return a location nearby of ${input.location}`,
});
what happens is that it applies schema validation in the tool calling loop (when it's not really expected to conform, yet).
Let me see if there's a quick fix for this...
