dify icon indicating copy to clipboard operation
dify copied to clipboard

Add new integration with Opik Tracking tool

Open Lothiraldan opened this issue 1 year ago • 15 comments

Summary

This PR integrates Opik (https://github.com/comet-ml/opik), an open-source platform for evaluating, testing, and monitoring LLM applications. It features built-in evaluation metrics and data annotation. Closes #11407, requested by a community user.

Screenshots

Before After
main branch

Checklist

[!IMPORTANT]
Please review the checklist below before submitting your pull request.

  • [x] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs/pulls)
  • [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • [x] I've updated the documentation accordingly.
  • [x] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Lothiraldan avatar Dec 09 '24 11:12 Lothiraldan

CleanShot 2024-12-12 at 15 43 28@2x CleanShot 2024-12-12 at 15 53 23@2x The elements have a position problem whenever I add another ops trace provider.

ZhouhaoJiang avatar Dec 12 '24 07:12 ZhouhaoJiang

CleanShot 2024-12-12 at 15 49 36@2x CleanShot 2024-12-12 at 15 50 49@2x When I add a credential, I don't fill in the Url field, because I see that it has a default value, but the actual request interface does not include this value, in order to better user experience should set it to the default value.

I will remove the api key after I finish test it.

ZhouhaoJiang avatar Dec 12 '24 07:12 ZhouhaoJiang

I have tested basic chat, agent chat, and workflow. There have no problems with data tracing.

ZhouhaoJiang avatar Dec 12 '24 08:12 ZhouhaoJiang

@ZhouhaoJiang Thanks for your review. I fixed the issue for the default url, it was actually on the backend. I'm guessing that the frontend was sending an empty string for it.

I'm not sure to understand your feedback about the position of the Opik Tracing Provider. I added Opik at the end of both lists, the first one with configured tracing providers and the second one with non-configured tracing providers. Do you prefer a different behavior?

Lothiraldan avatar Dec 12 '24 15:12 Lothiraldan

@ZhouhaoJiang Thanks for your review. I fixed the issue for the default url, it was actually on the backend. I'm guessing that the frontend was sending an empty string for it.

I'm not sure to understand your feedback about the position of the Opik Tracing Provider. I added Opik at the end of both lists, the first one with configured tracing providers and the second one with non-configured tracing providers. Do you prefer a different behavior?

I think my explanation is not clear. What I mean is that after one provider is configured, there is no gap between the components of the other two providers, and there is a problem with the ui

ZhouhaoJiang avatar Dec 12 '24 16:12 ZhouhaoJiang

@ZhouhaoJiang I was able to fix the spacing issue but I realized that the Opik provider is missing the "IN USE" tag, I will check it out and fix the merge conflict on api/poetry.lock

Screenshot 2024-12-13 at 10-30-36 Long Story Generator (Iteration) - Dify Screenshot 2024-12-13 at 10-30-19 Long Story Generator (Iteration) - Dify

Lothiraldan avatar Dec 13 '24 09:12 Lothiraldan

I fixed the merged conflict on poetry.lock and finally understood that you can have only a single provider enabled per project. That's why I only saw only one provider with the "IN USE" tag. I think I fixed everything, let me know otherwise

Lothiraldan avatar Dec 13 '24 13:12 Lothiraldan

@ZhouhaoJiang Thanks, I will update the workflow trace method accordingly. Can you remind me in which case the message_if field in the WorkflowTraceInfo could be None? I want to double-check that everything works nicely after the update.

Lothiraldan avatar Dec 16 '24 09:12 Lothiraldan

@ZhouhaoJiang I've updated the Opik tracer to use workflow_run_id or message_id when present.

Lothiraldan avatar Dec 17 '24 15:12 Lothiraldan

Thank you for this work, we'll test this feature in our development environment and may need to do some work on documentation.

laipz8200 avatar Dec 19 '24 18:12 laipz8200

@laojianzi Let me know if there is anything I can do on the documentation or the code. Do you want me to take care of the poetry.lock conflict?

Lothiraldan avatar Jan 06 '25 08:01 Lothiraldan

@Lothiraldan Can you rebase the origin/main branch and push it again. I will merge it later.

crazywoola avatar Jan 08 '25 04:01 crazywoola

@crazywoola Sure, I rebased and pushed

Lothiraldan avatar Jan 08 '25 08:01 Lothiraldan

Hello, it seems that the tests are failed.

Please add this to this file https://github.com/langgenius/dify/blob/d91960905971316ef27d7da5777cfa71ac85e521/web/i18n/zh-Hans/app.ts as well

 opik: {
      title: 'Opik',
      description: '一个全方位的开发者平台,适用于 LLM 驱动应用程序生命周期的每个步骤。',
 },

crazywoola avatar Jan 08 '25 09:01 crazywoola

@crazywoola I see it, I'm working on resolving it now. Will also update this file.

Lothiraldan avatar Jan 08 '25 09:01 Lothiraldan