openai-cookbook icon indicating copy to clipboard operation
openai-cookbook copied to clipboard

Add LLMOps folder and W&B + OpenAI integration quickstart

Open ash0ts opened this issue 1 year ago • 5 comments

Hello! I work at Weights and Biases and we have many offerings/teachings related to OpenAI as we work closely with them. Our MLOps platform has many first class integrations to for usage with OpenAI that we would love to show the community.

The W&B OpenAI AutoLogger logs requests, responses, token counts and model metadata with 1 line of code for all OpenAI models, allowing users to operationalize their LLM workflows.

Let me know if there are any suggestions or changes!

ash0ts avatar May 12 '23 18:05 ash0ts

Hey @ted-at-openai! I was wondering if there were any other changes I needed to make

ash0ts avatar May 23 '23 16:05 ash0ts

Thanks for the ping. I will take a look. (Apologies for the slow pace on this repo.)

ted-at-openai avatar May 23 '23 23:05 ted-at-openai

Ok, going through the notebook now. Immediately hit an error in the first cell:

Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb: Currently logged in as: {username}. Use `wandb login --relogin` to force relogin
wandb: ERROR Error while calling W&B API: permission denied (<Response [403]>)
Problem at: [path] 108 _run_init
---------------------------------------------------------------------------
CommError                                 Traceback (most recent call last)
...
CommError: It appears that you do not have permission to access the requested resource. Please reach out to the project owner to grant you access. If you have the correct permissions, verify that there are no issues with your networking setup.(Error 403: Forbidden)

Not obvious to me what I did wrong or how to fix it, so I expect other users might also get confused here. Are users supposed to enter a pre-existing llm project name?

ted-at-openai avatar May 26 '23 22:05 ted-at-openai

In the meantime, I've added a link on the Cookbook homepage advertising W&B as "A paid product for tracking model training and prompt engineering experiments." Hope this helps people discover what you're building.

ted-at-openai avatar May 26 '23 22:05 ted-at-openai

One thought: This notebook looks pretty brief. Do you think it would make sense to discuss the value of logging inputs and outputs? Or a scenario in which a user might benefit from the logs? Right now I worry people might stumble upon it, run the ABC process of (a) turn logger on, (b) call API, (c) turn logger off, without really understanding what that might be useful to them.

ted-at-openai avatar May 26 '23 22:05 ted-at-openai

Still want to add this? In the meantime, we've added a folder third_party_examples where this would fit nicely.

ted-at-openai avatar Jul 12 '23 00:07 ted-at-openai

#714 Hey @ted-at-openai, I decided to open a new PR to add W&B examples to the third_party_examples you mentioned.

ash0ts avatar Sep 13 '23 16:09 ash0ts