[BUG] Running with Groq: too many con

[oscmejia]

Describe the bug A clear and concise description of what the bug is.

To Reproduce I'm calling it from the root directory:

python examples/run_storm_wiki_groq.py \
    --output-dir test_output \
    --retriever serper \
    --do-research \
    --do-generate-outline \
    --do-generate-article \
    --do-polish-article 

Topic: Dogs
Traceback (most recent call last):
  File "/home/user1/storm/examples/run_storm_wiki_groq.py", line 121, in main
    runner.run(
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/knowledge_storm/storm_wiki/engine.py", line 293, in run
    information_table = self.run_knowledge_curation_module(ground_truth_url=ground_truth_url,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/knowledge_storm/interface.py", line 376, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/knowledge_storm/storm_wiki/engine.py", line 165, in run_knowledge_curation_module
    information_table, conversation_log = self.storm_knowledge_curation_module.research(
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/knowledge_storm/storm_wiki/modules/knowledge_curation.py", line 311, in research
    conversations = self._run_conversation(conv_simulator=self.conv_simulator,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/knowledge_storm/storm_wiki/modules/knowledge_curation.py", line 278, in _run_conversation
    conv = future.result()
           ^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/knowledge_storm/storm_wiki/modules/knowledge_curation.py", line 259, in run_conv
    return conv_simulator(
           ^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/dspy/primitives/program.py", line 26, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/knowledge_storm/storm_wiki/modules/knowledge_curation.py", line 49, in forward
    user_utterance = self.wiki_writer(topic=topic, persona=persona, dialogue_turns=dlg_history).question
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/dspy/primitives/program.py", line 26, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/knowledge_storm/storm_wiki/modules/knowledge_curation.py", line 92, in forward
    question = self.ask_question_with_persona(topic=topic, persona=persona, conv=conv).question
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/dspy/predict/predict.py", line 61, in __call__
    return self.forward(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/dspy/predict/chain_of_thought.py", line 59, in forward
    return super().forward(signature=signature, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/dspy/predict/predict.py", line 103, in forward
    x, C = dsp.generate(template, **config)(x, stage=self.stage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/dsp/primitives/predict.py", line 77, in do_generate
    completions: list[dict[str, Any]] = generator(prompt, **kwargs)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/storm/knowledge_storm/lm.py", line 352, in __call__
    response = self._create_completion(prompt, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user1/storm/knowledge_storm/lm.py", line 338, in _create_completion
    response.raise_for_status()
  File "/home/user1/miniconda3/envs/storm/lib/python3.11/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://api.groq.com/openai/v1/chat/completions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user1/storm/examples/run_storm_wiki_groq.py", line 174, in <module>
    main(parser.parse_args())
  File "/home/user1/storm/examples/run_storm_wiki_groq.py", line 132, in main
    logger.exception(f"An error occurred: {str(e)}")
    ^^^^^^
NameError: name 'logger' is not defined

Environment: Ubuntu 22LTS

[danieldekay]

did you try to add import logger to the top of that script?

[Yucheng-Jiang]

as danieldekay suggested, by importing logger should resolve the issue. Close the issue now. If you run into further problems, feel free to open another issue.