microsoft
[Bug]: No try-except for Content Filtering issues
Do you need to file an issue?
- [X] I have searched the existing issues and this bug is not already filed.
- [X] My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
- [X] I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug
As an experiment I tried to run the GraphRAG code on 1000 randomly selected wikipedia articles. I have tried different random samples, but most of them usually end up with some articles related to War, Nazis or similar.
This usually triggers a content filtering error when I use Azure OpenAI models which makes sense. However, why does it have to crash the whole GraphRAG pipeline? I think it would make more sense to "skip errors". An annoying workaround is to manually go and remove articles / entities relating to topics that I think might trigger content filtering errors, but this is not a good approach.
Steps to reproduce
try to run GraphRAG using Azure OpenAI on any article that could trigger content filtering. Like the following: https://no.wikipedia.org/wiki/Erich%20Priebke
Expected Behavior
Ignore errors, move on and remove corresponding entities / summaries / relationships if necessary.
GraphRAG Config Used
# Paste your config here
encoding_model: cl100k_base
skip_workflows: []
llm:
api_key: ${AZURE_OPENAI_API_KEY}
type: azure_openai_chat #openai_chat # or azure_openai_chat
model: gpt-4o #gpt-4-turbo-preview
model_supports_json: true # recommended if this is available for your model.
# max_tokens: 4000
# request_timeout: 180.0
api_base: https://myresource.openai.azure.com/
api_version: 2024-02-15-preview
# organization: <organization_id>
deployment_name: gpt-4o
tokens_per_minute: 800_000 # set a leaky bucket throttle
requests_per_minute: 5000 # set a leaky bucket throttle
#max_retries: 10
# max_retry_wait: 10.0
# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times
concurrent_requests: 400 # the number of parallel inflight requests that may be made
parallelization:
stagger: 0.3
# num_threads: 50 # the number of threads to use for parallel processing
async_mode: threaded # or asyncio
embeddings:
## parallelization: override the global parallelization settings for embeddings
async_mode: threaded # or asyncio
llm:
api_key: ${AZURE_OPENAI_API_KEY}
type: azure_openai_embedding #openai_embedding # or azure_openai_embedding
model: text-embedding-3-large #text-embedding-3-small
api_base: https://myresource.openai.azure.com
api_version: 2024-02-15-preview
# organization: <organization_id>
deployment_name: text-embedding-3-large
tokens_per_minute: 300_000 # set a leaky bucket throttle
requests_per_minute: 1500 # set a leaky bucket throttle
# max_retries: 10
# max_retry_wait: 10.0
# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times
concurrent_requests: 200 # the number of parallel inflight requests that may be made
# batch_size: 16 # the number of documents to send in a single request
# batch_max_tokens: 8191 # the maximum number of tokens to send in a single request
# target: required # or optional
chunks:
size: 300
overlap: 100
group_by_columns: [id] # by default, we don't allow chunks to cross documents
input:
type: file # or blob
file_type: text # or csv
base_dir: "input"
file_encoding: utf-8
file_pattern: ".*\\.txt$"
cache:
type: file # or blob
base_dir: "cache"
# connection_string: <azure_blob_storage_connection_string>
# container_name: <azure_blob_storage_container_name>
storage:
type: file # or blob
base_dir: "output/${timestamp}/artifacts"
# connection_string: <azure_blob_storage_connection_string>
# container_name: <azure_blob_storage_container_name>
reporting:
type: file # or console, blob
base_dir: "output/${timestamp}/reports"
# connection_string: <azure_blob_storage_connection_string>
# container_name: <azure_blob_storage_container_name>
entity_extraction:
## llm: override the global llm settings for this task
## parallelization: override the global parallelization settings for this task
## async_mode: override the global async_mode settings for this task
prompt: "prompts/entity_extraction.txt"
entity_types: [organization,person,geo,event]
max_gleanings: 0
summarize_descriptions:
## llm: override the global llm settings for this task
## parallelization: override the global parallelization settings for this task
## async_mode: override the global async_mode settings for this task
prompt: "prompts/summarize_descriptions.txt"
max_length: 500
claim_extraction:
## llm: override the global llm settings for this task
## parallelization: override the global parallelization settings for this task
## async_mode: override the global async_mode settings for this task
enabled: true
prompt: "prompts/claim_extraction.txt"
description: "Any claims or facts that could be relevant to information discovery."
max_gleanings: 0
community_report:
## llm: override the global llm settings for this task
## parallelization: override the global parallelization settings for this task
## async_mode: override the global async_mode settings for this task
prompt: "prompts/community_report.txt"
max_length: 2000
max_input_length: 8000
cluster_graph:
max_cluster_size: 10
embed_graph:
enabled: false # if true, will generate node2vec embeddings for nodes
# num_walks: 10
# walk_length: 40
# window_size: 2
# iterations: 3
# random_seed: 597832
umap:
enabled: false # if true, will generate UMAP embeddings for nodes
snapshots:
graphml: true
raw_entities: true
top_level_nodes: true
local_search:
# text_unit_prop: 0.5
# community_prop: 0.1
# conversation_history_max_turns: 5
# top_k_mapped_entities: 10
# top_k_relationships: 10
# max_tokens: 12000
global_search:
# max_tokens: 12000
# data_max_tokens: 12000
# map_max_tokens: 1000
# reduce_max_tokens: 2000
# concurrency: 32
Logs and screenshots
{"type": "error", "data": "Error Invoking LLM", "stack": "Traceback (most recent call last):\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/llm/base/base_llm.py", line 53, in _invoke\n output = await self._execute_llm(input, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/llm/openai/openai_chat_llm.py", line 55, in _execute_llm\n completion = await self.client.chat.completions.create(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/openai/resources/chat/completions.py", line 1289, in create\n return await self._post(\n ^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/openai/_base_client.py", line 1805, in post\n return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/openai/_base_client.py", line 1503, in request\n return await self._request(\n ^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/openai/_base_client.py", line 1599, in _request\n raise self._make_status_error_from_response(err.response) from None\nopenai.BadRequestError: Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': True, 'severity': 'medium'}}}}}\n", "source": "Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': True, 'severity': 'medium'}}}}}", "details": {"input": "\nYou are a helpful assistant responsible for generating a comprehensive summary of the data provided below.\nGiven one or two entities, and a list of descriptions, all related to the same entity or group of entities.\nPlease concatenate all of these into a single, comprehensive description. Make sure to include information collected from all the descriptions.\nIf the provided descriptions are contradictory, please resolve the contradictions and provide a single, coherent summary.\nMake sure it is written in third person, and include the entity names so we the have full context.\n\n#######\n-Data-\nEntities: "\"AUSCHWITZ\""\nDescription List: ["\"Auschwitz is a Nazi concentration and extermination camp located in Poland.\"", "\"Auschwitz is a Nazi concentration and extermination camp where many Jews from M\u00f8re og Romsdal were killed.\"", "\"Auschwitz is a Nazi concentration and extermination camp where over 770 Jews from Norway were sent during World War II.\"", "\"Auschwitz is a Nazi extermination camp where over 770 Jews from Norway were sent during World War II.\"", "\"Auschwitz is a location in Poland where the Red Army found Nazi concentration camps in 1945.\"", "\"Auschwitz is a location in Poland where the extermination camp was situated, and to which 158 Norwegian Jews were transported in 1943.\"", "\"Auschwitz is a location in Poland where the first female prisoners arrived during World War II in 1942.\"", "\"Auschwitz is a location known for being a Nazi concentration and extermination camp during World War II.\"", "\"Auschwitz is a location where Priebke is believed to have sent 6,000-7,000 Jews from Italy during his involvement in the Holocaust.\"", "\"Auschwitz is the concentration camp to which Jews from Hungary were deported in 1944.\"", "\"Auschwitz was a German extermination camp during World War II, located in occupied Poland.\"", "\"Auschwitz was a network of concentration and extermination camps built and operated by Nazi Germany during World War II.\"", "\"Auschwitz, a network of concentration and extermination camps operated by Nazi Germany during World War II.\""]\n#######\nOutput:\n"}}
{"type": "error", "data": "Error executing verb "summarize_descriptions" in create_summarized_entities: Error code: 400 - {'error': {'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': False, 'severity': 'safe'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': True, 'severity': 'medium'}}}}}", "stack": "Traceback (most recent call last):\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/datashaper/workflow/workflow.py", line 415, in _execute_verb\n result = await result\n ^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/index/verbs/entities/summarize/description_summarize.py", line 184, in summarize_descriptions\n await get_resolved_entities(row, semaphore) for row in output.itertuples()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/index/verbs/entities/summarize/description_summarize.py", line 147, in get_resolved_entities\n results = await asyncio.gather(*futures)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/index/verbs/entities/summarize/description_summarize.py", line 167, in do_summarize_descriptions\n results = await strategy_exec(\n ^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/index/verbs/entities/summarize/strategies/graph_intelligence/run_graph_intelligence.py", line 34, in run\n return await run_summarize_descriptions(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/index/verbs/entities/summarize/strategies/graph_intelligence/run_graph_intelligence.py", line 67, in run_summarize_descriptions\n result = await extractor(items=items, descriptions=descriptions)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/index/graph/extractors/summarize/description_summary_extractor.py", line 73, in call\n result = await self._summarize_descriptions(items, descriptions)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/index/graph/extractors/summarize/description_summary_extractor.py", line 106, in _summarize_descriptions\n result = await self._summarize_descriptions_with_llm(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/index/graph/extractors/summarize/description_summary_extractor.py", line 125, in _summarize_descriptions_with_llm\n response = await self._llm(\n ^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/llm/openai/json_parsing_llm.py", line 34, in call\n result = await self._delegate(input, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/llm/openai/openai_token_replacing_llm.py", line 37, in call\n return await self._delegate(input, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/llm/openai/openai_history_tracking_llm.py", line 33, in call\n output = await self._delegate(input, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/llm/base/caching_llm.py", line 104, in call\n result = await self._delegate(input, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 177, in call\n result, start = await execute_with_retry()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/graphrag/llm/base/rate_limiting_llm.py", line 159, in execute_with_retry\n async for attempt in retryer:\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 166, in anext\n do = await self.iter(retry_state=self._retry_state)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 153, in iter\n result = await action(retry_state)\n ^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner\n return call(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/Users/SKFR/.pyenv/versions/3.12.3/lib/python3.12/site-packages/tenacity/init.py", line 398, in
Additional Information
- GraphRAG Version:
- Operating System:
- Python Version:
- Related Issues:
