[Feature]: Slack message redaction

[krrishdholakia]

The Feature

Allow proxy admin to enable message redaction for slack alerts (e.g. cost failure alert contains the entire prompt sent to model).

It can be similar to how we do it for otel -

callback_settings:
  otel:
    message_logging: False

Motivation, pitch

User request - they don't want to leak PII

Are you a ML Ops Team?

No

Twitter / LinkedIn details

No response

[krrishdholakia]

Example -

{
  "text": "Level: `High`\nTimestamp: `06:59:39`\n\nMessage: LLM API call failed: `litellm.AuthenticationError: AzureException AuthenticationError - Access denied due to invalid subscription key or wrong API endpoint. Make sure to provide a valid key for an active subscription and use a correct regional API endpoint for your resource.. Received Model Group=gpt-4o-azure\nAvailable Model Group Fallbacks=None\nModel: azure/gpt-4o\nAPI Base: `https://openai-gpt-4-test-v-1.openai.azure.com`\nMessages: `[{'role': 'system', 'content': 'You are a helpful AI assistant.'}, {'role': 'user', 'content': [{'ty`\nmodel_group: `gpt-4o-azure`\n\ndeployment: `azure/gpt-4o`\n`\n\n🪢 Langfuse Trace: `https://us.cloud.langfuse.com/trace/None`\n\n"
}

Steps to repro:

Pre-requisites

SLACK_WEBHOOK_URL="your-webhook-url" # can be anything - e.g. https://webhook.site/#!/view/84938e6d-d530-492f-bc96-837c0e5d2ced/be9dfea3-17c8-45e7-b05d-8aecd02aa13e/1

1. Setup config.yaml with slack + invalid message subscription

model_list:
  - model_name: "gpt-4o-azure"
    litellm_params:
      model: azure/gpt-4o
      api_key: os.environ/AZURE_API_KEY
      api_base: os.environ/AZURE_API_BASE

general_settings:
  alerting: ["slack"]

2. Test it

curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
    "messages": [
        {
          "role": "system",
          "content": "You are a helpful AI assistant."
        },
        {
          "role": "user",
          "content": [
            {
                "type": "text",
                "text": "Hello"
            }
          ]
        }
      ],
        "max_tokens": 2048,
        "temperature": 0.8,
        "top_p": 0.1,
        "model": "gpt-4.1",
        "metadata": {
            "tags": ["tag-2"]
        }
}'

3. View log

{
  "text": "Level: `High`\nTimestamp: `06:59:39`\n\nMessage: LLM API call failed: `litellm.AuthenticationError: AzureException AuthenticationError - Access denied due to invalid subscription key or wrong API endpoint. Make sure to provide a valid key for an active subscription and use a correct regional API endpoint for your resource.. Received Model Group=gpt-4o-azure\nAvailable Model Group Fallbacks=None\nModel: azure/gpt-4o\nAPI Base: `https://openai-gpt-4-test-v-1.openai.azure.com`\nMessages: `[{'role': 'system', 'content': 'You are a helpful AI assistant.'}, {'role': 'user', 'content': [{'ty`\nmodel_group: `gpt-4o-azure`\n\ndeployment: `azure/gpt-4o`\n`\n\n🪢 Langfuse Trace: `https://us.cloud.langfuse.com/trace/None`\n\n"
}

[krrishdholakia]

2nd test

Level: High
Timestamp: 15:43:55
Message: Failed Tracking Cost for Error in tracking cost callback - User API key and team id and user id missing from custom callback.
 Traceback:Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/litellm/proxy/hooks/proxy_track_cost_callback.py", line 179, in _PROXY_track_cost_callback
    raise Exception(
        "User API key and team id and user id missing from custom callback."
    )
Exception: User API key and team id and user id missing from custom callback.
 Args to _PROXY_track_cost_callback
 model: gpt-4o-mini
 metadata: {'requester_metadata': {}, 'user_api_key_hash': '97a696212922e0ea4d8606d810f04cc62a78e24d5697b897580080811d064e94', 'user_api_key_alias': 'tables-production', 'user_api_key_team_id': 'fa33f772-3d28-4f59-ab13-24c538ae888c', 'user_api_key_user_id': 'john.doe@test_company.com', 'user_api_key_org_id': None, 'user_api_key_team_alias': 'Tables', 'user_api_key_end_user_id': None, 'user_api_key_user_email': 'john.doe@test_company.com', 'user_api_key': '97a696212922e0ea4d8606d810f04cc62a78e24d5697b897580080811d064e94', 'user_api_end_user_max_budget': None, 'litellm_api_version': '1.66.0', 'global_max_parallel_requests': None, 'spend_logs_metadata': {'owner': 'team-tables', 'product': 'tables', 'environment': 'production'}, 'user_api_key_team_max_budget': None, 'user_api_key_team_spend': 204.79840064498023, 'user_api_key_spend': 299.2005019400869, 'user_api_key_max_budget': None, 'user_api_key_model_max_budget': {}, 'user_api_key_metadata': {'tags': None, 'guardrails': [], 'spend_logs_metadata': {'owner': 'team-tables', 'product': 'tables', 'environment': 'production'}}, 'headers': {'host': 'litellm.test_company.com', 'x-request-id': '057cb0b139c0c2f04694413374bad8b9', 'x-real-ip': '52.204.156.156', 'x-forwarded-for': '52.204.156.156', 'x-forwarded-host': 'litellm.test_company.com', 'x-forwarded-port': '443', 'x-forwarded-proto': 'https', 'x-forwarded-scheme': 'https', 'x-scheme': 'https', 'content-length': '7501', 'sentry-trace': 'a6bbedd97d7a41918a7920be620a1c86-a606e00f79e26f25-1', 'baggage': 'sentry-trace_id=a6bbedd97d7a41918a7920be620a1c86,sentry-environment=production,sentry-public_key=6c84a89d3c9647b28f8d026baafab9d5,sentry-transaction=/tables/%7Btable_id%7D/fields,sentry-sample_rate=1.0', 'user-agent': 'tablestore', 'accept-encoding': 'gzip, deflate', 'accept': '*/*', 'content-type': 'application/json', 'x-datadog-trace-id': '5001835662382784589', 'x-datadog-parent-id': '16915630724735267537', 'x-datadog-sampling-priority': '1', 'x-datadog-origin': 'rum', 'traceparent': '00-0000000000000000456a1708b934e04d-eac06485738772d1-01', 'tracestate': 'dd=s:1;o:rum'}, 'endpoint': 'http://litellm.test_company.com/v1/chat/completions', 'litellm_parent_otel_span': _Span(name="Received Proxy Server Request", context=SpanContext(trace_id=0x0000000000000000456a1708b934e04d, span_id=0x2bd024d1cc6064de, trace_flags=0x01, trace_state=[], is_remote=False)), 'requester_ip_address': '10.4.60.153', 'guardrails': [], 'model_group': 'gpt-4o-mini', 'model_group_size': 1, 'deployment': 'openai/gpt-4o-mini', 'model_info': {'id': '4c891f196c9d7bcb564a3ee990fc5e536cd1481ef91c53ab0b63dd827679548b', 'db_model': False}, 'api_base': None, 'hidden_params': {'custom_llm_provider': 'openai', 'region_name': None, 'headers': {'date': 'Wed, 23 Apr 2025 15:43:52 GMT', 'content-type': 'application/json', 'transfer-encoding': 'chunked', 'connection': 'keep-alive', 'access-control-expose-headers': 'X-Request-ID', 'openai-organization': 'test_company-new-account-1', 'openai-processing-ms': '1850', 'openai-version': '2020-10-01', 'x-ratelimit-limit-requests': '30000', 'x-ratelimit-limit-tokens': '150000000', 'x-ratelimit-remaining-requests': '29999', 'x-ratelimit-remaining-tokens': '149998977', 'x-ratelimit-reset-requests': '2ms', 'x-ratelimit-reset-tokens': '0s', 'x-request-id': 'req_e5510ce7924249f39335e61014b2cf49', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'cf-cache-status': 'DYNAMIC', 'x-content-type-options': 'nosniff', 'server': 'cloudflare', 'cf-ray': '934e7a949d0d0826-IAD', 'content-encoding': 'gzip', 'alt-svc': 'h3=":443"; ma=86400'}, 'additional_headers': {'x-ratelimit-limit-requests': '30000', 'x-ratelimit-remaining-requests': '29999', 'x-ratelimit-limit-tokens': '150000000', 'x-ratelimit-remaining-tokens': '149998977', 'llm_provider-date': 'Wed, 23 Apr 2025 15:43:52 GMT', 'llm_provider-content-type': 'application/json', 'llm_provider-transfer-encoding': 'chunked', 'llm_provider-connection': 'keep-alive', 'llm_provider-access-control-expose-headers': 'X-Request-ID', 'llm_provider-openai-organization': 'test_company-new-account-1', 'llm_provider-openai-processing-ms': '1850', 'llm_provider-openai-version': '2020-10-01', 'llm_provider-x-ratelimit-limit-requests': '30000', 'llm_provider-x-ratelimit-limit-tokens': '150000000', 'llm_provider-x-ratelimit-remaining-requests': '29999', 'llm_provider-x-ratelimit-remaining-tokens': '149998977', 'llm_provider-x-ratelimit-reset-requests': '2ms', 'llm_provider-x-ratelimit-reset-tokens': '0s', 'llm_provider-x-request-id': 'req_e5510ce7924249f39335e61014b2cf49', 'llm_provider-strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'llm_provider-cf-cache-status': 'DYNAMIC', 'llm_provider-x-content-type-options': 'nosniff', 'llm_provider-server': 'cloudflare', 'llm_provider-cf-ray': '934e7a949d0d0826-IAD', 'llm_provider-content-encoding': 'gzip', 'llm_provider-alt-svc': 'h3=":443"; ma=86400', 'x-litellm-model-group': 'gpt-4o-mini', 'x-litellm-attempted-retries': 0, 'x-litellm-attempted-fallbacks': 0}, 'optional_params': {'functions': [{'name': 'map_keys', 'description': 'establishes a one-to-one mapping of actual_keys to sample_record_keys given a sample_record', 'parameters': {'type': 'object', 'description': 'a one-to-one mapping of actual_keys to sample_record_keys', 'required': ['f1', 'f2', 'f5', 'f6', 'f7', 'f8', 'f9', 'f16', 'f24', 'f25', 'f26', 'f27', 'f28', 'f29', 'f30', 'f33', 'f34', 'f35', 'f36'], 'properties': {'f1': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f1 with name 'Student First Name'"}, 'f2': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f2 with name 'Student Last Name'"}, 'f5': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f5 with name 'College'"}, 'f6': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f6 with name 'Deal Name'"}, 'f7': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f7 with name 'Reporting Program'"}, 'f8': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f8 with name 'High School'"}, 'f9': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f9 with name 'Graduation Year'"}, 'f16': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f16 with name 'Student Phone Number'"}, 'f24': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f24 with name 'Consent of Child's Media Release'"}, 'f25': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f25 with name 'Record ID'"}, 'f26': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f26 with name 'Deal ID'"}, 'f27': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f27 with name 'Attendance Date'"}, 'f28': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f28 with name 'Notes'"}, 'f29': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f29 with name 'Site'"}, 'f30': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f30 with name 'Class'"}, 'f33': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f33 with name 'Student Email'"}, 'f34': {'type': 'string', 'd
8:44
escription': "Suggested sample_record_key for the actual_key f34 with name 'Major'"}, 'f35': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f35 with name 'Scholarship Name'"}, 'f36': {'type': 'string', 'description': "Suggested sample_record_key for the actual_key f36 with name 'What are excited about'"}}}}], 'function_call': {'name': 'map_keys'}, 'temperature': 0, 'stream': False, 'max_tokens': 1000, 'max_retries': 0, 'extra_body': {}}, 'litellm_call_id': '72c026da-bf9f-4a0f-808f-5e5e445385ea', 'api_base': 'https://api.openai.com', 'model_id': '4c891f196c9d7bcb564a3ee990fc5e536cd1481ef91c53ab0b63dd827679548b', 'response_cost': 0.0003336, 'litellm_model_name': 'openai/gpt-4o-mini', 'litellm_overhead_time_ms': 2.298, '_response_ms': 4805.329000000001}}
 call_type: acompletion

[NiharP31]

Hey @krrishdholakia, is anyone working on this? If not I'd like to resolve this. Just need an idea of files related to this issue.

[krrishdholakia]

Sure. Go for it @NiharP31

Try to reproduce the issue first through the steps, and let me know if you face any issues.

[NiharP31]

I tried reproducing the issue. Is this good to proceed @krrishdholakia ?

Curl Command:

curl -Method POST `
  -Uri "http://localhost:4000/chat/completions" `
  -Headers @{
    "Content-Type" = "application/json"
    "Authorization" = "Bearer sk-1234"
  } `
  -Body '{
    "messages": [
      {
        "role": "user",
        "content": "My API key is sk-abcdef123456 and my password is SuperSecret123. Please analyze this document containing confidential business strategy."
      }
    ],
    "model": "gpt-4o-azure",
    "max_tokens": 2048
  }

Slack Issue:

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.