core Google Assistant integration suddenly broke with no input.

trafficstars

The problem

Had the integration working mostly flawlessly, as of a few days ago, all Home Assistant devices were unable to be reached. Attempted to go as far as unlinking and re-linking, now I cannot add the [test] application in Google Home. It appears, but after following the OAuth flow and signing into Home Assistant, if immediately fails with "Could not reach [test] App. Please try again." Completely starting over from square 1 with the integration still does not let me connect it. Updated HA to the latest version for good measure. No luck.

Would sincerely appreciate any pointers in the right direction, and please let me know what info I can provide! Thanks in advance!

What version of Home Assistant Core has the issue?

core-2024.10.0

What was the last working version of Home Assistant Core?

core-2024.9.0

What type of installation are you running?

Home Assistant OS

Integration causing the issue

google_assistant

Link to integration documentation on our website

https://www.home-assistant.io/integrations/google_assistant

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

Oct 04 '24 04:10 ghxstxch

Hey there @home-assistant/cloud, mind taking a look at this issue as it has been labeled with an integration (google_assistant) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of google_assistant can trigger bot actions by commenting:

@home-assistant close Closes the issue.
@home-assistant rename Awesome new title Renames the issue.
@home-assistant reopen Reopen the issue.
@home-assistant unassign google_assistant Removes the current integration label and assignees on the issue, add the integration domain after the command.
@home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
@home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

google_assistant documentation google_assistant source _{^{(message by IssueLinks)}}

Oct 04 '24 04:10 home-assistant[bot]

I also have the same issue. Try to deleting and creating a new service account and key, but still doesn't help

Oct 04 '24 05:10 gondlong

I'm not sure if it is related to the above issues but I'm also experiencing the assistant answering: Home Assistant unreachable.

I've given a look to the Google API logs and I could find this error:

{
    "insertId": "aljz4kf16cpv7",
    "jsonPayload": {
      "locale": "it-IT",
      "executionLog": {
        "executionResults": [
          {
            "executionType": "PARTNER_CLOUD",
            "actionResults": [
              {
                "status": {
                  "statusType": "EXECUTION_GAL_BAD_3P_RESPONSE",
                  "externalDebugString": "Failed to add authorization token to the HTTP header.",
                  "isSuccess": false
                },
                "action": {
                  "trait": "TRAIT_ON_OFF",
                  "actionType": "ONOFF_OFF"
                },
                "device": {
                  "deviceType": "LIGHT"
                }
              }
            ],
            "requestId": "8300907704095611035",
            "latencyMsec": "536"
          }
        ]
      }
    },
    "resource": {
      "type": "assistant_action_project",
      "labels": {
        "project_id": "myproject"
      }
    },
    "timestamp": "2024-10-03T21:08:18.488217683Z",
    "severity": "ERROR",
    "logName": "projects/myproject/logs/assistant_smarthome%2Fassistant_smarthome_logs",
    "receiveTimestamp": "2024-10-03T21:08:18.488217683Z"
  }

For the others experiencing the same issue please verify if your logs here:

https://console.cloud.google.com/logs/query;query=resource.type%3D%22assistant_action_project%22;cursorTimestamp=2024-10-04T07:19:05.406Z;duration=PT1H?project=[your project name]

It has something to do with REFRESH_ACCESS_TOKEN but I cannot find any expired IAM credentials or other in my google console. The Home Assistant access token is still valid.

I have not tried to redo the config procedure based on the failed attempt of the other users before.

In my case I'm still in Home Assistant version 2024.9.2.

Oct 04 '24 08:10 adizanni

I believe I'm experiencing the same issue so I completely rebuilt my google assistant integration and now receive these two errors when attempting the final Step 6 through the Google Home app:

{ insertId: "4h5v4uar" jsonPayload: { @type: "type.googleapis.com/google.identity.accountlinking.type.AccountLinkingError" errorReason: "Can't parse the response. The response needs to be JSON format." response: {2} step: "AUTH_CODE_EXCHANGE" } logName: "projects/home-58926/logs/accountlinking-pa.googleapis.com%2Ferror" receiveTimestamp: "2024-10-04T03:41:54.621902516Z" resource: {2} severity: "ERROR" timestamp: "2024-10-04T03:41:54.526Z" }

as well as

{ insertId: "-k8zytqav" jsonPayload: { @type: "type.googleapis.com/google.identity.accountlinking.type.AccountLinkingError" request: {4} step: "AUTH_CODE_EXCHANGE" } logName: "projects/home-58926/logs/accountlinking-pa.googleapis.com%2Ferror" receiveTimestamp: "2024-10-04T03:51:31.756765940Z" resource: {2} severity: "ERROR" timestamp: "2024-10-04T03:51:30.807Z" }

Just tried with latest dev container and received same error messages.

Oct 04 '24 12:10 idontcare99999

I'm also having same issue.

{
  "insertId": "-gvg47db7i",
  "jsonPayload": {
    "sessionId": 1633426304,
    "errorReason": "Failed validating the response: Invalid JSON response: <!DOCTYPE html><html lang=\"en-US\"><head><title>Just a moment...</title><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"><meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\"><meta name=\"robots\" content=\"noindex,nofollow\"><meta 
  "timestamp": "2024-10-04T13:02:57.193Z",
  "severity": "ERROR",
}

Oct 04 '24 13:10 mang0db

@idontcare99999, even if your error is different (as you receive it in a brand new config) but similar and there are no traces of errors in Home Assistant log, this means that the error happens upstream.

We can say with confidence that the google assistant integration is broken; probably a change in the google OAUTH authentication/authorization flow.

This is happening since around 2 days ago.

Oct 04 '24 13:10 adizanni

I've found only this reference to changes happened recently in Google but I'm not sure this is related, maybe our HA integration problem is a colllateral impact of this change.

Oct 04 '24 13:10 adizanni

I'm having the same problem and basically retraced the setup incase of any changes and now when i try to re-add it to google home I get this error in HA

ERROR (MainThread) [homeassistant.components.google_assistant.http] Request for https://homegraph.googleapis.com/v1/devices:reportStateAndNotification failed: 404

Oct 04 '24 14:10 errordlien

Same problem. Nothing with google Assistant is working anymore since 1 or 2 days. When I try to sync my devices in HA, I get the following:

2024-10-04 16:47:05.038 DEBUG (MainThread) [homeassistant.components.google_assistant.http] Response on https://homegraph.googleapis.com/v1/devices:requestSync with data {'agentUserId': 'eea098214540468d8f229243c4a4034d'} was {
  "error": {
    "code": 500,
    "message": "Internal error encountered.",
    "status": "INTERNAL"
  }
}

2024-10-04 16:47:05.039 ERROR (MainThread) [homeassistant.components.google_assistant.http] Request for https://homegraph.googleapis.com/v1/devices:requestSync failed: 500
2024-10-04 16:47:05.039 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140367566659520] Unable to sync devices with result code: 500, check log for more info.
Traceback (most recent call last):
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/components/websocket_api/commands.py", line 226, in handle_call_service
    await hass.services.async_call(
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/core.py", line 2012, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/core.py", line 2049, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/helpers/entity_component.py", line 235, in handle_service
    return await service.entity_service_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/helpers/service.py", line 876, in entity_service_call
    response_data = await _handle_entity_call(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/helpers/service.py", line 948, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/components/button/__init__.py", line 124, in _async_press_action
    await self.async_press()
  File "/lsiopy/lib/python3.11/site-packages/homeassistant/components/google_assistant/button.py", line 57, in async_press
    raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Unable to sync devices with result code: 500, check log for more info.

Oct 04 '24 14:10 brintal

Are other folks with this problem a) using Cloudflare and b) using Bot fight mode? I checked my Cloudflare firewall logs and saw a lot of requests to my HASS url from Google were getting served CAPTCHA challenges by Bot fight mode (even though I had a custom rule set up to allow requests from Google's ASN, which seems to be working flawlessly for other subdomains I host).

I disabled Bot fight mode in Cloudflare and re-synced my Home Assistant devices in Google Assistant, and everything is suddenly working again.

Oct 04 '24 17:10 carjed

I disabled Bot fight mode in Cloudflare and re-synced my Home Assistant devices in Google Assistant, and everything is suddenly working again.

Fixed my issue thank you!!

Oct 04 '24 19:10 alec-pinson

Are other folks with this problem a) using Cloudflare and b) using Bot fight mode? I checked my Cloudflare firewall logs and saw a lot of requests to my HASS url from Google were getting served CAPTCHA challenges by Bot fight mode (even though I had a custom rule set up to allow requests from Google's ASN, which seems to be working flawlessly for other subdomains I host).

I disabled Bot fight mode in Cloudflare and re-synced my Home Assistant devices in Google Assistant, and everything is suddenly working again.

My situation is the same as yours - Google was allowed via a rule but it was doing the same thing. Disabling BFM allowed me to re-sync as well. Thank you so much!

Oct 04 '24 20:10 ghxstxch

Same here. Thanks a lot.

In the meantime it would be good to dig a little, I'm not comfortable with the bot fight wide open.

Oct 04 '24 20:10 adizanni

Works for me as well...but would like to see a resolution that includes the bot...cheers to the workaround

Oct 04 '24 20:10 kwkid

https://console.cloud.google.com/logs/query;query=resource.type%3D%22assistant_action_project%22;cursorTimestamp=2024-10-04T07:19:05.406Z;duration=PT1H?project=

Nice find! this was my problem also. Like others now we need to understand why the allow rules were ignored.

Oct 04 '24 21:10 synth1234

Hello guys, I need a little time to confirm, but based on this article, the custome rules are not in effect for free Bot Fight Mode.

If you add this IP address rule:

and you activate bot fight mode, it works.

We still have all google cloud IPs open for bots but it is better than nothing

Oct 04 '24 21:10 adizanni

Hello guys, I need a little time to confirm, but based on this article, the custome rules are not in effect for free Bot Fight Mode.

If you add this IP address rule:

and you activate bot fight mode, it works.

We still have all google cloud IPs open but it is better than nothing

That works for me! Thank you :)

Oct 04 '24 21:10 Lansenou

Turning off bot fight mode on CF isn't really a viable solution.
We ideally need to be looking into what IPs specifically need to be listed, as the BF function is a good security measure to have in place.
For example, in my CF logs, i see the following with Google ASNs -

108.177.68.38
108.177.68.39
108.177.68.40
108.177.68.43
108.177.68.44
108.177.68.45

If we were to assume 16x overall IPs in said source range, we would assume a whitelist of 108.177.68.32/28, or x32 IPs, of 108.177.68.32/27.

For example -

Securit _WAF_Custom_rules

Oct 04 '24 21:10 boomam

@boomam have you tried my solution above ? IP Address Rule with Google ASN + Bot Fight Mode activated works well. Whitelisting Google Cloud source IP is not a stable solution, those guys do all type of regional load balancing, there are a lot of IP addresses involved.

Custom rules are not working with free Bot Fight Mode, you need to use IP Address Rules, and it is better to use Google ASN AS15169. It is a large IP range but it is stable

Oct 04 '24 21:10 adizanni

@boomam have you tried my solution above ? IP Address Rule with Google ASN + Bot Fight Mode activated works well. Whitelisting Google Cloud source IP is not a stable solution, those guys do all type of regional load balancing, there are a lot of IP addresses involved

I have not, as i look at it from the other direction - i would rather be prescriptive in what is whitelisted, instead of 'all of GCP'.

Whilst you're correct they have a lot of IPs, they don't have a lot of IPs (comparatively) that does this specific function within a given CIDR range on GCP.
The regional load balancing in this scenario only really comes into play when there's either a massive outage in a region, or you are traveling with your HA install.

For others reference - Complete list of GCP IPs

Oct 04 '24 21:10 boomam

Up to you. Based on my experience with Cloud services, if you do not have a tagging mechanism (like AS), you better prepare yourself with unexpected blocked IPs, and google assistant not reaching your HA

Oct 04 '24 22:10 adizanni

Up to you. Based on my experience with Cloud services, if you do not have a tagging mechanism (like AS), you better prepare yourself with unexpected blocked IPs, and google assistant not reaching your HA

I'm fine with doing this - i work in that particular industry, so i know what to look out for ;-)

To be clear, i do appreciate the views and conversation around it though, always good to talk things through. :-)

Oct 04 '24 22:10 boomam

Me too I like this thread, you learn a lot from these troubleshootings (a lot more than when you are lucky and everything works fine).

The important note is that Custom Rules is not working with free Bot Fight Mode, use instead IP Access Rules with the IP range you are more comfortable with.

Oct 04 '24 22:10 adizanni

I've done a little combination of what everyone has proposed for a more targeted approached as I have several sites. I also have a custom rule to block bots rather than using the bot fight mode.

Oct 04 '24 22:10 errordlien

I'm back up. Thank you all!

Oct 04 '24 23:10 idontcare99999

Me too I like this thread, you learn a lot from these troubleshootings (a lot more than when you are lucky and everything works fine).

The important note is that Custom Rules is not working with free Bot Fight Mode, use instead IP Access Rules with the IP range you are more comfortable with.

listing on IP Access Rules and Bot Fight Mode activated work for me. thanks

Oct 05 '24 04:10 gondlong

thank you guys i had this issue aswell whitelist google fixed everything yah!

Oct 05 '24 07:10 slaygirlz

thank you guys! this also helped me

Oct 06 '24 01:10 OttoUsualConsumer

Hello guys, I need a little time to confirm, but based on this article, the custome rules are not in effect for free Bot Fight Mode.

If you add this IP address rule:

and you activate bot fight mode, it works.

We still have all google cloud IPs open for bots but it is better than nothing