core icon indicating copy to clipboard operation
core copied to clipboard

Add Backblaze B2 integration for backups

Open frenck opened this issue 11 months ago • 10 comments

⚠️ This PR/integration isn't ready yet.

At this point, it works, it is using the SDK provided by Backblaze themselves, which is all super nice.

However, their library is sync-only, and built on top of requests. This makes it less ideal for our backup agent implementation. I do want to test this on some bigger backup sizes.

Additionally, there is a mypy error left, that I've not resolved. Dunno why, I'm probably overlooking something simple at this point.

🤗 Feel free to jump in and push improvements to this branch directly ❤️

Proposed change

This PR adds the first steps in adding an integration for Backblaze B2.

The integration provides a backup agent that works with the Home Assistant backup solution introduced in Home Assistant 2025.1.

CleanShot 2024-12-25 at 23 43 53@2x

CleanShot 2024-12-25 at 23 44 28@2x

CleanShot 2024-12-25 at 23 44 48@2x

CleanShot 2024-12-25 at 23 41 13@2x

CleanShot 2024-12-25 at 23 41 49@2x

CleanShot 2024-12-25 at 23 42 33@2x

Type of change

  • [ ] Dependency upgrade
  • [ ] Bugfix (non-breaking change which fixes an issue)
  • [x] New integration (thank you!)
  • [ ] New feature (which adds functionality to an existing integration)
  • [ ] Deprecation (breaking change to happen in the future)
  • [ ] Breaking change (fix/feature causing existing functionality to break)
  • [ ] Code quality improvements to existing code or addition of tests

Additional information

  • This PR fixes or closes issue: fixes #
  • This PR is related to issue:
  • Link to documentation pull request:

Checklist

  • [x] The code change is tested and works locally.
  • [x] Local tests pass. Your PR cannot be merged unless tests pass
  • [x] There is no commented out code in this PR.
  • [x] I have followed the development checklist
  • [x] I have followed the perfect PR recommendations
  • [x] The code has been formatted using Ruff (ruff format homeassistant tests)
  • [x] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

If the code communicates with devices, web services, or third-party tools:

  • [x] The manifest file has all fields filled out correctly.
    Updated and included derived files by running: python3 -m script.hassfest.
  • [x] New or updated dependencies have been added to requirements_all.txt.
    Updated by running python3 -m script.gen_requirements_all.
  • [x] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

frenck avatar Dec 25 '24 22:12 frenck

HI @frenck, I just took your code and replaced backblaze client with boto3 client to support any kind of S3 buckets. I tested it with Idrive e2, azure storage and minio. Is this something for the current beta - or for later? What do you think?

mkohns avatar Dec 27 '24 01:12 mkohns

Could you please also test to download a backup? Reproduction: -> select a backup -> click on the three dots at backblaze -> select "download from this location"

I am getting the error (in my S3 implementation)

TypeError: object async_generator can't be used in 'await' expression in line: "/workspaces/ha-core/homeassistant/components/backup/http.py", line 66, in get stream = await agent.async_download_backup(backup_id)

I can not test this with Backblaze as I do not have an account.

But this line here:

stream = await agent.async_download_backup(backup_id) (in http.py:66)

Seems to be wrong. Without the await - it works.

mkohns avatar Dec 27 '24 08:12 mkohns

B2 might already be covered by this integration

mbrevda avatar Jan 07 '25 10:01 mbrevda

B2 might already be covered by this integration

There are some limitations to be aware of. Otherwise "random" reports of a backup failure will pop up.

https://www.backblaze.com/docs/cloud-storage-s3-compatible-api

I'd say a fully B2 based solution is best, but that's my personal preference.

jeroenleenarts avatar Feb 10 '25 15:02 jeroenleenarts

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days. If you are the author of this PR, please leave a comment if you want to keep it open. Also, please rebase your PR onto the latest dev branch to ensure that it's up to date with the latest changes. Thank you for your contribution!

github-actions[bot] avatar Apr 11 '25 16:04 github-actions[bot]

Still relevant.

denysdovhan avatar Apr 14 '25 10:04 denysdovhan

Considering that the new Boto3 S3 integration has been made AWS S3 only, will this pull request get renewed attention? 🤞

jeroenleenarts avatar May 08 '25 07:05 jeroenleenarts

Also submitted a support ticket to Backblaze in hopes of them having a look, linking to this pull request.

jeroenleenarts avatar May 08 '25 07:05 jeroenleenarts

has been made AWS S3 only

where/how was this restricted to s3 only?

mbrevda avatar May 08 '25 09:05 mbrevda

@mbrevda It's stated in this pull request. https://github.com/home-assistant/core/pull/144318

jeroenleenarts avatar May 08 '25 10:05 jeroenleenarts

Set up a local dev container. And am running this pull request's branch.

Getting this console output:

2025-05-13 10:42:55.175 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved (None):   File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspaces/core/homeassistant/__main__.py", line 227, in <module>
    sys.exit(main())
  File "/workspaces/core/homeassistant/__main__.py", line 213, in main
    exit_code = runner.run(runtime_conf)
  File "/workspaces/core/homeassistant/runner.py", line 154, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 706, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 677, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 2026, in _run_once
    handle._run()
  File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
  File "/workspaces/core/homeassistant/components/backblaze/__init__.py", line 72, in _notify_backup_listeners
    listener()
  File "/workspaces/core/homeassistant/components/backup/manager.py", line 426, in listener
    self.hass.async_create_task(
  File "/workspaces/core/homeassistant/core.py", line 808, in async_create_task
    return self.async_create_task_internal(target, name, eager_start)
  File "/workspaces/core/homeassistant/core.py", line 830, in async_create_task_internal
    task = create_eager_task(target, name=name, loop=self.loop)
  File "/workspaces/core/homeassistant/util/async_.py", line 45, in create_eager_task
    return Task(coro, loop=loop, name=name, eager_start=True)
Traceback (most recent call last):
  File "/workspaces/core/homeassistant/components/backup/manager.py", line 448, in _async_reload_backup_agents
    agents = await platform.async_get_backup_agents(self.hass)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/backblaze/backup.py", line 29, in async_get_backup_agents
    return [BackblazeBackupAgent(hass, entry) for entry in entries]
            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/backblaze/backup.py", line 58, in __init__
    self._bucket = entry.runtime_data.bucket
                   ^^^^^^^^^^^^^^^^^^
AttributeError: 'ConfigEntry' object has no attribute 'runtime_data'
2025-05-13 10:42:55.178 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved (None):   File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/workspaces/core/homeassistant/__main__.py", line 227, in <module>
    sys.exit(main())
  File "/workspaces/core/homeassistant/__main__.py", line 213, in main
    exit_code = runner.run(runtime_conf)
  File "/workspaces/core/homeassistant/runner.py", line 154, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 706, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 677, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.13/asyncio/base_events.py", line 2026, in _run_once
    handle._run()
  File "/usr/local/lib/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
  File "/workspaces/core/homeassistant/components/backblaze/__init__.py", line 72, in _notify_backup_listeners
    listener()
  File "/workspaces/core/homeassistant/components/backup/manager.py", line 426, in listener
    self.hass.async_create_task(
  File "/workspaces/core/homeassistant/core.py", line 808, in async_create_task
    return self.async_create_task_internal(target, name, eager_start)
  File "/workspaces/core/homeassistant/core.py", line 830, in async_create_task_internal
    task = create_eager_task(target, name=name, loop=self.loop)
  File "/workspaces/core/homeassistant/util/async_.py", line 45, in create_eager_task
    return Task(coro, loop=loop, name=name, eager_start=True)
Traceback (most recent call last):
  File "/workspaces/core/homeassistant/components/backup/manager.py", line 449, in _async_reload_backup_agents
    self.backup_agents.update({agent.agent_id: agent for agent in agents})
                               ^^^^^^^^^^^^^^
  File "src/propcache/_helpers_c.pyx", line 82, in propcache._helpers_c.cached_property.__get__
  File "/workspaces/core/homeassistant/components/backup/agent.py", line 34, in agent_id
    return f"{self.domain}.{self.unique_id}"
                            ^^^^^^^^^^^^^^
AttributeError: 'BackblazeBackupAgent' object has no attribute 'unique_id'

Of note, the bucket was freshly created on my Backblaze account.

jeroenleenarts avatar May 13 '25 10:05 jeroenleenarts

I noticed the application key provided to the new integration requires full access, if I try to pass a key restricted to a single bucket I get an error logged in the console informing me of this fact. Would be great to provide this integration with minimal access.

jeroenleenarts avatar May 13 '25 10:05 jeroenleenarts