dvc icon indicating copy to clipboard operation
dvc copied to clipboard

Published 20 hours ago • verified publisher icon iterative

Push failed to OSS

Open Mickychen00 opened this issue 1 year ago • 3 comments

Bug Report

Issue name

failed to push to oss

Description

When I use dvc push to oss. Something went wrong. Here is the log: ending a large body directly with raw bytes might lock the event loop. You should probably pass an io.BytesIO object instead.
dvc.exceptions.UploadError: 1 files failed to upload. But the latest updated file's size is just 3Mb.

Mickychen00 avatar Dec 29 '23 20:12 Mickychen00

Requested full logs in discord (context: https://discord.com/channels/485586884165107732/563406153334128681/1190315522462580826)

pmrowla avatar Dec 30 '23 01:12 pmrowla

The entire log is shown below:

dvc push -v                                                   3.9.12 08:47:23
2023-12-30 08:51:29,395 DEBUG: v3.35.0 (brew), CPython 3.11.6 on macOS-13.2.1-arm64-arm-64bit
2023-12-30 08:51:29,395 DEBUG: command: /opt/homebrew/bin/dvc push -v
2023-12-30 08:51:29,570 DEBUG: Preparing to transfer data from '/Users/chen/Dataset/dataset-critical/.dvc/cache/files/md5' to 'oss://chenchen-hz/dvc/dataset-critical/files/md5'
2023-12-30 08:51:29,570 DEBUG: Preparing to collect status from 'chenchen-hz/dvc/dataset-critical/files/md5'
2023-12-30 08:51:29,570 DEBUG: Collecting status from 'chenchen-hz/dvc/dataset-critical/files/md5'
2023-12-30 08:51:30,559 DEBUG: Estimated remote size: 4096 files
2023-12-30 08:51:30,561 DEBUG: Large remote (10 oids < 40.96 traverse weight), using object_exists for remaining oids
2023-12-30 08:51:30,562 DEBUG: Querying 10 oids via object_exists
2023-12-30 08:51:31,701 DEBUG: Preparing to collect status from '/Users/chen/Dataset/dataset-critical/.dvc/cache/files/md5'
2023-12-30 08:51:31,703 DEBUG: Collecting status from '/Users/chen/Dataset/dataset-critical/.dvc/cache/files/md5'
Pushing                                                                                            |0.00 [00:02,     ?file/sSending a large body directly with raw bytes might lock the event loop. You should probably pass an io.BytesIO object instead.
2023-12-30 08:52:32,630 ERROR: failed to transfer 'd95159fba4614a3712779319af3e9d55' - :
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/ossfs/async_oss.py", line 396, in _put_file
    await self._call_oss(
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/ossfs/async_oss.py", line 162, in _call_oss
    out = await method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/aiooss2/api.py", line 581, in put_object_from_file
    return await self.put_object(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/aiooss2/api.py", line 291, in put_object
    resp: "AioResponse" = await self._do_object(
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/aiooss2/api.py", line 254, in _do_object
    return await self._do(method, self.bucket_name, key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/aiooss2/api.py", line 144, in _do
    resp: "AioResponse" = await self.session.do_request(req, timeout=self.timeout)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/aiooss2/http.py", line 150, in do_request
    resp = await self.session.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/aiohttp/client.py", line 601, in _request
    await resp.start(conn)
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 960, in start
    with self._timer:
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/aiohttp/helpers.py", line 735, in __exit__
    raise asyncio.TimeoutError from None
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc_objects/fs/generic.py", line 330, in transfer
    _try_links(
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc_objects/fs/generic.py", line 262, in _try_links
    return copy(
           ^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc_objects/fs/generic.py", line 87, in copy
    return _put(
           ^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc_objects/fs/generic.py", line 156, in _put
    return _put_one(from_paths[0], to_paths[0])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc_objects/fs/generic.py", line 148, in _put_one
    return put_file(from_path, to_path, callback=callback, **put_file_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc_objects/fs/callbacks.py", line 204, in func
    res = fn(path1, path2, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc_objects/fs/base.py", line 645, in put_file
    self.fs.put_file(os.fspath(from_file), to_info, callback=callback, **kwargs)
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/fsspec/asyn.py", line 101, in sync
    raise FSTimeoutError from return_result
fsspec.exceptions.FSTimeoutError

2023-12-30 08:52:39,782 ERROR: failed to push data to the cloud - 1 files failed to upload
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc/commands/data_sync.py", line 64, in run
    processed_files_count = self.repo.push(
                            ^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/dvc/3.35.0/libexec/lib/python3.11/site-packages/dvc/repo/push.py", line 146, in push
    raise UploadError(failed_count)
dvc.exceptions.UploadError: 1 files failed to upload

2023-12-30 08:52:39,798 DEBUG: Analytics is enabled.
2023-12-30 08:52:39,872 DEBUG: Trying to spawn ['daemon', 'analytics', '/var/folders/qm/p6yv8w616nl2mxr1rrqfj24r0000gn/T/tmp4i4imxhr', '-v']
2023-12-30 08:52:39,877 DEBUG: Spawned ['daemon', 'analytics', '/var/folders/qm/p6yv8w616nl2mxr1rrqfj24r0000gn/T/tmp4i4imxhr', '-v'] with pid 19627

Mickychen00 avatar Dec 30 '23 07:12 Mickychen00

Discussed with @pmrowla that this looks like a simple timeout issue so far. We expose timeout config option in other clouds like https://github.com/iterative/dvc-s3/blob/43f70226160f1a5c9ffdad4092d41a8bab7ec19b/dvc_s3/init.py#L176 , but not in oss. So one would likely need to expose it a similar way in https://github.com/iterative/dvc-oss Contributions are welcome.

efiop avatar Jan 02 '24 17:01 efiop