webdav4
webdav4 copied to clipboard
Published
20 hours ago •
skshetry
skshetry
Several test failures
I’m trying to package release 0.9.8 for Fedora Linux, and I’m encountering several test failures, all of which I am able to reproduce in a plain virtualenv with current main, 4c2046e2250f001bdad76541c0e877e4b40c332e.
$ python3 -V
Python 3.12.2
$ python3 -m venv _e
$ . _e/bin/activate
(_e) $ pip install -e '.[tests,all]'
(_e) $ python -m pytest -v
============================================== short test summary info ===============================================
SKIPPED [1] tests/test_fsspec.py:849: the test server does not work without content-length :(
FAILED tests/test_cli.py::test_sync_cli_local_to_remote - AssertionError: assert {'/data/./bar.../foo': b'foo'} == {'/data/bar':.../foo': b'foo'}
FAILED tests/test_cli.py::test_sync_remote_to_local - AssertionError: assert {'data': {'da...foo': 'foo'}}} == {'data': {'foo': 'foo'}}
FAILED tests/test_cli.py::test_cp_cli - AssertionError: assert False
FAILED tests/test_cli.py::test_mv_cli - AssertionError: assert {'dir': {'ips...m': 'lorem'}}} == {'dir': {'ips...em': 'lorem'}}
FAILED tests/test_cli.py::test_sync_remote_to_remote - FileNotFoundError: /data/.
FAILED tests/test_cli.py::test_ls_cli - AssertionError: assert {Row(date='-'... isdir=False)} == {Row(date='-'... isdir=False)}
============================== 6 failed, 330 passed, 1 skipped, 2621 warnings in 9.60s ===============================
I will post the full output from pytest -vv for these six failures in a comment.
I haven’t tried to dig into the root causes. Is there something that needs to be updated for current dependency versions?
====================================================== FAILURES ======================================================
____________________________________________________ test_cp_cli _____________________________________________________
[gw9] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python
storage_dir = TmpDir('/tmp/pytest-of-ben/pytest-3/popen-gw9/webdav0')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fad7e42e810>
def test_cp_cli(storage_dir: TmpDir, monkeypatch: MonkeyPatch):
"""Test cp command."""
memfs = MemoryFileSystem()
memfs.mkdir("data1")
memfs.pipe({"data1/foo": b"foo", "data1/bar": b"bar"})
ns = Namespace(
path1="memory://data1", path2="memory://data2", recursive=False
)
CommandCopy(ns, memfs).run()
assert memfs.cat("data1", recursive=True, on_error="ignore") == {
"/data1/bar": b"bar",
"/data1/foo": b"foo",
}
> assert memfs.isdir("data2")
E AssertionError: assert False
E + where False = <bound method AbstractFileSystem.isdir of <tests.test_cli.MemoryFileSystem object at 0x7fad7e42e7e0>>('data2')
E + where <bound method AbstractFileSystem.isdir of <tests.test_cli.MemoryFileSystem object at 0x7fad7e42e7e0>> = <tests.test_cli.MemoryFileSystem object at 0x7fad7e42e7e0>.isdir
tests/test_cli.py:306: AssertionError
___________________________________________ test_sync_cli_local_to_remote ____________________________________________
[gw6] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python
storage_dir = TmpDir('/tmp/pytest-of-ben/pytest-3/popen-gw6/webdav0')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5dd695a9f0>
def test_sync_cli_local_to_remote(
storage_dir: TmpDir, monkeypatch: MonkeyPatch
):
"""Test syncing between local to remote filesystem."""
storage_dir.gen({"data": {"foo": "foo"}})
memfs = MemoryFileSystem()
monkeypatch.chdir(storage_dir)
ns = Namespace(path1="data", path2="memory://data", delete=False)
cmd = CommandSync(ns, memfs)
cmd.run()
assert memfs.cat("data", True, "ignore") == {"/data/foo": b"foo"}
(storage_dir / "data" / "bar").write_text("bar")
cmd.run()
> assert memfs.cat("data", True, "ignore") == {
"/data/foo": b"foo",
"/data/bar": b"bar",
}
E AssertionError: assert {'/data/./bar': b'bar', '/data/./foo': b'foo', '/data/bar': b'bar', '/data/foo': b'foo'} == {'/data/foo': b'foo', '/data/bar': b'bar'}
E
E Common items:
E {'/data/bar': b'bar', '/data/foo': b'foo'}
E Left contains 2 more items:
E {'/data/./bar': b'bar', '/data/./foo': b'foo'}
E
E Full diff:
E {
E + '/data/./bar': b'bar',
E + '/data/./foo': b'foo',
E '/data/bar': b'bar',
E '/data/foo': b'foo',
E }
/home/ben/src/forks/webdav4/tests/test_cli.py:537: AssertionError
------------------------------------------------ Captured stdout call ------------------------------------------------
copy: /tmp/pytest-of-ben/pytest-3/popen-gw6/webdav0/data to /data
copy: /tmp/pytest-of-ben/pytest-3/popen-gw6/webdav0/data/. to /data/.
copy: /tmp/pytest-of-ben/pytest-3/popen-gw6/webdav0/data/bar to /data/bar
_____________________________________________ test_sync_remote_to_local ______________________________________________
[gw6] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python
storage_dir = TmpDir('/tmp/pytest-of-ben/pytest-3/popen-gw6/webdav1')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f5dd67ad2b0>
def test_sync_remote_to_local(storage_dir: TmpDir, monkeypatch: MonkeyPatch):
"""Test syncing between remote to local filesystem."""
memfs = MemoryFileSystem()
memfs.mkdir("data")
memfs.pipe({"data/foo": b"foo"})
monkeypatch.chdir(storage_dir)
ns = Namespace(path1="memory://data", path2="data", delete=False)
cmd = CommandSync(ns, memfs)
cmd.run()
> assert storage_dir.cat() == {"data": {"foo": "foo"}}
E AssertionError: assert {'data': {'data': {'foo': 'foo'}}} == {'data': {'foo': 'foo'}}
E
E Differing items:
E {'data': {'data': {'foo': 'foo'}}} != {'data': {'foo': 'foo'}}
E
E Full diff:
E {
E 'data': {
E + 'data': {
E - 'foo': 'foo',
E + 'foo': 'foo',
E ? ++++
E + },
E },
E }
/home/ben/src/forks/webdav4/tests/test_cli.py:591: AssertionError
------------------------------------------------ Captured stdout call ------------------------------------------------
copy: /data to /tmp/pytest-of-ben/pytest-3/popen-gw6/webdav1/data
____________________________________________________ test_mv_cli _____________________________________________________
[gw9] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python
storage_dir = TmpDir('/tmp/pytest-of-ben/pytest-3/popen-gw9/webdav1')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fad7e42e570>
def test_mv_cli(storage_dir: TmpDir, monkeypatch: MonkeyPatch):
"""Test mv command."""
memfs = MemoryFileSystem()
memfs.mkdir("data1")
memfs.pipe({"data1/foo": b"foo", "data1/bar": b"bar"})
ns = Namespace(
path1="memory://data1", path2="memory://data2", recursive=True
)
CommandMove(ns, memfs).run()
assert not memfs.isdir("data1")
assert memfs.cat("data2", recursive=True, on_error="ignore") == {
"/data2/bar": b"bar",
"/data2/foo": b"foo",
}
assert memfs.isdir("data2")
ns = Namespace(
path1="memory://data2/foo",
path2="memory://data2/foobar",
recursive=True,
)
CommandMove(ns, memfs).run()
assert memfs.cat_file("data2/foobar") == b"foo"
assert not memfs.exists("memory://data2/foo")
monkeypatch.chdir(storage_dir)
d = {"lorem": "lorem", "ipsum": "ipsum"}
storage_dir.gen({"dir": d})
ns = Namespace(path1="dir", path2="memory://dir", recursive=True)
CommandMove(ns, memfs).run()
assert memfs.cat("dir", True, "ignore") == {
"/dir/ipsum": b"ipsum",
"/dir/lorem": b"lorem",
}
ns = Namespace(path1="memory://dir", path2="dir2", recursive=True)
CommandMove(ns, memfs).run()
> assert storage_dir.cat() == {"dir": d, "dir2": d}
E AssertionError: assert {'dir': {'lorem': 'lorem', 'ipsum': 'ipsum'}, 'dir2': {'dir': {'ipsum': 'ipsum', 'lorem': 'lorem'}}} == {'dir': {'lorem': 'lorem', 'ipsum': 'ipsum'}, 'dir2': {'lorem': 'lorem', 'ipsum': 'ipsum'}}
E
E Common items:
E {'dir': {'ipsum': 'ipsum', 'lorem': 'lorem'}}
E Differing items:
E {'dir2': {'dir': {'ipsum': 'ipsum', 'lorem': 'lorem'}}} != {'dir2': {'ipsum': 'ipsum', 'lorem': 'lorem'}}
E
E Full diff:
E {
E 'dir': {
E 'ipsum': 'ipsum',
E 'lorem': 'lorem',
E },
E 'dir2': {
E + 'dir': {
E - 'ipsum': 'ipsum',
E + 'ipsum': 'ipsum',
E ? ++++
E - 'lorem': 'lorem',
E + 'lorem': 'lorem',
E ? ++++
E + },
E },
E }
/home/ben/src/forks/webdav4/tests/test_cli.py:427: AssertionError
____________________________________________________ test_ls_cli _____________________________________________________
[gw9] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python
capsys = <_pytest.capture.CaptureFixture object at 0x7fad7e499910>
def test_ls_cli(capsys: CaptureFixture):
"""Test ls command."""
mfs = MemoryFileSystem()
mfs.mkdir("data")
mfs.mkdir("dir")
mfs.pipe(
{
"data/foo": b"foo",
"data/bar": b"bar",
"dir/nested/foo": b"dirfoo" * 1000,
}
)
ns = Namespace(path="data", recursive=False, level=None, full_path=False)
> assert set(CommandLS(ns, mfs).ls()) == {
Row("-", Size("3"), "foo"),
Row("-", Size("3"), "bar"),
}
E AssertionError: assert {Row(date='-', size=Size(nbytes='0', suff=''), file='data', isdir=True), Row(date='-', size=Size(nbytes='3', suff=''), file='foo', isdir=False), Row(date='-', size=Size(nbytes='3', suff=''), file='bar', isdir=False)} == {Row(date='-', size=Size(nbytes='3', suff=''), file='bar', isdir=False), Row(date='-', size=Size(nbytes='3', suff=''), file='foo', isdir=False)}
E
E Extra items in the left set:
E Row(date='-', size=Size(nbytes='0', suff=''), file='data', isdir=True)
E
E Full diff:
E {
E + Row(date='-', size=Size(nbytes='0', suff=''), file='data', isdir=True),
E Row(date='-', size=Size(nbytes='3', suff=''), file='bar', isdir=False),
E Row(date='-', size=Size(nbytes='3', suff=''), file='foo', isdir=False),
E }
tests/test_cli.py:445: AssertionError
_____________________________________________ test_sync_remote_to_remote _____________________________________________
[gw6] linux -- Python 3.12.2 /home/ben/src/forks/webdav4/_e/bin/python
def test_sync_remote_to_remote():
"""Test syncing between remote to remote filesystem."""
memfs = MemoryFileSystem()
memfs.mkdir("data")
memfs.pipe({"data/foo": b"foo"})
ns = Namespace(path1="memory://data", path2="memory://data2", delete=False)
cmd = CommandSync(ns, memfs)
cmd.run()
assert memfs.cat("data2", True, "ignore") == {"/data2/foo": b"foo"}
memfs.pipe({"data/bar": b"bar"})
> cmd.run()
tests/test_cli.py:635:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/webdav4/cli.py:906: in run
return self.sync(src, dest, src_fs, dest_fs)
src/webdav4/cli.py:879: in sync
self.sync(new_src, new_dest, src_fs, dest_fs)
src/webdav4/cli.py:848: in sync
details_src = src_fs.info(src)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.test_cli.MemoryFileSystem object at 0x7f5dd67addc0>, path = '/data/.', kwargs = {}
def info(self, path, **kwargs):
path = self._strip_protocol(path)
if path in self.pseudo_dirs or any(
p.startswith(path + "/") for p in list(self.store) + self.pseudo_dirs
):
return {
"name": path,
"size": 0,
"type": "directory",
}
elif path in self.store:
filelike = self.store[path]
return {
"name": path,
"size": filelike.size,
"type": "file",
"created": getattr(filelike, "created", None),
}
else:
> raise FileNotFoundError(path)
E FileNotFoundError: /data/.
_e/lib64/python3.12/site-packages/fsspec/implementations/memory.py:159: FileNotFoundError
------------------------------------------------ Captured stdout call ------------------------------------------------
copy: /data to /data2
There were some changes in MemoryFileSystem in fsspec that are breaking cli tests.
I'll try to look at it over the weekend. Thank you for creating this issue.
I was looking back over the Snakemake 8 plugins that I haven’t yet packaged for Fedora, and noticed I still need webdav4 in order to package https://github.com/snakemake/snakemake-storage-plugin-webdav.
Did you happen to have any luck looking into the fsspec changes?
I suppose that I could package the webdav4 library without the dav command-line tool for now, since it looks like any issues are contained to the CLI…
Hi Ben, did you submit a review request? I'm interested in a Fedora package as well - though the cli might important for some of my use cases.
Hi Ben, did you submit a review request? I'm interested in a Fedora package as well - though the cli might important for some of my use cases.
I was going to submit one without the CLI (since I am not comfortable supporting a package that seems like it might not work correctly). It hadn’t bubbled up to the top of my list, but since it might be useful to you, I can double-check my spec file and submit it within the next few days.
It hadn’t bubbled up to the top of my list, but since it might be useful to you, I can double-check my spec file and submit it within the next few days.
Thanks - but you don't have to rush it. We are currently installing webdav4 with pip but a Fedora/EPEL rpm would be nice of course. Feel free to cc me in the review request, I can to review it if you like (FAS: fschwarz).