testcontainers-python icon indicating copy to clipboard operation
testcontainers-python copied to clipboard

feat: add copying files to/from container

Open mgorsk1 opened this issue 1 year ago • 13 comments

solves https://github.com/testcontainers/testcontainers-python/issues/665

mgorsk1 avatar Aug 12 '24 16:08 mgorsk1

i usually prefer using typing.Union[str, os.PathLike] - if you wanna use strings with this as well as paths

alexanderankin avatar Aug 12 '24 16:08 alexanderankin

Codecov Report

Attention: Patch coverage is 32.00000% with 17 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@1e43923). Learn more about missing BASE report.

Files Patch % Lines
core/testcontainers/core/container.py 32.00% 16 Missing and 1 partial :warning:
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #676   +/-   ##
=======================================
  Coverage        ?   76.44%           
=======================================
  Files           ?       12           
  Lines           ?      624           
  Branches        ?       96           
=======================================
  Hits            ?      477           
  Misses          ?      120           
  Partials        ?       27           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Aug 12 '24 16:08 codecov[bot]

one other thing to consider is that there is this great interface "Transferable" in other language implementations-- does this PR make it harder or easier to add this to this implementation as well.

the interface (found here) is basically:

class Transferable(abc.ABC):
  @abstractmethod
  def transfer_to(content_stream, destination: typing.Union[str, os.PathLike]):
    ...

where content_stream is some bytesio or something pythonic for a place where you can write bytes to.

alexanderankin avatar Aug 12 '24 16:08 alexanderankin

when i tried to do it, i accidentally interpreted the interface a bit too widely and instead of passing just the content output stream, i passed the whole container as that first argument.

https://github.com/testcontainers/testcontainers-python/commit/4b7db74176c0f00ca275bc39409ebb891a93cbb9 - misc functions, next commit doesnt make sense without reading first

https://github.com/testcontainers/testcontainers-python/commit/bbc44ec8e6d0ce35d40466f1d636106d75e0b197 - the commit where i add the transferable stuff

alexanderankin avatar Aug 12 '24 16:08 alexanderankin

so im tiny bit hesitant to merge without a plan for adding transferable on top of this

alexanderankin avatar Aug 12 '24 16:08 alexanderankin

do we need it to be 1:1 with java implementation though? any binary object now can be passed here using tempfile.NamedTemporaryObject:

with tempfile.NamedTemporaryObject() as f:

    f.write(binary_data)
    DockerContainer("nginx").with_copy_file_to_container(f.name, "/tmp/target.data").start()

mgorsk1 avatar Aug 12 '24 17:08 mgorsk1

we dont need to be 1-1 but what if instead of a file you had a string? i think it is beneficial to not force people to create temporary files.

alexanderankin avatar Aug 12 '24 18:08 alexanderankin

refactored for Transferable although creating temp files cannot be avoided, now instead of asking users for 3 extra lines we add some complexity to codebase here. working example

input:

from pathlib import Path

from testcontainers.core.container import DockerContainer, Transferable
from testcontainers.core.network import Network

c = DockerContainer('nginx')

network = Network()

c \
    .with_copy_file_to_container(Transferable(b'asd', Path('/tmp/mariusz.md1'))) \
    .with_copy_file_to_container(Transferable(Path('/Users/mariusz/Documents/Programming/testcontainers-python/README.md'),Path('/tmp/mariusz.md2'))) \
    .start()

c2 = DockerContainer('nginx')

print(c._container.exec_run('head /tmp/mariusz.md1'))
print(c._container.exec_run('head /tmp/mariusz.md2'))

output

ExecResult(exit_code=0, output=b'asd')
ExecResult(exit_code=0, output=b'[![Poetry](https://img.shields.io/endpoint?url=https://python-poetry.org/badge/v0.json)](https://python-poetry.org/)\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n![PyPI - Version](https://img.shields.io/pypi/v/testcontainers)\n[![PyPI - License](https://img.shields.io/pypi/l/testcontainers.svg)](https://github.com/testcontainers/testcontainers-python/blob/main/LICENSE)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/testcontainers.svg)](https://pypi.python.org/pypi/testcontainers)\n[![codecov](https://codecov.io/gh/testcontainers/testcontainers-python/branch/master/graph/badge.svg)](https://codecov.io/gh/testcontainers/testcontainers-python)\n![Core Tests](https://github.com/testcontainers/testcontainers-python/actions/workflows/ci-core.yml/badge.svg)\n![Community Tests](https://github.com/testcontainers/testcontainers-python/actions/workflows/ci-community.yml/badge.svg)\n[![Docs](https://readthedocs.org/projects/testcontainers-python/badge/?version=latest)](http://testcontainers-python.readthedocs.io/en/latest/?badge=latest)\n\n')

mgorsk1 avatar Aug 12 '24 20:08 mgorsk1

@mgorsk1 I have opened #677 to solve the same issue (I did not see you already had opened one 😓). Yours have the Transferable feature. But I have the read_only feature. And they are not implemented the same way :

  • I am passing the paths in the HostConfig.Bind to the docker_client.create (so the file exist before the container starts)
  • You are writing the files using the docker.models.containers.Container.put_archive method (which works after the container has been started)

Are our two approaches complementary ? Reading again the Java documentation for copying files, there does not seem to be a difference, at the API level at least, between copying to a container before or after its startup.

Lenormju avatar Aug 12 '24 23:08 Lenormju

I think copying before can already be solved by with_volume_mapping 😅

mgorsk1 avatar Aug 13 '24 05:08 mgorsk1

committed some code here - https://github.com/testcontainers/testcontainers-python/compare/main...feat/665-with-copy-file-to-container

alexanderankin avatar Aug 13 '24 12:08 alexanderankin

rebased here https://github.com/testcontainers/testcontainers-python/compare/feat/665-with-copy-file-to-container_rebased

alexanderankin avatar Aug 13 '24 12:08 alexanderankin

since this is not a high priority feature this can probably wait for 1) more api refinement - feedback welcomed on my suggestions above 2) more of my free time, as I'm not sure ill be able to spend too much more time on this this week (just spent my entire week's time budget on this today).

alexanderankin avatar Aug 13 '24 12:08 alexanderankin