gitman icon indicating copy to clipboard operation
gitman copied to clipboard
verified publisher icon jacebrowning

Support creating hard links (in additional to symlinks)

Open dxlr8r opened this issue 2 years ago • 7 comments

With for example, gitman.yml:

location: vendor/gitman
sources:
  - repo: https://github.com/jacebrowning/gitman.git
    name: gitman
    rev: main
    type: git
    links:
      - source: doc
        target: vendor/doc/gitman

And .gitignore:

vendor/gitman

vendor/doc/gitman will be a symlink, and content in vendor/gitman will be ignored.

For my project I want to keep copies of the libraries I use, and since git does not follow symlinks the approach above does not work.

I could of course remove the entry in .gitignore, but then I get a lot of unneeded resources.

Hard links seem to be the way to resolve this. Suggested implementation, gitman.yml:

location: vendor/gitman
sources:
  - repo: https://github.com/jacebrowning/gitman.git
    name: gitman
    rev: main
    type: git
    hardlinks:
      - source: doc
        target: vendor/doc/gitman

vendor/doc/gitman would be a regular directory, each file inside would be a hardlink, and any nesting directory would be recreated with the same name as a directory (not symlink). Like a copy (cp -r) but the files are hard linked instead of copies.

Of course hard links has some limitations, has to be on same file system for instance. So an option to fallback to a copy would perhaps required.

dxlr8r avatar Mar 03 '23 14:03 dxlr8r

Git does not follow symlinks

Can you expand on this a bit? If you attempt to use soft links for you use case, what happens?

Suggested implementation

Initially, my thought was something like this:

...
links:
      - source: doc
        target: vendor/doc/gitman
        type: hard

or this:

...
links:
      - source: doc
        target: vendor/doc/gitman
        symbolic: false

jacebrowning avatar Mar 03 '23 15:03 jacebrowning

Git does not follow symlinks

Can you expand on this a bit? If you attempt to use soft links for you use case, what happens?

Git in general does not follow symlinks, for git, a symlink is a regular file. And if the directory/file it points to is in .gitignore, it will be copied in it's broken (pointing to a non existing directory/file) state.

Suggested implementation

Initially, my thought was something like this:

...
links:
      - source: doc
        target: vendor/doc/gitman
        type: hard

or this:

...
links:
      - source: doc
        target: vendor/doc/gitman
        symbolic: false

Seems good to me. The important part is the feature itself :)

dxlr8r avatar Mar 03 '23 15:03 dxlr8r

Most likely this function will need an additional parameter if someone wants to work on a proposal.

jacebrowning avatar Mar 03 '23 17:03 jacebrowning

Not an experience Python programmer myself, but I can take a look at it. First, how do I get started? I want to import shell.py to get access to functions mkdir, call etc. but I seem to be unable to do this. I could not find developer documentation either.

dxlr8r avatar May 12 '23 14:05 dxlr8r

Trying to build following instructions https://gitman.readthedocs.io/en/latest/about/contributing/#installation , from my virtual environment:

(mygitman) sBook github/gitman % make all
poetry run isort gitman tests
('Configuration file exists at /Users/simen/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/simen/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)
poetry run black gitman tests
('Configuration file exists at /Users/simen/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/simen/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)

poetry run mypy gitman tests --config-file=.mypy.ini
('Configuration file exists at /Users/simen/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/simen/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)
Success: no issues found in 34 source files
poetry run pylint gitman tests --rcfile=.pylint.ini
('Configuration file exists at /Users/simen/Library/Application Support/pypoetry, reusing this directory.\n\nConsider moving TOML configuration files to /Users/simen/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.',)
Traceback (most recent call last):
  File "/Users/simen/projects/private/github/gitman/mygitman/bin/pylint", line 4, in <module>
    run_pylint()
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/__init__.py", line 22, in run_pylint
    PylintRun(sys.argv[1:])
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/run.py", line 349, in __init__
    linter.check(args)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 862, in check
    self._check_files(
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 896, in _check_files
    self._check_file(get_ast, check_astroid_module, name, filepath, modname)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 922, in _check_file
    check_astroid_module(ast_node)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 1054, in check_astroid_module
    retval = self._check_astroid_module(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/lint/pylinter.py", line 1099, in _check_astroid_module
    walker.walk(ast_node)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  [Previous line repeated 1 more time]
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/utils/ast_walker.py", line 72, in walk
    callback(astroid)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/base.py", line 782, in visit_call
    self._check_inferred_class_is_abstract(inferred, node)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/base.py", line 799, in _check_inferred_class_is_abstract
    abstract_methods = _has_abstract_methods(inferred)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/base.py", line 386, in _has_abstract_methods
    return len(utils.unimplemented_abstract_methods(node)) > 0
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/utils.py", line 825, in unimplemented_abstract_methods
    inferred = safe_infer(obj)
               ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/pylint/checkers/utils.py", line 1143, in safe_infer
    value = next(infer_gen)
            ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/inference.py", line 227, in infer_call
    for callee in self.func.infer(context):
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
          ^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/decorators.py", line 93, in wrapped
    generator = _func(node, context, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/inference.py", line 273, in infer_import_from
    module = self.do_import_module()
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/mixins.py", line 99, in do_import_module
    return mymodule.import_module(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/scoped_nodes.py", line 642, in import_module
    return MANAGER.ast_from_module_name(absmodname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/manager.py", line 189, in ast_from_module_name
    return self.ast_from_file(found_spec.location, modname, fallback=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/manager.py", line 98, in ast_from_file
    return AstroidBuilder(self).file_build(filepath, modname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/builder.py", line 137, in file_build
    module = self._data_build(data, modname, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/builder.py", line 191, in _data_build
    module = builder.visit_module(node, modname, node_file, package)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 113, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 113, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 125, in visit
    return visit_method(node, parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 633, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 623, in _visit_functiondef
    body=[self.visit(child, newnode) for child in node.body],
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 623, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/simen/projects/private/github/gitman/mygitman/lib/python3.11/site-packages/astroid/rebuilder.py", line 123, in visit
    visit_method = getattr(self, visit_name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'TreeRebuilder' object has no attribute 'visit_match'. Did you mean: 'visit_with'?
make: *** [check] Error 1
Exit: 2
(mygitman) sBook github/gitman %

dxlr8r avatar May 16 '23 12:05 dxlr8r

I gave up using macOS, so tried using Debian Bookworm. Sadly got many of the same issues as I did with macOS. I found some references to that Python 3.11 was too new etc. so I installed pyenv in Debian, and found Python 3.8.17 working perfectly.

Now the make commands actually work.

And the code is done as well. I am not a Python developer, and could not wrap my head around the test framework though, but except for that everything should be in place and working.

dxlr8r avatar Jun 10 '23 08:06 dxlr8r

https://github.com/jacebrowning/gitman/pull/310

dxlr8r avatar Jun 10 '23 08:06 dxlr8r