Add option to not poll the repo every time

[oleg-nenashev]

When using branch references as a source, the Multirepo plugin always fetches content on rebuild, e.g. when there are changes in mkdocs.yml that actually do not require re-fetch. Eventually it ends up with rate limiting by GitHub CDN on bigger sites.

I think it would be nice to have a developer mode option that disables fetching and hence also speeds up the rebuilds massively.

As an alternative, maybe tag/commit references should be recommended in docs

INFO    -  Multirepo plugin importing docs...
🔳 graphql
🔳 grpc
✅ state (3.676 secs)
✅ 9cookies (8.878 secs)
✅ jwt (9.313 secs)
🔳 cors
✅ stub-mapping-tracker (3.592 secs)
✅ prometheus (9.09 secs)
✅ record-delay (9.335 secs)
🔳 community
✅ dotgithub (9.109 secs)
INFO    -  DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
             File "/usr/lib/python3/dist-packages/apport/report.py", line 13, in <module>
               import fnmatch, glob, traceback, errno, sys, atexit, imp, stat
             File "/usr/lib/python3.8/imp.py", line 31, in <module>
               warnings.warn("the imp module is deprecated in favour of importlib; "
Traceback (most recent call last):
  File "/home/onenashev/.local/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs/__main__.py", line 270, in serve_command
    serve.serve(**kwargs)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs/commands/serve.py", line 86, in serve
    builder(config)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs/commands/serve.py", line 67, in builder
    build(config, live_server=None if is_clean else server, dirty=is_dirty)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs/commands/build.py", line 277, in build
    config = config.plugins.on_config(config)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs/plugins.py", line 527, in on_config
    return self.run_event('config', config)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs/plugins.py", line 507, in run_event
    result = method(item, **kwargs)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs_multirepo_plugin/plugin.py", line 312, in on_config
    return self.handle_nav_repos_import(config, nav_repos)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs_multirepo_plugin/plugin.py", line 270, in handle_nav_repos_import
    asyncio_run(batch_execute(repos=docs_repo_objs, method=Repo.sparse_clone))
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs_multirepo_plugin/util.py", line 114, in asyncio_run
    asyncio.run(futures)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs_multirepo_plugin/structure.py", line 416, in batch_execute
    repo = await future
  File "/usr/lib/python3.8/asyncio/tasks.py", line 619, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs_multirepo_plugin/structure.py", line 193, in sparse_clone
    await execute_bash_script("sparse_clone.sh", args, self.temp_dir)
  File "/home/onenashev/.local/lib/python3.8/site-packages/mkdocs_multirepo_plugin/util.py", line 108, in execute_bash_script
    raise BashException(f"\n{stderr}\n")
mkdocs_multirepo_plugin.util.BashException:
Cloning into 'cors'...
fatal: unable to access 'https://github.com/RichieLoco/WiremockCorsExtension/': Could not resolve host: github.com
fatal: could not fetch 007fb4494b6dbfb343d9267a27964342d67097b3 from promisor remote
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

[Spacetown]

The repository is cloned in a temporary directory which is cleaned after the build. Maybe a option to skip this clean step and use the previous data or update the clone.

[oleg-nenashev]

@Spacetown might be. I started a local patch, will try to submit a PR today