Always complains about invalid config.ini

[binarykitchen]

What's the bug all about?

Each time I run pipupgrade -ly --pip-path pip it complains about invalid entries in .pipupgrade/config.ini

What command led to this issue?

~ ❯❯❯ pipupgrade -ly --pip-path pip                                                                                                                                                                            ✘ 130
Traceback (most recent call last):
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pipupgrade", line 5, in <module>
    from pipupgrade.__main__ import main
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/__init__.py", line 7, in <module>
    from pipupgrade.__main__    import main
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/__main__.py", line 5, in <module>
    from pipupgrade.commands import command as main
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/commands/__init__.py", line 12, in <module>
    from pipupgrade.commands.helper import (
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/commands/helper.py", line 10, in <module>
    from pipupgrade.model         	import Registry
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/model/__init__.py", line 3, in <module>
    from pipupgrade.model.package  import Package
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/model/package.py", line 20, in <module>
    settings	= Settings()
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/config.py", line 79, in __init__
    self._init()
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/config.py", line 83, in _init
    self.set(k, v)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/config.py", line 89, in set
    self.config.set("settings", key, value)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/config.py", line 53, in set
    config = self.config
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/config.py", line 35, in config
    config.read(path)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/configparser.py", line 697, in read
    self._read(fp, filename)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/configparser.py", line 1113, in _read
    raise e
configparser.ParsingError: Source contains parsing errors: '/Users/michael-heuberger/.pipupgrade/config.ini'
	[line  4]: '= 1.6.8\n'

Logs

/Users/michael-heuberger/.pipupgrade ❯❯❯ pipupgrade --verbose
2020-03-18 12:55:13,885 | INFO | Executing command: /Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip --version
2020-03-18 12:55:14,085 | INFO | Executing command: /Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip3 --version
2020-03-18 12:55:14,297 | INFO | Executing command: /usr/local/bin/pip2 --version
2020-03-18 12:55:14,538 | INFO | Environment: {'version': '1.6.8', 'python_version': '3.8.2', 'os': 'macOS-10.15.3-x86_64-i386-64bit', 'config': {'path': {'BASE': '/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade', 'DATA': '/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pipupgrade/data', 'CACHE': '/Users/michael-heuberger/.pipupgrade'}}, 'pip_executables': [{'executable': '/Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip', 'version': 'pip 20.0.2 from /Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pip (python 3.8)'}, {'executable': '/Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip3', 'version': 'pip 20.0.2 from /Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pip (python 3.8)'}, {'executable': '/usr/local/bin/pip2', 'version': 'pip 20.0.2 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)'}], 'settings': {'settings': {'version': '1.6.8', 'cache_timeout': '86400', 'jobs': '8'}}}
2020-03-18 12:55:14,538 | INFO | Arguments Passed: {'packages': [], 'pip_path': None, 'requirements': None, 'pipfile': None, 'project': None, 'pull_request': False, 'git_username': None, 'git_email': None, 'github_access_token': None, 'github_reponame': None, 'github_username': None, 'target_branch': 'master', 'latest': 0, 'format': 'table', 'all': 0, 'pip': 0, 'self': False, 'jobs': 8, 'user': 0, 'check': 0, 'interactive': 0, 'yes': 0, 'no_included_requirements': 0, 'no_cache': 0, 'no_color': 0, 'output': None, 'ignore_error': 0, 'force': 0, 'verbose': True}
Checking...
2020-03-18 12:55:14,541 | INFO | `pip` executables found: ['/Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip', '/Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip3', '/usr/local/bin/pip2']
2020-03-18 12:55:14,542 | INFO | Using 8 jobs...
2020-03-18 12:55:15,056 | INFO | Fetching installed packages for /Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip...
2020-03-18 12:55:15,057 | INFO | Executing command: /Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip list --outdated --format json
2020-03-18 12:55:15,067 | INFO | Fetching installed packages for /Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip3...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
2020-03-18 12:55:15,068 | INFO | Executing command: /Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pip3 list --outdated --format json
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    exitcode = _main(fd, parent_sentinel)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    prepare(preparation_data)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
    main_content = runpy.run_path(main_path,
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 263, in run_path
    main_content = runpy.run_path(main_path,
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 263, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/michael-heuberger/.pyenv/versions/3.8.2/bin/pipupgrade", line 5, in <module>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
... NEVER ENDS, SCROLLS ENDLESSLY

[achillesrasquinha]

Seems like your configuration file is corrupted. Try deleting the cache

$ rm -rf ~/.pipupgrade

[binarykitchen]

Done that but coming back. What is corrupting it?

[achillesrasquinha]

Could you cat ~/.pipupgrade/config.ini for me?

[binarykitchen]

[settings]
cache_timeout = 86400
version = 1.6.8
jobs = 8

but it changes during the upgrade and becomes corrupt.

[binarykitchen]

here an example when corrupt

[settings]
jobs = 8

1.7.1

[Arzte]

Hiya! I'm experiencing a similar issue, only difference is running pipupgrade at all will corrupt the config.ini and spam the console with errors, removing the cache does not solve it, as it corrupts the moment I run the command again.

[JamesMcGuigan]

I'm getting similar issues. If I edit ~/.pipupgrade/config.ini to try and fix the issue and I get non-deterministic results when rerunning the command (it mostly breaks again, but occasionally works). This affects cases where config.ini has been removed completely.

The main symptoms are:

• duplicate keys
• key=value pairs only containing a random sized tail end of the string output
  • version = 1.7.3
  • = 1.7.3
  • 1.7.3
  • `out = 86400

It would seem there is some sort of race-condition buffering issue with how config.ini lines are written to disk, as occasionally there will be no race-condition and everything will work as expected.

Guesses:

• Two threads are trying to write to the same file at the same time
• Internal text buffers are being passed by reference rather than pass by copy

~/.pipupgrade/config.ini

 [settings]
 version = 1.7.3
  
 = 1.7.3
 cache_timeout = 86400

[JamesMcGuigan]

Looking at the code, I can see that config.ini is managed by the Settings class (which in turn wraps Configuration and ConfigParser). The file.write() function is called on every execution of Configuration.set().

• https://github.com/achillesrasquinha/pipupgrade/blob/bc63e53e8d29bb77bb9b600a7044647f13cdf0cc/src/pipupgrade/config.py#L68

Searching through the codebase, I find that Settings is not a singleton, but actually exists as three separate instances:

• https://github.com/achillesrasquinha/pipupgrade/blob/bc63e53e8d29bb77bb9b600a7044647f13cdf0cc/src/pipupgrade/init.py#L11
• https://github.com/achillesrasquinha/pipupgrade/blob/bc63e53e8d29bb77bb9b600a7044647f13cdf0cc/src/pipupgrade/model/package.py#L20
• https://github.com/achillesrasquinha/pipupgrade/blob/bc63e53e8d29bb77bb9b600a7044647f13cdf0cc/tests/pipupgrade/test_config.py#L5

This is most likely the source of your race condition.

Suggested fix:

• refactor Settings to be a singleton object
• else strictly deduplicate the line settings = Settings() and have this invocation exist exactly once in the codebase

[JamesMcGuigan]

I tried implementing a Singleton, but this caused problems with multiprocessing. Eventually found the solution, which is to add threading.Lock() around the Configuration read/write disk IO.

I've created a pull request that should fix this issue:

• https://github.com/achillesrasquinha/pipupgrade/pull/72

[achillesrasquinha]

Released with 1.7.4

[wilcoxon]

I ran into this for a bit today (in 1.7.4). Uninstalling pip and reinstalling ('pip uninstall pipupgrade', 'pip install pipupgrade') fixed the issue for me. My best guess is something got left behind that was causing issues.