Return of Store getting corrupted after !shutdown

[Steveveepee]

I am...

• Reporting a bug

I am running...

• Errbot version: 6.1.4
• OS version:Windows 10
• Python version: 3.8.2
• Using a virtual environment: yes

Issue description

The core store is being corrupted after shutdown after configuring a Webserver. Repeat of issue #1125

Steps to reproduce

See Gist for full output.

>mkdir .venv
>pipenv --three
>pipenv shell
>pipenv install errbot  #errors out because of a pre-release dependency
>pipenv lock --pre
>pipenv sync
>errbot -T
[@CHANGE_ME ➡ @errbot] >>> !shutdown --confirm
>errbot
[@CHANGE_ME ➡ @errbot] >>> !plugin config Webserver {'HOST': '0.0.0.0', 'PORT': 3141, 'SSL': {'certificate': '', 'enabled': False, 'host': '127.0.0.1', 'key': '', 'port': 3142}}
Plugin configuration done.

[@CHANGE_ME ➡ @errbot] >>> !shutdown --confirm
Roger that. I am shutting down.
❯ errbot
15:50:01 INFO     errbot.bootstrap          Found Storage plugin: Shelf.
15:50:01 INFO     errbot.bootstrap          Found Backend plugin: Text
15:50:01 DEBUG    errbot.storage            Opening storage 'repomgr'
15:50:01 DEBUG    errbot.storage.shelf      Open shelf storage I:\errbot_test\data\repomgr.db
15:50:02 DEBUG    errbot.core               ErrBot init.
15:50:02 DEBUG    errbot.backends.base      Backend init.
15:50:02 DEBUG    errbot.core               created a thread pool of size 10.
15:50:02 DEBUG    errbot.backends.text      Text Backend Init.
15:50:02 DEBUG    errbot.backends.text      Bot username set at @errbot.
15:50:02 DEBUG    MARKDOWN                  Successfuly imported extension module "markdown.extensions.smart_strong".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "markdown.extensions.smart_strong.SmartEmphasisExtension".
15:50:02 DEBUG    MARKDOWN                  Successfuly imported extension module "markdown.extensions.fenced_code".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "markdown.extensions.fenced_code.FencedCodeExtension".
15:50:02 DEBUG    MARKDOWN                  Successfuly imported extension module "markdown.extensions.footnotes".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "markdown.extensions.footnotes.FootnoteExtension".
15:50:02 DEBUG    MARKDOWN                  Successfuly imported extension module "markdown.extensions.attr_list".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "markdown.extensions.attr_list.AttrListExtension".
15:50:02 DEBUG    MARKDOWN                  Successfuly imported extension module "markdown.extensions.def_list".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "markdown.extensions.def_list.DefListExtension".
15:50:02 DEBUG    MARKDOWN                  Successfuly imported extension module "markdown.extensions.tables".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "markdown.extensions.tables.TableExtension".
15:50:02 DEBUG    MARKDOWN                  Successfuly imported extension module "markdown.extensions.abbr".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "markdown.extensions.abbr.AbbrExtension".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "markdown.extensions.extra.ExtraExtension".
15:50:02 DEBUG    MARKDOWN                  Successfully loaded extension "errbot.rendering.ansiext.AnsiExtension".
15:50:02 DEBUG    errbot.storage            Opening storage 'core'
15:50:02 DEBUG    errbot.storage.shelf      Open shelf storage I:\errbot_test\data\core.db
15:50:02 ERROR    errbot.bootstrap          Unable to load or configure the backend.
Traceback (most recent call last):
  File "I:\Python3\lib\shelve.py", line 111, in __getitem__
    value = self.cache[key]
KeyError: 'configs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "i:\errbot_test\.venv\lib\site-packages\errbot\bootstrap.py", line 179, in setup_bot
    botpm = BotPluginManager(storage_plugin,
  File "i:\errbot_test\.venv\lib\site-packages\errbot\plugin_manager.py", line 175, in __init__
    if CONFIGS not in self:
  File "i:\errbot_test\.venv\lib\site-packages\errbot\storage\__init__.py", line 78, in __contains__
    self._store.get(x)
  File "i:\errbot_test\.venv\lib\site-packages\errbot\storage\shelf.py", line 19, in get
    return self.shelf[key]
  File "I:\Python3\lib\shelve.py", line 114, in __getitem__
    value = Unpickler(f).load()
_pickle.UnpicklingError: pickle data was truncated

errbot.log

[sijis]

This may be a Windows specific bug. I don't have access to a Windows 10 machine to fully validate this.

However, I did try to reproduce, using your steps using docker with docker run --rm -it docker.io/library/python:3.8-slim bash

Could you retry and see if you can reproduce using the same docker image?

[sijis]

There is also something odd about using pipenv install errbot command that gives a

[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved.
...
ERROR: ERROR: Could not find a version that matches pygments-markdown-lexer>=0.1.0.dev39
Skipped pre-versions: 0.1.0.dev39

While just doing pip3 install errbot just installs dependencies without warnings.

[Steveveepee]

There is also something odd about using pipenv install errbot command that gives a

[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved.
...
ERROR: ERROR: Could not find a version that matches pygments-markdown-lexer>=0.1.0.dev39
Skipped pre-versions: 0.1.0.dev39

While just doing pip3 install errbot just installs dependencies without warnings.

I have split it out in the example above, but you can see from the gists that if you do either a pipenv install errbot --pre or a pipenv lock --pre it will grab that pre-release library and work

I've tried in WSL linux (ubuntu) and it works fine, so It's definately a Windows problem.

Let me know where to start looking for problems or troubleshooting

[Steveveepee]

Ran the sequence in that docker image and no problems.