WIP: Shinken python 3 port

[geektophe]

This patch starts porting Shinken core to pyhon3

It includes:

• Python 3 portage with backward python2 compatibility
• Stop maintaining a modified older version of bottle, use the public library instead
• Improved test suite
• Rationalized/simplified the cross services communications
  • Isolated the serialization/deserialization routines in a dedicated (common) module
  • Serialized data is pushed using PUT in binary mode rather than encapsulated in a JSON data using POST
• Standardized setup.py
  • Use standard linux/BSD paths rather than placing all in /var//lib/shinken
  • Use scripts feature for executables deployment to be sure to run the interpreter used for installation
  • Allow installation in a virtualenv
  • Deploy configuration in examples rather than in /etc/shinken to prevent any running configuration overwriting. The configuration deployment will be under the user responsibility.
  • Got rid of the wrapping optparse, let command line parsing to distutils.
  • Stop executing post-installation tasks during install, define a custom post_install instead which:
    • Processes configuration, default and init scripts/systemd unit files templates
    • Set files and directories privileges
    • Install configuration files, default/environment files, and init scripts/systemd unit files from examples
  • Parameters in post_install custom command:
    • --user: define the user the shinken services run under
    • --group: define the group the shinken service run under
    • --confdir: Set the configuration directory
    • --defaultdir: Set default/environment directory containing the files configuring the init scripts
    • --workdir: Set the directory where the logs and retention files are located
    • --logdir: Set the directory where the logs files are located
    • --lockdir: Set the directory where the lock files are located
    • --modules: Set the directory where the shinken modules should be installed
    • --install-conf: Install the configuration files from the processed templates in examples
    • --install-default: Install the default/environment files from the processed templates in examples
    • --install-init: Install the init scripts/systemd unit files from the processed templates in examples

Please note that this patch only applies to the Shinken core, the additional modules will have to be ported independently to be used with this release.

[coveralls]

Coverage decreased (-1.8%) to 25.987% when pulling 91991ad026a4438b4b8b88aab5e19e86413bbc0a on geektophe:python3_port into 2dae40fd1e713aec9e1966a0ab7a580b9180cff2 on naparuba:master.

[naparuba]

test in progress

[naparuba]

J'ai juste un soucis avec systemd (en même temps, je l'aime aps, il me le rends bien ^^)

Oct 29 16:01:13 localhost.localdomain systemd[1]: [/etc/systemd/system/shinken-arbiter.service:12] Executable path is not ...UGCMD Oct 29 16:01:13 localhost.localdomain systemd[1]: shinken-arbiter.service lacks both ExecStart= and ExecStop= setting. Refusing. (centos 7 à jour). Je regarde comment changer ça.

[naparuba]

Une fois les systemd fixés (j'ai pas réussi à avoir les variables d'env, et j'ai mis les chemins absolus)

j'ai une erreur que je sais pas trop quoi faire: [1667053198] ERROR: [Shinken] object: <shinken.daemons.schedulerdaemon.IChecks object at 0x7f028445b7b8> [1667053198] ERROR: [Shinken] Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/Shinken-3.0.0rc1-py3.6.egg/shinken/http_daemon.py", line 496, in f_wrapper args File "/usr/local/lib/python3.6/site-packages/Shinken-3.0.0rc1-py3.6.egg/shinken/http_daemon.py", line 434, in _parse_request_params raise HTTPError('Missing argument %s' % arg) NameError: name 'HTTPError' is not defined

C'est censé être quoi comme Exception?

[naparuba]

Une fois changé par Exception, là je bloque: [1667053443] ERROR: [Shinken] Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/Shinken-3.0.0rc1-py3.6.egg/shinken/http_daemon.py", line 496, in f_wrapper args File "/usr/local/lib/python3.6/site-packages/Shinken-3.0.0rc1-py3.6.egg/shinken/http_daemon.py", line 434, in _parse_request_params raise Exception('Missing argument %s' % arg) Exception: Missing argument max_actions

pourtant il est en kwargs :'(

[geektophe]

Je vais regarder ça rapidement, je te tiens au courant.

[YanAgrit]

Une fois les systemd fixés (j'ai pas réussi à avoir les variables d'env, et j'ai mis les chemins absolus)

j'ai une erreur que je sais pas trop quoi faire: [1667053198] ERROR: [Shinken] object: <shinken.daemons.schedulerdaemon.IChecks object at 0x7f028445b7b8> [1667053198] ERROR: [Shinken] Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/Shinken-3.0.0rc1-py3.6.egg/shinken/http_daemon.py", line 496, in f_wrapper args File "/usr/local/lib/python3.6/site-packages/Shinken-3.0.0rc1-py3.6.egg/shinken/http_daemon.py", line 434, in _parse_request_params raise HTTPError('Missing argument %s' % arg) NameError: name 'HTTPError' is not defined

C'est censé être quoi comme Exception?

Changez juste vers l'exeption globale ;)

def _parse_request_params(self, method, request, args=[]):
        """
        Parses the incoming request, and process the callback parametrs

        :param list args: The callback parameters
        :rtype: mixed
        :return: The callback parameters
        """
        if method in ('get', 'post'):
            parms = {}
            for arg in args:
                val = None
                if method == 'post':
                    val = request.forms.get(arg, None)
                elif method == 'get':
                    val = request.GET.get(arg, None)
                if val:
                    parms[arg] = val
                else:
                    # Checks if the missing arg has a default value
                    default_args = self.registered_fun_defaults.get(arg, {})
                    if arg not in default_args:
                        raise Exception(f'Missing argument {arg}')
            return parms
        elif method == 'put':
            content = request.body
            return deserialize(content)
        else:
            raise Exception(f'Unmanaged HTTP method: {method}')

[YanAgrit]

Une fois changé par Exception, là je bloque: [1667053443] ERROR: [Shinken] Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/Shinken-3.0.0rc1-py3.6.egg/shinken/http_daemon.py", line 496, in f_wrapper args File "/usr/local/lib/python3.6/site-packages/Shinken-3.0.0rc1-py3.6.egg/shinken/http_daemon.py", line 434, in _parse_request_params raise Exception('Missing argument %s' % arg) Exception: Missing argument max_actions

pourtant il est en kwargs :'(

@geektophe le soucis est que les valeurs 'max_q_size' et 'q_factor' n'ont pas ses valeurs par défaut.

Regarde la fonction 'get_available_slots' dans satellite.py ;)

P.S : 'slots' est toujours None donc 'max_actions' n'est jamais mise dans la requête.

[geektophe]

Merci pour le retour @taylorvatem . Je vais corriger ça 👍

[geektophe]

I introduced 2 regressions when I refactored HTTP callback arguments parsing:

• The exception risen when a callback parameter was missing was undefined. I replaced it with bottle.abort() which raises the correct exception for us.
• When I isolated the callback argument parsing in _parse_request_params(), I forgot to transmit the callback name from cbname. The default arguments was looked for using the wrong variable, thus no defaults was found, raising the error.

I missed them because we throttle broks and actions in our production configuration, which I used to ensure the services were behaving as expected, meaning the max_actions parameter was set.

This should be fixed in the last commit.

To follow up the previous comment from @taylorvatem , max_q_size q_factor have well default values, set by PollerLinkand ReactionnerLinkclasses.

https://github.com/naparuba/shinken/blob/9c0836861549859af612c0bbc253c09635f38f90/shinken/objects/pollerlink.py#L29-L50

It's absolutely normal that slots can be None. This is the case when no actions/broks throttling is configured. In this situation, max_actions shouldn't be passed as request parameter, letting the callback implementer decide what the default value should be.

The error related to this parameter was in fact due to a refactoring issue described above. Thanks for having pointed that anyway @taylorvatem :+1:

[naparuba]

test in progress :)

[naparuba]

After a big week-end, I'll give a final test today, should be ok to merge I think ^^

[naparuba]

hum last change do not seems ok for ini. It do not break the use_local_log anymore (cool), but all the ini are ointing to arbiterd.log:

cat /etc/shinken/daemons/reactionnerd.ini
[...]
local_log=/var/log/shinken/arbiterd.log

I think it's because of the setup.py::l620 'local_log=%s/arbiterd.log' % self.logdir

We do not have access to the daemon name here, so maybe try to not match the log_file name in the '^local_log=.+', regexp?

[naparuba]

I think it's the last one, and then we merge and start to change modules (I'll work on the pickle retention for start to update it)

[geektophe]

I'm fixing and let you know. Thanks !

[geektophe]

Same problem with:

lock_file=%s/arbiterd.pid' % self.lockdir

[geektophe]

It should be fixed @naparuba

I used capturing parenthesis in the replacement pattern to keep the original log filename.

'^local_log=.+/([^/]+.log)' -> 'local_log=%s/\1' % self.logdir

[naparuba]

Youhou, working well, congrats ✩◝(◍⌣̎◍)◜✩