OpenCTIConnectorHelper does not output stack traces due to a custom sys.excepthook

[Markus98]

Description

After an instance of OpenCTIConnectorHelper has been initialized, any exception raised directly after it leads to the program getting terminated and the stack trace will not be outputted. This makes debugging custom connectors harder, since logs do not show the reason for program termination.

Environment

1. OS (where OpenCTI server runs): Official docker-compose, Ubuntu 22.04
2. OpenCTI version: 6.1.3
3. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

import os
from pycti import OpenCTIConnectorHelper
import yaml

# Load config file
config_file_path = os.path.dirname(
    os.path.abspath(__file__)) + "/config.yml"
connector_config = (
    yaml.load(open(config_file_path, encoding="UTF-8"),
              Loader=yaml.FullLoader)
    if os.path.isfile(config_file_path)
    else {}
)

# OpenCTI configuration
connector_helper = OpenCTIConnectorHelper(config=connector_config)
raise Exception("This should output a stack trace")

Expected Output

Should output the stack trace:

INFO:api:Health check (platform version)...
INFO:api:Health check (platform version)...
INFO:test-connector:Connector registered with ID
INFO:test-connector:Starting PingAlive thread
Traceback (most recent call last):
  File "/home/user/test.py", line 20, in <module>
    raise Exception("This should output a stack trace")
Exception: This should output a stack trac

Actual Output

Stack trace gets omitted due to the custom excepthook which simply terminates the program:

INFO:api:Health check (platform version)...
INFO:api:Health check (platform version)...
INFO:test-connector:Connector registered with ID
INFO:test-connector:Starting PingAlive thread
fish: Job 1, 'python test.py' terminated by signal SIGTERM (Polite quit request)

Additional information

The culprit is the custom excepthook which is defined here: https://github.com/OpenCTI-Platform/client-python/blob/4c182df5ee82632cd5b10d6397a6984c08c70a90/pycti/connector/opencti_connector_helper.py#L31-L32 And set here: https://github.com/OpenCTI-Platform/client-python/blob/4c182df5ee82632cd5b10d6397a6984c08c70a90/pycti/connector/opencti_connector_helper.py#L657

Currently the custom hook simply terminates the program without outputting anything. It should at least print the stack trace to help with debugging.

[Jipegien]

@helene-nguyen

[helene-nguyen]

@Markus98 Thank you for your feedback! We have made an effort to simplify the debugging process. Could you please provide more details about what you're trying to achieve?

• What specific issue are you encountering?
• What steps have you taken so far to resolve the problem?
• What is your desired outcome or goal?
• Are there particular features or functionalities you are focusing on?

Your detailed responses will help us assist you more effectively :)

[Markus98]

Hello!

• What specific issue are you encountering?

When there is an uncaught exception in my custom connectors, the logs do not show what exception caused the crash. This is illustrated in the example I provided above. This makes it hard to find reasons why a connector crashed in a production environment for example, since the logs just state Terminated without additional logs.

• What steps have you taken so far to resolve the problem?

For local debugging I have modified the pycti library code by replacing the custom exception hook and replacing it with one that raises an exception instead of terminating the program with SIGTERM. This works well for local debugging and prints the exceptions and stack traces that caused the program to crash, but it may not be a robust enough solution for the library if additional cleanup needs to be done.

• What is your desired outcome or goal?

I want to see logs for why a custom connector using OpenCTIConnectorHelper has crashed.

• Are there particular features or functionalities you are focusing on?

This is for development quality of life in general for custom connectors.

[peritz]

+1 It is making it very difficult to understand what has gone wrong when deploying connectors that do not report issues to stdout before crasahing, such as Feedly and any custom connectors

[marcel-fox]

I'm also running into this. The traceback used to be printed in killProgramHook, but that code was removed in https://github.com/OpenCTI-Platform/client-python/commit/37b4587f0e76ae5de69624c209d9b92752203826

[helene-nguyen]

@marcel-fox @peritz @Markus98 sorry for the late answer, some of our connectors haven't any error explanation and get only a Terminated logged and we have corrected most of them.

If you are developing a new connector, we have reworked connector templates maintain by our team.

Here an example using traceback library: https://github.com/OpenCTI-Platform/connectors/blob/master/templates/external-import/src/main.py#L19

And here all of the instructions with best practices to use for creating connectors for OpenCTI: https://github.com/OpenCTI-Platform/connectors/tree/master/templates

Hope this can be helpful for you.

I close the issue for now but it can be re-opened if needed.