opal icon indicating copy to clipboard operation
opal copied to clipboard
verified publisher icon permitio

OPAL Server doesn't clean up symbolic links when github is down

Open kreyyser opened this issue 1 year ago • 15 comments

When opal server has github policies setup and github is down for some time opal server seems like spawn zombie processes but apparently looks like it is just a list of symbolic links that are not cleaned up.

To Reproduce run OPAL with github policies source as a container make somehow github return 500 list processes in opal server container

Expected behavior No zombie processes or broken links proc directory

Screenshots opal-server-proc

OPAL version

  • Version: 0.7.6

kreyyser avatar Aug 06 '24 08:08 kreyyser

/bounty $100

gemanor avatar Sep 18 '24 10:09 gemanor

💎 $100 bounty • Permit.io

Steps to solve:

  1. Start working: Comment /attempt #634 with your implementation plan
  2. Submit work: Create a pull request including /claim #634 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

❗ Important guidelines:

  • To claim a bounty, you need to provide a short demo video of your changes in your pull request
  • If anything is unclear, ask for clarification before starting as this will help avoid potential rework
  • Low quality AI PRs will not receive review and will be closed
  • Do not ask to be assigned unless you've contributed before

Thank you for contributing to permitio/opal!

Attempt Started (UTC) Solution Actions
🟢 @FelipeFMA Dec 01, 2025, 08:51:56 PM #850 Reward
🟢 @krisharyan Jul 02, 2025, 01:28:30 PM WIP
🟢 @adarsh6980 Nov 04, 2025, 02:33:28 PM WIP
🟢 @Tibs15 Jul 05, 2025, 11:54:23 PM WIP
🟢 @Brijeshthummar02 Jun 16, 2025, 12:31:40 PM WIP

algora-pbc[bot] avatar Sep 18 '24 10:09 algora-pbc[bot]

Hi @gemanor I'm interested in this task. Can I be assigned? Thank You.

sroopsai avatar Sep 20 '24 11:09 sroopsai

@gemanor I am trying to replicate this; seems to be working fine;

Replicating logs using toxiproxy 

┌─[onyedikachi@Davids-MacBook-Air] - [~/Documents/codes/algora-bounties/opal] - [Fri Sep 20, 18:36]
└─[$] <git:(master)> docker exec toxiproxy wget -O- --post-data='{"name": "github", "listen": "0.0.0.0:8666", "upstream": "github.com:443"}' --header='Content-Type: application/json' http://localhost:8474/proxies
Connecting to localhost:8474 (127.0.0.1:8474)
{"name":"github","listen":"[::]:8666","upstream":"github.com:443","enabled":true,"toxics":[]}-                    100% |*******************************|    93   0:00:00 ETA

┌─[onyedikachi@Davids-MacBook-Air] - [~/Documents/codes/algora-bounties/opal] - [Fri Sep 20, 18:38]
└─[$] <git:(master)> docker exec toxiproxy wget -O- --post-data='{"type": "latency", "attributes": {"latency": 5000}}' --header='Content-Type: application/json' http://localhost:8474/proxies/github/toxics
Connecting to localhost:8474 (127.0.0.1:8474)
-                    100%{"attributes":{"latency":5000,"jitter":0},"name":"latency_downstream","type":"latency","stream":"downstream","toxicity":1} |*******************************|   122   0:00:00 ETA

┌─[onyedikachi@Davids-MacBook-Air] - [~/Documents/codes/algora-bounties/opal] - [Fri Sep 20, 18:38]
└─[$] <git:(master)> docker exec toxiproxy wget -O- --post-data='{"type": "timeout", "attributes": {"timeout": 10000}}' --header='Content-Type: application/json' http://localhost:8474/proxies/github/toxics
Connecting to localhost:8474 (127.0.0.1:8474)
-                    100% |*******************************|   112   0:00:00 ETA

{"attributes":{"timeout":10000},"name":"timeout_downstream","type":"timeout","stream":"downstream","toxicity":1}%                                                                                    
┌─[onyedikachi@Davids-MacBook-Air] - [~/Documents/codes/algora-bounties/opal] - [Fri Sep 20, 18:38]
└─[$] <git:(master)> docker exec toxiproxy wget -O- http://localhost:8474/proxies
Connecting to localhost:8474 (127.0.0.1:8474)
-                    100%{"github":{"name":"github","listen":"[::]:8666","upstream":"github.com:443","enabled":true,"toxics":[{"attributes":{"latency":5000,"jitter":0},"name":"latency_downstream","type":"latency","stream":"downstream","toxicity":1},{"attributes":{"timeout":10000},"name":"timeout_downstream","type":"timeout","stream":"downstream","toxicity":1}]}} |*******************************|   339   0:00:00 ETA

┌─[onyedikachi@Davids-MacBook-Air] - [~/Documents/codes/algora-bounties/opal] - [Fri Sep 20, 18:38]
└─[$] <git:(master)> docker-compose -f docker/docker-compose-example.yml stop opal_server opal_client
[+] Stopping 2/2
 ✔ Container docker-opal_client-1  Stopped                                                                                                                                                     10.3s 
 ✔ Container docker-opal_server-1  Stopped                                                                                                                                                     10.3s 
┌─[onyedikachi@Davids-MacBook-Air] - [~/Documents/codes/algora-bounties/opal] - [Fri Sep 20, 18:39]
└─[$] <git:(master)> docker-compose -f docker/docker-compose-example.yml up -d opal_server opal_client
[+] Running 3/3
 ✔ Container docker-broadcast_channel-1  Running                                                                                       0.0s 
 ✔ Container docker-opal_server-1        Started                                                                                       0.4s 
 ✔ Container docker-opal_client-1        Started                                                                                       0.5s 
┌─[onyedikachi@Davids-MacBook-Air] - [~/Documents/codes/algora-bounties/opal] - [Fri Sep 20, 18:41]
└─[$] <git:(master*)> docker-compose -f docker/docker-compose-example.yml logs -f opal_server
opal_server-1  | + exec gunicorn -b 0.0.0.0:7002 -k uvicorn.workers.UvicornWorker --workers=4 -c ./gunicorn_conf.py opal_server.main:app -t 30 --keep-alive 5
opal_server-1  | [2024-09-20 17:41:57 +0000] [7] [INFO] Starting gunicorn 22.0.0
opal_server-1  | [2024-09-20 17:41:57 +0000] [7] [INFO] Listening at: http://0.0.0.0:7002 (7)
opal_server-1  | [2024-09-20 17:41:57 +0000] [7] [INFO] Using worker: uvicorn.workers.UvicornWorker
opal_server-1  | 2024-09-20 17:41:57.771 | WARNING  | __config__:when_ready:17 - Finished pre loading scopes...
opal_server-1  | [2024-09-20 17:41:57 +0000] [11] [INFO] Booting worker with pid: 11
opal_server-1  | [2024-09-20 17:41:57 +0000] [12] [INFO] Booting worker with pid: 12
opal_server-1  | [2024-09-20 17:41:57 +0000] [15] [INFO] Booting worker with pid: 15
opal_server-1  | 2024-09-20T17:41:58.012734+0000 | 12 | opal_common.monitoring.apm              | INFO  | DataDog APM disabled
opal_server-1  | 2024-09-20T17:41:58.012798+0000 | 11 | opal_common.monitoring.apm              | INFO  | DataDog APM disabled
opal_server-1  | 2024-09-20T17:41:58.013167+0000 | 11 | opal_common.monitoring.metrics          | INFO  | DogStatsD metrics disabled
opal_server-1  | 2024-09-20T17:41:58.013166+0000 | 12 | opal_common.monitoring.metrics          | INFO  | DogStatsD metrics disabled
opal_server-1  | 2024-09-20T17:41:58.014262+0000 | 11 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server-1  | 2024-09-20T17:41:58.014262+0000 | 12 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server-1  | 2024-09-20T17:41:58.014370+0000 | 11 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
opal_server-1  | 2024-09-20T17:41:58.014372+0000 | 12 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
opal_server-1  | 2024-09-20T17:41:58.045725+0000 | 12 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
opal_server-1  | 2024-09-20T17:41:58.045734+0000 | 11 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
opal_server-1  | 2024-09-20T17:41:58.045904+0000 | 11 | opal_server.server                      | INFO  | leadership lock acquired, leader pid: 11
opal_server-1  | 2024-09-20T17:41:58.045956+0000 | 11 | opal_common.topics.publisher            | INFO  | started broadcaster keepalive task: topic is '__broadcast_session_keepalive__', interval is 3600 seconds
opal_server-1  | 2024-09-20T17:41:58.046093+0000 | 11 | opal_server.policy.watcher.factory      | INFO  | Policy repo will be cloned to: /opal/regoclone/opal_repo_clone
opal_server-1  | 2024-09-20T17:41:58.046174+0000 | 11 | opal_server.policy.watcher.task         | INFO  | Launching policy watcher
opal_server-1  | 2024-09-20T17:41:58.046431+0000 | 11 | fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
opal_server-1  | 2024-09-20T17:41:58.046487+0000 | 11 | opal_server.policy.watcher.task         | INFO  | listening on webhook topic: 'webhook'
opal_server-1  | 2024-09-20T17:41:58.048350+0000 | 11 | opal_common.git_utils.repo_cloner       | INFO  | Cloning repo from 'https://github.com/permitio/opal-example-policy-repo' to '/opal/regoclone/opal_repo_clone' (branch: 'master')
opal_server-1  | 2024-09-20T17:41:58.049119+0000 | 11 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
opal_server-1  | [2024-09-20 17:41:58 +0000] [17] [INFO] Booting worker with pid: 17
opal_server-1  | 2024-09-20T17:41:58.084883+0000 | 15 | opal_common.monitoring.apm              | INFO  | DataDog APM disabled
opal_server-1  | 2024-09-20T17:41:58.085484+0000 | 15 | opal_common.monitoring.metrics          | INFO  | DogStatsD metrics disabled
opal_server-1  | 2024-09-20T17:41:58.086213+0000 | 15 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server-1  | 2024-09-20T17:41:58.086425+0000 | 15 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
opal_server-1  | 2024-09-20T17:41:58.116861+0000 | 15 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
opal_server-1  | 2024-09-20T17:41:58.195044+0000 | 17 | opal_common.monitoring.apm              | INFO  | DataDog APM disabled
opal_server-1  | 2024-09-20T17:41:58.195432+0000 | 17 | opal_common.monitoring.metrics          | INFO  | DogStatsD metrics disabled
opal_server-1  | 2024-09-20T17:41:58.195966+0000 | 17 | opal_server.server                      | INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!
opal_server-1  | 2024-09-20T17:41:58.196045+0000 | 17 | opal_server.pubsub                      | INFO  | Initializing broadcaster for server<->server communication
opal_server-1  | 2024-09-20T17:41:58.198119+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:41:58.216269+0000 | 17 | opal_server.server                      | INFO  | *** OPAL Server Startup ***
opal_server-1  | 2024-09-20T17:41:58.476571+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:41:58.598549+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:41:59.099518+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:41:59.648438+0000 | 15 | fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
opal_server-1  | 2024-09-20T17:41:59.648460+0000 | 17 | fastapi_websocket_pubsub.event_broadc...| INFO  | Listening for incoming events from broadcast channel (first listener started)
opal_server-1  | 2024-09-20T17:41:59.649088+0000 | 17 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
opal_server-1  | 2024-09-20T17:41:59.649092+0000 | 15 | fastapi_websocket_pubsub.event_broadc...| INFO  | Starting broadcaster listener
opal_server-1  | 2024-09-20T17:41:59.669025+0000 | 17 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
opal_server-1  | 2024-09-20T17:41:59.669685+0000 | 17 | websockets.legacy.server                | INFO  | connection open
opal_server-1  | 2024-09-20T17:41:59.669957+0000 | 15 | fastapi_websocket_rpc.websocket_rpc_e...| INFO  | Client connected
opal_server-1  | 2024-09-20T17:41:59.670631+0000 | 15 | websockets.legacy.server                | INFO  | connection open
opal_server-1  | 2024-09-20T17:41:59.708588+0000 | 11 | opal_server.data.api                    | INFO  | Serving source configuration
opal_server-1  | 2024-09-20T17:41:59.708823+0000 | 12 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.19.0.4:57610 - "GET /policy?path=. HTTP/1.1" 503
opal_server-1  | 2024-09-20T17:41:59.708903+0000 | 11 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.19.0.4:57616 - "GET /data/config HTTP/1.1" 200
opal_server-1  | 2024-09-20T17:41:59.711237+0000 | 11 | opal_server.data.api                    |WARNING | Serving default all-data route, meaning DATA_CONFIG_SOURCES was not configured!
opal_server-1  | 2024-09-20T17:41:59.711377+0000 | 11 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.19.0.4:57630 - "GET /policy-data HTTP/1.1" 200
opal_server-1  | 2024-09-20T17:42:00.433137+0000 | 17 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.19.0.4:57632 - "GET /policy?path=. HTTP/1.1" 503
opal_server-1  | 2024-09-20T17:42:00.911339+0000 | 17 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.19.0.4:57640 - "GET /policy?path=. HTTP/1.1" 503
opal_server-1  | 2024-09-20T17:42:02.333280+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:42:03.825769+0000 | 17 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.19.0.4:58364 - "GET /policy?path=. HTTP/1.1" 503
opal_server-1  | 2024-09-20T17:42:09.837089+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:42:11.090002+0000 | 17 | uvicorn.protocols.http.httptools_impl   | INFO  | 172.19.0.4:58372 - "GET /policy?path=. HTTP/1.1" 503
opal_server-1  | 2024-09-20T17:42:11.801752+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:42:36.468977+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:42:40.567739+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:43:07.746330+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:43:14.849461+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:43:28.527766+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:43:48.992953+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:43:56.768192+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:44:06.267090+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:44:21.376191+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:44:49.010690+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:44:49.124212+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:45:08.329527+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:45:37.381865+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:46:00.265777+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:46:25.112950+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:46:45.171187+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:47:09.747848+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:47:21.355348+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:47:28.499200+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:47:44.470503+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:47:59.221237+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:48:27.461919+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:48:30.746226+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:48:31.105973+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:48:44.245960+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:48:47.811824+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:48:49.139385+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:49:08.729005+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:49:17.909007+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:49:33.220675+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:49:47.082229+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:50:11.703199+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:50:18.889581+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:50:22.436961+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:50:24.396999+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:50:44.178181+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:51:03.512839+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:51:09.938511+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:51:32.587447+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:51:45.621470+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:51:49.469047+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:51:57.782177+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:52:01.611626+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:52:29.540788+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:52:37.886343+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:53:03.061564+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:53:10.775901+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:53:11.052095+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:53:25.802246+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:53:29.603888+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:53:54.701828+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:54:08.243837+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:54:35.494441+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:54:54.073312+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:55:06.485363+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:55:09.990394+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:55:20.564042+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:55:33.882323+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:55:59.056847+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:56:13.184216+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:56:27.473293+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:56:50.704705+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:56:54.311478+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:57:03.650724+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:57:18.090909+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:57:40.700203+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:57:56.296209+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:58:14.987426+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:58:38.114188+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:58:51.931545+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:58:58.598166+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:59:05.920967+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:59:24.014810+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:59:24.795867+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T17:59:51.922849+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '
opal_server-1  | 2024-09-20T18:00:05.005612+0000 | 11 | opal_common.git_utils.repo_cloner       |ERROR  | cannot clone policy repo: Cmd('git') failed due to: exit code(128)
opal_server-1  |   cmdline: git clone -v --branch=master -- https://github.com/permitio/opal-example-policy-repo /opal/regoclone/opal_repo_clone
opal_server-1  |   stderr: 'Cloning into '/opal/regoclone/opal_repo_clone'...
opal_server-1  | fatal: unable to access 'https://github.com/permitio/opal-example-policy-repo/': gnutls_handshake() failed: Internal error
opal_server-1  | '

Checking proc for zombies:

> docker-compose -f docker/docker-compose-example.yml exec opal_server sh -c 'for pid in /proc/[0-9]*; do echo -n "${pid##*/} "; cat $pid/cmdline 2>/dev/null; echo; done'
1 bash./start.sh
11 /usr/local/bin/python/usr/local/bin/gunicorn-b0.0.0.0:7002-kuvicorn.workers.UvicornWorker--workers=4-c./gunicorn_conf.pyopal_server.main:app-t30--keep-alive5
12 /usr/local/bin/python/usr/local/bin/gunicorn-b0.0.0.0:7002-kuvicorn.workers.UvicornWorker--workers=4-c./gunicorn_conf.pyopal_server.main:app-t30--keep-alive5
15 /usr/local/bin/python/usr/local/bin/gunicorn-b0.0.0.0:7002-kuvicorn.workers.UvicornWorker--workers=4-c./gunicorn_conf.pyopal_server.main:app-t30--keep-alive5
17 /usr/local/bin/python/usr/local/bin/gunicorn-b0.0.0.0:7002-kuvicorn.workers.UvicornWorker--workers=4-c./gunicorn_conf.pyopal_server.main:app-t30--keep-alive5
294 sh-cfor pid in /proc/[0-9]*; do echo -n "${pid##*/} "; cat $pid/cmdline 2>/dev/null; echo; done
7 /usr/local/bin/python/usr/local/bin/gunicorn-b0.0.0.0:7002-kuvicorn.workers.UvicornWorker--workers=4-c./gunicorn_conf.pyopal_server.main:app-t30--keep-alive5

> docker-compose -f docker/docker-compose-example.yml exec opal_server ls -l /proc/*/exe
zsh: no matches found: /proc/*/exe

onyedikachi-david avatar Sep 20 '24 18:09 onyedikachi-david

@kreyyser can you please point @onyedikachi-david for reproduction steps?

@sroopsai let wait for @kreyyser comments to check the exact reproduction steps beforehand.

gemanor avatar Sep 21 '24 19:09 gemanor

Hi @gemanor @sroopsai . I'll try to make a reproducible example this week. To give more info - we have opal server in 2 replicas and kafka as a backbone. basically they work properly for some time with github and then it goes down and we observe these orphan symlinks. After opal server restart and github up - those links go away and do not appear until next github issue.

kreyyser avatar Sep 23 '24 06:09 kreyyser

@sroopsai / @kreyyser any updates?

gemanor avatar Sep 30 '24 08:09 gemanor

Hi @gemanor,

sorry, haven't got a chance to try to reproduce this. was focused last week more with this https://github.com/permitio/opal/discussions/671 as it affects us more.

I'll let you know the outcomes as soon as possible.

kreyyser avatar Sep 30 '24 11:09 kreyyser

Hi @gemanor, Is this issue still open??.

Girma35 avatar Jan 17 '25 10:01 Girma35

/attempt #634

Brijeshthummar02 avatar Jun 16 '25 12:06 Brijeshthummar02

/bounty $100

blocator23 avatar Jun 30 '25 19:06 blocator23

/attempt #634

krisharyan avatar Jul 02 '25 13:07 krisharyan

/attempt #634

Tibs15 avatar Jul 05 '25 23:07 Tibs15

/attempt #634

Implementation Plan

I'll fix the issue where OPAL Server doesn't clean up symbolic links when GitHub is down by:

  1. Root Cause Analysis: Investigate the git_fetcher.py module to understand how symbolic links are created during GitHub policy fetches

  2. Implement Cleanup Logic:

    • Add a cleanup mechanism in the fetch error handling path
    • Detect and remove broken symbolic links in /proc when GitHub returns 500 errors
    • Ensure cleanup happens gracefully without affecting running processes

  3. Testing Strategy:

    • Set up local OPAL server with GitHub policy source
    • Simulate GitHub downtime using network mocking (toxiproxy or similar)
    • Verify symbolic links are properly cleaned up
    • Confirm no zombie processes remain

  4. Implementation Files:

    • packages/opal-server/opal_server/git_fetcher.py - Add cleanup logic
    • Add error handling for GitHub failures
    • Implement symlink detection and removal

Expected timeline: 2-3 days for implementation and testing.

adarsh6980 avatar Nov 04 '25 14:11 adarsh6980

/attempt #634

FelipeFMA avatar Dec 01 '25 20:12 FelipeFMA