unstract icon indicating copy to clipboard operation
unstract copied to clipboard
verified publisher icon Zipstack

fix: [ISSUE] Unable to configure unstract (mock_org/onboard): unable to submit form

Open gaetanquentin opened this issue 9 months ago • 8 comments

Describe the bug

after loggin in, in "you're 4 steps away from nirvana', test is ok but when i press submit button it displays a popup :

Image

Image

Image

for every configurations

To reproduce

docker installation with docker compose.

VERSION=dev docker compose -f docker-compose.build.yaml build VERSION=dev docker compose -f docker-compose.yaml up -d

installed on a server X (not localhost) frontend.unstract.mydomain.intr declared in dns

updates of backend .env:

DJANGO_APP_BACKEND_URL=http://frontend.unstract.mydomain.intr


LOGIN_NEXT_URL="http://frontend.unstract.mydomain.intr/org"
LANDING_URL="http://frontend.unstract.mydomain.intr/landing"
ERROR_URL="http://frontend.unstract.mydomain.intr/error"
WEB_APP_ORIGIN_URL="http://frontend.unstract.mydomain.intr"


# Default user auth credentials
DEFAULT_AUTH_USERNAME="xxxx"
DEFAULT_AUTH_PASSWORD="yyyy"

using proxy_overrides.yaml:

http:
  routers:
    frontend:
      service: frontend
      rule: Host(`frontend.unstract.mydomain.intr`)
    backend:
      service: backend
      rule: Host(`frontend.unstract.mydomain.intr`) && PathPrefix(`/api/v1`, `/deployment`)

  services:
    frontend:
      loadBalancer:
        servers:
        # Point to service running on docker host.
        - url: http://host.docker.internal:3000/
        passHostHeader: true
    backend:
      loadBalancer:
        servers:
        # Point to service running on docker host.
        - url: http://host.docker.internal:8000/
        passHostHeader: true

Expected behavior

configuration should be accepted and written

Environment details

  • Version: dev (15 march 2025)

Additional context

Logs:

unstract-backend                | [2025-03-16 11:27:08 +0000] [19] [DEBUG] GET /api/v1/socket/
unstract-proxy                  | 172.16.99.219 - - [16/Mar/2025:11:27:08 +0000] "GET /api/v1/socket/?EIO=4&transport=websocket HTTP/1.1" 400 25 "-" "-" 8894 "backend@file" "http://host.docker.internal:8000/" 5ms
unstract-backend                | 172.20.0.1 - - [16/Mar/2025:11:27:08 +0000] "GET /api/v1/socket/?EIO=4&transport=websocket HTTP/1.1" 400 25 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36"
unstract-celery-beat            | INFO : [2025-03-16 11:27:11,001]{module:base process:1 thread:135973213555584 request_id:none} :- DB connection (ID: 135972218967680) is established or reused.
unstract-backend                | [2025-03-16 11:27:11 +0000] [19] [DEBUG] POST /api/v1/unstract/mock_org/adapter/
unstract-backend                | INFO : [2025-03-16 11:27:11,499]{module:base process:19 thread:137646667990720 request_id:35a14ba52f5941af84137c0cb9c7c5be} :- DB connection (ID: 137646002629504) is established or reused.
unstract-backend                | ERROR : [2025-03-16 11:27:11,504]{module:exception process:19 thread:137646667990720 request_id:35a14ba52f5941af84137c0cb9c7c5be} :- POST http://frontend.unstract.mydomain.intr/api/v1/unstract/mock_org/adapter/ 500
unstract-backend                | 
unstract-backend                | ValueError('Fernet key must be 32 url-safe base64-encoded bytes.')
unstract-backend                | 
unstract-backend                | ````Traceback (most recent call last):
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/cryptography/fernet.py", line 34, in __init__
unstract-backend                |     key = base64.urlsafe_b64decode(key)
unstract-backend                |   File "/usr/local/lib/python3.9/base64.py", line 133, in urlsafe_b64decode
unstract-backend                |     return b64decode(s)
unstract-backend                |   File "/usr/local/lib/python3.9/base64.py", line 87, in b64decode
unstract-backend                |     return binascii.a2b_base64(s)
unstract-backend                | binascii.Error: Incorrect padding
unstract-backend                | 
unstract-backend                | The above exception was the direct cause of the following exception:
unstract-backend                | 
unstract-backend                | Traceback (most recent call last):
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
unstract-backend                |     response = handler(request, *args, **kwargs)
unstract-backend                |   File "/app/adapter_processor_v2/views.py", line 180, in create
unstract-backend                |     serializer.is_valid(raise_exception=True)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/serializers.py", line 227, in is_valid
unstract-backend                |     self._validated_data = self.run_validation(self.initial_data)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/serializers.py", line 426, in run_validation
unstract-backend                |     value = self.to_internal_value(data)
unstract-backend                |   File "/app/adapter_processor_v2/serializers.py", line 51, in to_internal_value
unstract-backend                |     f: Fernet = Fernet(encryption_secret.encode("utf-8"))
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/cryptography/fernet.py", line 36, in __init__
unstract-backend                |     raise ValueError(
unstract-backend                | ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
unstract-backend                | ````
unstract-backend                | Internal Server Error: /api/v1/unstract/mock_org/adapter/
unstract-backend                | Traceback (most recent call last):
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/cryptography/fernet.py", line 34, in __init__
unstract-backend                |     key = base64.urlsafe_b64decode(key)
unstract-backend                |   File "/usr/local/lib/python3.9/base64.py", line 133, in urlsafe_b64decode
unstract-backend                |     return b64decode(s)
unstract-backend                |   File "/usr/local/lib/python3.9/base64.py", line 87, in b64decode
unstract-backend                |     return binascii.a2b_base64(s)
unstract-backend                | binascii.Error: Incorrect padding
unstract-backend                | 
unstract-backend                | The above exception was the direct cause of the following exception:
unstract-backend                | 
unstract-backend                | Traceback (most recent call last):
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
unstract-backend                |     response = get_response(request)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
unstract-backend                |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
unstract-backend                |     return view_func(*args, **kwargs)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
unstract-backend                |     return self.dispatch(request, *args, **kwargs)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
unstract-backend                |     response = self.handle_exception(exc)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
unstract-backend                |     self.raise_uncaught_exception(exc)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
unstract-backend                |     raise exc
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
unstract-backend                |     response = handler(request, *args, **kwargs)
unstract-backend                |   File "/app/adapter_processor_v2/views.py", line 180, in create
unstract-backend                |     serializer.is_valid(raise_exception=True)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/serializers.py", line 227, in is_valid
unstract-backend                |     self._validated_data = self.run_validation(self.initial_data)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/serializers.py", line 426, in run_validation
unstract-backend                |     value = self.to_internal_value(data)
unstract-backend                |   File "/app/adapter_processor_v2/serializers.py", line 51, in to_internal_value
unstract-backend                |     f: Fernet = Fernet(encryption_secret.encode("utf-8"))
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/cryptography/fernet.py", line 36, in __init__
unstract-backend                |     raise ValueError(
unstract-backend                | ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
unstract-backend                | ERROR : [2025-03-16 11:27:11,560]{module:log process:19 thread:137646667990720 request_id:35a14ba52f5941af84137c0cb9c7c5be} :- Internal Server Error: /api/v1/unstract/mock_org/adapter/
unstract-backend                | Traceback (most recent call last):
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/cryptography/fernet.py", line 34, in __init__
unstract-backend                |     key = base64.urlsafe_b64decode(key)
unstract-backend                |   File "/usr/local/lib/python3.9/base64.py", line 133, in urlsafe_b64decode
unstract-backend                |     return b64decode(s)
unstract-backend                |   File "/usr/local/lib/python3.9/base64.py", line 87, in b64decode
unstract-backend                |     return binascii.a2b_base64(s)
unstract-backend                | binascii.Error: Incorrect padding
unstract-backend                | 
unstract-backend                | The above exception was the direct cause of the following exception:
unstract-backend                | 
unstract-backend                | Traceback (most recent call last):
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
unstract-backend                |     response = get_response(request)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 197, in _get_response
unstract-backend                |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
unstract-backend                |     return view_func(*args, **kwargs)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
unstract-backend                |     return self.dispatch(request, *args, **kwargs)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
unstract-backend                |     response = self.handle_exception(exc)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
unstract-backend                |     self.raise_uncaught_exception(exc)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
unstract-backend                |     raise exc
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
unstract-backend                |     response = handler(request, *args, **kwargs)
unstract-backend                |   File "/app/adapter_processor_v2/views.py", line 180, in create
unstract-backend                |     serializer.is_valid(raise_exception=True)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/serializers.py", line 227, in is_valid
unstract-backend                |     self._validated_data = self.run_validation(self.initial_data)
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/rest_framework/serializers.py", line 426, in run_validation
unstract-backend                |     value = self.to_internal_value(data)
unstract-backend                |   File "/app/adapter_processor_v2/serializers.py", line 51, in to_internal_value
unstract-backend                |     f: Fernet = Fernet(encryption_secret.encode("utf-8"))
unstract-backend                |   File "/app/.venv/lib/python3.9/site-packages/cryptography/fernet.py", line 36, in __init__
unstract-backend                |     raise ValueError(
unstract-backend                | ValueError: Fernet key must be 32 url-safe base64-encoded bytes.
unstract-backend                | 172.20.0.1 - - [16/Mar/2025:11:27:11 +0000] "POST /api/v1/unstract/mock_org/adapter/ HTTP/1.1" 500 145556 "http://frontend.unstract.mydomain.intr/mock_org/onboard" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
unstract-proxy                  | 172.16.99.6 - - [16/Mar/2025:11:27:11 +0000] "POST /api/v1/unstract/mock_org/adapter/ HTTP/1.1" 500 145556 "-" "-" 8895 "backend@file" "http://host.docker.internal:8000/" 87ms
unstract-backend                | [2025-03-16 11:27:11 +0000] [19] [DEBUG] POST /api/v1/unstract/mock_org/logs/
unstract-backend                | INFO : [2025-03-16 11:27:11,611]{module:base process:19 thread:137646540064448 request_id:94bec21c5c3f465da0392a5fc3d9147e} :- DB connection (ID: 137646002629504) is established or reused.
unstract-proxy                  | 172.16.99.6 - - [16/Mar/2025:11:27:11 +0000] "POST /api/v1/unstract/mock_org/logs/ HTTP/1.1" 200 54 "-" "-" 8896 "backend@file" "http://host.docker.internal:8000/" 18ms
unstract-backend                | 172.20.0.1 - - [16/Mar/2025:11:27:11 +0000] "POST /api/v1/unstract/mock_org/logs/ HTTP/1.1" 200 54 "http://frontend.unstract.mydomain.intr/mock_org/onboard" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
unstract-backend                | [2025-03-16 11:27:11 +0000] [19] [DEBUG] GET /api/v1/socket/
unstract-proxy                  | 172.16.99.6 - - [16/Mar/2025:11:27:11 +0000] "GET /api/v1/socket/?EIO=4&transport=websocket HTTP/1.1" 400 25 "-" "-" 8897 "backend@file" "http://host.docker.internal:8000/" 6ms
unstract-backend                | 172.20.0.1 - - [16/Mar/2025:11:27:11 +0000] "GET /api/v1/socket/?EIO=4&transport=websocket HTTP/1.1" 400 25 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
unstract-backend                | [2025-03-16 11:27:13 +0000] [19] [DEBUG] GET /api/v1/socket/
unstract-proxy                  | 172.16.99.6 - - [16/Mar/2025:11:27:13 +0000] "GET /api/v1/socket/?EIO=4&transport=websocket HTTP/1.1" 400 25 "-" "-" 8898 "backend@file" "http://host.docker.internal:8000/" 4ms
unstract-backend                | 172.20.0.1 - - [16/Mar/2025:11:27:13 +0000] "GET /api/v1/socket/?EIO=4&transport=websocket HTTP/1.1" 400 25 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
unstract-backend                | [2025-03-16 11:27:13 +0000] [19] [DEBUG] GET /api/v1/socket/
unstract-proxy                  | 172.16.99.219 - - [16/Mar/2025:11:27:13 +0000] "GET /api/v1/socket/?EIO=4&transport=websocket HTTP/1.1" 400 25 "-" "-" 8899 "backend@file" "http://host.docker.internal:8000/" 4ms
unstract-backend                | 172.20.0.1 - - [16/Mar/2025:11:27:13 +0000] "GET /api/v1/socket/?EIO=4&transport=websocket HTTP/1.1" 400 25 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36

Screenshots

If applicable, add screenshots to help explain your problem.

gaetanquentin avatar Mar 16 '25 11:03 gaetanquentin

@gaetanquentin looks like to be issue with ENCRYPTION_KEY . Can you check the value of this in backend/.env. Same value should be present in platform/.env file also.

Usually this won't have any issues if you use ./run-platform.sh .

And these values needs to be base 64 encoded

ritwik-g avatar Mar 16 '25 12:03 ritwik-g

@gaetanquentin looks like to be issue with ENCRYPTION_KEY . Can you check the value of this in backend/.env. Same value should be present in platform/.env file also.

Usually this won't have any issues if you use ./run-platform.sh .

And these values needs to be base 64 encoded

i think i must have missed something: i have set ENCRYPTION_KEY in base64 in the two files but the problem is still here. even when launched with run-platform.sh

here is the key i set (i will change it later): ENCRYPTION_KEY=b'rf5tkDlBckkEIoZR6MxNjh_DiptKJIurmirQxW1aJFQ=' one decoded it does 32 bytes

with bash -x run-platforms.sh i can see this not my encryption key set in .env files:

./run-platform.sh:189: python3 -c 'import secrets, base64; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())' ./run-platform.sh:189: ENCRYPTION_KEY=N5XU26l6gPk3NHAMngTzuAthvFzbIRbujpsXuP_7nAY= ./run-platform.sh:190: DEFAULT_AUTH_KEY=unstract

gaetanquentin avatar Mar 16 '25 15:03 gaetanquentin

ok so i got the encryption key with bash -x run-platforms.sh and paste it into env files and restarted the services, now it is ok. i think something is miising in doc?

gaetanquentin avatar Mar 16 '25 17:03 gaetanquentin

@gaetanquentin which was the doc that you followed for the setup?

ritwik-g avatar Mar 17 '25 11:03 ritwik-g

@gaetanquentin which was the doc that you followed for the setup?

mainly READMEs: main one, docker directory one, .env files comments, . is there a doc explaining ENCRYPTION_KEY management?

gaetanquentin avatar Mar 17 '25 12:03 gaetanquentin

https://github.com/Zipstack/unstract?tab=readme-ov-file#-backup-encryption-key

Is this what you are looking for?

ritwik-g avatar Mar 17 '25 12:03 ritwik-g

https://github.com/Zipstack/unstract?tab=readme-ov-file#-backup-encryption-key

Is this what you are looking for?

Do copy the value of ENCRYPTION_KEY config in

-> what does that mean? the ENCRYPTION_KEY is never shown by run-platform.sh, does he? only shown if i do a bash -x run-platforms.sh.

gaetanquentin avatar Mar 17 '25 13:03 gaetanquentin

Image

Copy from the .env files of the backend or platform

cc: @hari-kuriakose

ritwik-g avatar Jul 04 '25 07:07 ritwik-g