huly-selfhost icon indicating copy to clipboard operation
huly-selfhost copied to clipboard
verified publisher icon hcengineering

Workspace get stuck: Creation in progress... 100%

Open anibal-aguila opened this issue 1 year ago • 13 comments

Hi, Using traefik and additional last image of workspace the instance never start..

bb@nixops-A1:~/dpl/now/huly/ > podman-compose ps
CONTAINER ID  IMAGE                                          COMMAND               CREATED         STATUS                   PORTS                             NAMES
fc7b2dbdcaa1  docker.io/library/mongo:7-jammy                mongod                24 minutes ago  Up 24 minutes            0.0.0.0:27017->27017/tcp          mongodb
3a32e5783b13  docker.io/minio/minio:latest                   server /data --ad...  24 minutes ago  Up 24 minutes            0.0.0.0:9000-9001->9000-9001/tcp  huly_minio_1
25d609ac4639  docker.io/library/elasticsearch:7.14.2         /bin/sh -c ./bin/...  24 minutes ago  Up 24 minutes (healthy)  0.0.0.0:9200->9200/tcp            huly_elastic_1
e0c185fa1e2c  docker.io/hardcoreeng/rekoni-service:v0.6.313  node ./bundle.js      24 minutes ago  Up 24 minutes            0.0.0.0:4004->4004/tcp            huly_rekoni_1
011885ab70c3  docker.io/hardcoreeng/account:v0.6.313         node bundle.js        24 minutes ago  Up 24 minutes            0.0.0.0:3000->3000/tcp            huly_account_1
b0185fdfaf9f  docker.io/hardcoreeng/workspace:v0.6.313       node bundle.js        24 minutes ago  Up 24 minutes                                              huly_workspace_1
d90bdaef429e  docker.io/hardcoreeng/transactor:v0.6.313      /bin/sh -c node b...  24 minutes ago  Up 24 minutes                                              huly_transactor_1
b99ecf7d17ce  docker.io/hardcoreeng/collaborator:v0.6.313    node bundle.js        24 minutes ago  Up 24 minutes                                              huly_collaborator_1
98331928498f  docker.io/hardcoreeng/front:v0.6.313           node ./bundle.js      24 minutes ago  Up 24 minutes            0.0.0.0:9010->8080/tcp            huly_front_1
bb@nixops-A1:~/dpl/now/huly/ > podman ps -a | grep traefik
7b3972836d81  docker.io/library/traefik:3.1.4                      traefik               3 hours ago     Up 3 hours               0.0.0.0:22->22/tcp, 0.0.0.0:80->80/tcp, 0.0.0.0:389->389/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:9011-9012->9011-9012/tcp, 0.0.0.0:37017->37017/tcp, 0.0.0.0:63306->63306/tcp, 0.0.0.0:65432->65432/tcp  traefik-front
image image

Thanks in advance,

anibal-aguila avatar Oct 08 '24 18:10 anibal-aguila

I have a similar issue. The deployment is stuck creating at 10%. I have deployed it on K8s.

0xtejas avatar Oct 10 '24 15:10 0xtejas

Same but mine is stuck at 0%

jondef avatar Oct 10 '24 18:10 jondef

I tried v0.6.313 releases for all components and it gets stuck at 100%

image

The upper half is workspace and the lower one is account.

MongoDB output

image

MinIO is created but empty

image

0xtejas avatar Oct 10 '24 18:10 0xtejas

I checked the next day, and it has failed to create but the UI says creating. image image

0xtejas avatar Oct 11 '24 02:10 0xtejas

I am having the same issue trying to run it on k8s. I tracked the error message down to line 203 of platform/server/workspace-service/src/service.ts and it seems to be from a failed fetch request from updateWorkspaceInfo on line 166. I don't know typescript well enough to get much further than that. It could be a set up issue or it could be a bug in the code, but something is happening at the fetch request.

FireflyHacker avatar Oct 13 '24 19:10 FireflyHacker

@FireflyHacker I think update fails because of transactor pod. In workspace deployment the transactor url should be corrected. It should bews://transactor instead of ws://transactor:3333. Reason is the service is set up at port 80 by default for transactor. Or you can change the transactor service port to 3333 (same as the target port)

davelindevelt avatar Oct 14 '24 16:10 davelindevelt

Stucked also at creation in progress... 100%

`services: mongodb: image: "mongo:4.4" container_name: mongodb environment: - PUID=${PUID} - PGID=${PGID} volumes: - db:/data/db ports: - 27017:27017
restart: unless-stopped

minio: image: "minio/minio" command: server /data --address ":9000" --console-address ":9001" ports: - 9020:9020 - 9021:9021
volumes: - files:/data restart: unless-stopped

elastic: image: "elasticsearch:7.14.2" command: | /bin/sh -c "./bin/elasticsearch-plugin list | grep -q ingest-attachment || yes | ./bin/elasticsearch-plugin install --silent ingest-attachment; /usr/local/bin/docker-entrypoint.sh eswrapper" volumes: - elastic:/usr/share/elasticsearch/data ports: - 9200:9200
environment: - ELASTICSEARCH_PORT_NUMBER=${ELASTICSEARCH_PORT_NUMBER} - BITNAMI_DEBUG=${BITNAMI_DEBUG} - discovery.type=single-node - ES_JAVA_OPTS=${ES_JAVA_OPTS} - http.cors.enabled=${CORS_ENABLED} - http.cors.allow-origin=${CORS_ALLOW_ORIGIN} healthcheck: interval: 20s retries: 10 test: curl -s http://localhost:9200/_cluster/health | grep -vq '"status":"red"' restart: unless-stopped

account: image: hardcoreeng/account:${HULY_VERSION} links: - mongodb - minio ports: - 3030:3001 environment: - SERVER_PORT=3030 - SERVER_SECRET=${HULY_SECRET} - MONGO_URL=mongodb://mongodb:27017 - DB_URL=mongodb://mongodb:27017 - TRANSACTOR_URL=ws://transactor;ws://${SERVER_ADDRESS}:3333 - STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY} - FRONT_URL=http://front:8080 - MODEL_ENABLED=${MODEL_ENABLED} - ACCOUNTS_URL=http://${SERVER_ADDRESS}:3001 - ACCOUNT_PORT=${ACCOUNT_PORT} restart: unless-stopped

workspace: image: hardcoreeng/workspace:${HULY_VERSION} links: - mongodb - minio environment: - SERVER_SECRET=${HULY_SECRET} - MONGO_URL=mongodb://mongodb:27017 - DB_URL=mongodb://mongodb:27017 - TRANSACTOR_URL=ws://transactor:3334 - STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY} - MODEL_ENABLED=${MODEL_ENABLED} - ACCOUNTS_URL=http://account:3001 - NOTIFY_INBOX_ONLY=${NOTIFY_INBOX_ONLY} restart: unless-stopped

front: image: hardcoreeng/front:${HULY_VERSION} links: - mongodb - minio - elastic - collaborator - transactor ports: - 8100:8087 environment: - SERVER_PORT=8087 - SERVER_SECRET=${HULY_SECRET} - ACCOUNTS_URL=http://${SERVER_ADDRESS}:3030 - REKONI_URL=http://${SERVER_ADDRESS}:4005 - CALENDAR_URL=http://${SERVER_ADDRESS}:8095 - GMAIL_URL=http://${SERVER_ADDRESS}:8088 - TELEGRAM_URL=http://${SERVER_ADDRESS}:8086 - UPLOAD_URL=/files - ELASTIC_URL=http://elastic:9201 # Updated port - COLLABORATOR_URL=ws://${SERVER_ADDRESS}:3080 # Changed from 3078 to 3079 - STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY} - MONGO_URL=mongodb://mongodb:27017 - TITLE=${TITLE} - DEFAULT_LANGUAGE=${DEFAULT_LANGUAGE} - LAST_NAME_FIRST=${LAST_NAME_FIRST} restart: unless-stopped

collaborator: image: hardcoreeng/collaborator:${HULY_VERSION} links: - mongodb - minio - transactor ports: - 3080:3080 # Changed from 3078 to 3079 environment: - COLLABORATOR_PORT=${COLLABORATOR_PORT} # Updated port - SECRET=${HULY_SECRET} - ACCOUNTS_URL=http://account:3001 - MONGO_URL=mongodb://mongodb:27017 - STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY} restart: unless-stopped

transactor: image: hardcoreeng/transactor:${HULY_VERSION} links: - mongodb - elastic - minio - account ports: - 3334:3334 environment: - SERVER_PORT=3334 - SERVER_SECRET=${HULY_SECRET} - SERVER_CURSOR_MAXTIMEMS=30000 - ELASTIC_URL=http://elastic:9201
- ELASTIC_INDEX_NAME=${ELASTIC_INDEX_NAME} - MONGO_URL=mongodb://mongodb:27017 - DB_URL=mongodb://mongodb:27017 - METRICS_CONSOLE=${METRICS_CONSOLE} - METRICS_FILE=${METRICS_FILE} - STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY} - REKONI_URL=http://rekoni:4004 - FRONT_URL=http://${SERVER_ADDRESS}:8088
- ACCOUNTS_URL=http://account:3001 - LAST_NAME_FIRST=${LAST_NAME_FIRST} restart: unless-stopped

rekoni: image: hardcoreeng/rekoni-service:${HULY_VERSION} ports: - 4004:4004
environment: - SECRET=${HULY_SECRET} deploy: resources: limits: memory: 500M restart: unless-stopped

volumes: db: files: elastic: etcd:`

{ "result": [ { "workspace": "test222222222", "workspaceUrl": "test222222222", "version": { "major": 0, "minor": 0, "patch": 0 }, "branding": "huly", "workspaceName": "test222222222", "disabled": true, "region": "", "mode": "pending-creation", "progress": 0, "createdOn": 1729005506907, "lastVisit": 1729005567288, "createdBy": "test2", "lastProcessingTime": 0, "attempts": 0, "workspaceId": "w-test2-test22222222-670e87c2-59534b17fb-ca746a" } ] }

here is the netstat of the workspace

Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.11:33615 0.0.0.0:* LISTEN tcp6 0 0 :::3334 :::* LISTEN udp 0 0 127.0.0.11:59903 0.0.0.0:*

drlshx avatar Oct 15 '24 15:10 drlshx

@F04C You are using SERVICE_PORT=3334 for transactor. In the account part you need to set TRANSACTOR_URL=ws://transactor:3334;ws://${SERVER_ADDRESS}:3334 similar to workspace.

davelindevelt avatar Oct 15 '24 18:10 davelindevelt

Thank you ❤️

drlshx avatar Oct 16 '24 00:10 drlshx

@muradbozik You were right. I tried changing everything from service.huly.example to service.huly.domain.com in the yaml files but that did not work. Changing all the ingress files to just the name of the service and changing the config to just the external IP from my load balancer seems to have resolved any issues (at least for v0.6.325). Oh and I had to add a postgres container to the k8s deployment.

Thank you for the advice!

FireflyHacker avatar Oct 17 '24 22:10 FireflyHacker

Hi,

Running in k8s, with traefik, and got same issue, here are all errors from all pods in the namespace:

Errors in logs for pod: front-654c8ff4db-jdnhf
----------------------------------------
Errors in logs for pod: collaborator-66cfc4d6db-rtvfz
----------------------------------------
Errors in logs for pod: rekoni-5fb7cb649f-vnprn
----------------------------------------
Errors in logs for pod: elastic-b654c467b-zn4vj
{"type": "server", "timestamp": "2024-11-12T00:05:11,077Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "elastic-b654c467b-zn4vj", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-1199020055595096545, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms1024m, -Xmx1024m, -XX:MaxDirectMemorySize=536870912, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
----------------------------------------
Errors in logs for pod: minio-68bf748cf7-g4dwk
----------------------------------------
Errors in logs for pod: workspace-f7bc9fb5d-rlz79
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T03:28:41.966Z","workspace":"w-cosmin-uhl-6732c779-554176d846-f8cc35","workspaceName":"UHL"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300287,"timestamp":"2024-11-12T03:28:41.967Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-uhl-6732c779-554176d846-f8cc35","workspaceName":"UHL"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T03:33:42.310Z","workspace":"w-cosmin-news-6732c829-554176d846-f8cc37","workspaceName":"News"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300322,"timestamp":"2024-11-12T03:33:42.311Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-news-6732c829-554176d846-f8cc37","workspaceName":"News"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T03:38:42.641Z","workspace":"w-cosmin-news-6732c829-554176d846-f8cc37","workspaceName":"News"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300302,"timestamp":"2024-11-12T03:38:42.643Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-news-6732c829-554176d846-f8cc37","workspaceName":"News"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T03:43:42.980Z","workspace":"w-cosmin-news-6732c829-554176d846-f8cc37","workspaceName":"News"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300312,"timestamp":"2024-11-12T03:43:42.982Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-news-6732c829-554176d846-f8cc37","workspaceName":"News"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T03:48:43.341Z","workspace":"w-cosmin-news-6732c829-554176d846-f8cc37","workspaceName":"News"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300327,"timestamp":"2024-11-12T03:48:43.343Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-news-6732c829-554176d846-f8cc37","workspaceName":"News"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T03:53:43.655Z","workspace":"w-cosmin-uhl-6732c779-554176d846-f8cc35","workspaceName":"UHL"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300278,"timestamp":"2024-11-12T03:53:43.656Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-uhl-6732c779-554176d846-f8cc35","workspaceName":"UHL"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T03:58:43.978Z","workspace":"w-cosmin-uhl-6732c779-554176d846-f8cc35","workspaceName":"UHL"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300299,"timestamp":"2024-11-12T03:58:43.981Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-uhl-6732c779-554176d846-f8cc35","workspaceName":"UHL"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:03:44.305Z","workspace":"w-cosmin-uhl-6732c779-554176d846-f8cc35","workspaceName":"UHL"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300290,"timestamp":"2024-11-12T04:03:44.306Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-uhl-6732c779-554176d846-f8cc35","workspaceName":"UHL"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:13:49.553Z","workspace":"w-cosmin-omnis-6732c6a4-554176d846-f8cc32","workspaceName":"Omnis"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300279,"timestamp":"2024-11-12T04:13:49.554Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-omnis-6732c6a4-554176d846-f8cc32","workspaceName":"Omnis"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:18:49.881Z","workspace":"w-kosmos-nono-6732cad4-554176d846-f8cc3a","workspaceName":"Nono"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300293,"timestamp":"2024-11-12T04:18:49.883Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-kosmos-nono-6732cad4-554176d846-f8cc3a","workspaceName":"Nono"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:23:50.223Z","workspace":"w-kosmos-nono-6732cad4-554176d846-f8cc3a","workspaceName":"Nono"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300306,"timestamp":"2024-11-12T04:23:50.225Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-kosmos-nono-6732cad4-554176d846-f8cc3a","workspaceName":"Nono"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:28:50.565Z","workspace":"w-kosmos-nono-6732cad4-554176d846-f8cc3a","workspaceName":"Nono"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300305,"timestamp":"2024-11-12T04:28:50.566Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-kosmos-nono-6732cad4-554176d846-f8cc3a","workspaceName":"Nono"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:33:50.895Z","workspace":"w-kosmos-nono-6732cad4-554176d846-f8cc3a","workspaceName":"Nono"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300301,"timestamp":"2024-11-12T04:33:50.896Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-kosmos-nono-6732cad4-554176d846-f8cc3a","workspaceName":"Nono"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:38:51.222Z","workspace":"w-cosmin-omnis-6732c6a4-554176d846-f8cc32","workspaceName":"Omnis"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300297,"timestamp":"2024-11-12T04:38:51.223Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-omnis-6732c6a4-554176d846-f8cc32","workspaceName":"Omnis"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:43:51.548Z","workspace":"w-cosmin-omnis-6732c6a4-554176d846-f8cc32","workspaceName":"Omnis"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300294,"timestamp":"2024-11-12T04:43:51.550Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-omnis-6732c6a4-554176d846-f8cc32","workspaceName":"Omnis"}
{"err":{"message":"fetch failed","stack":"TypeError: fetch failed\n    at node:internal/deps/undici/undici:13185:13\n    at async updateWorkspaceInfo (/usr/src/app/bundle.js:240:528740)\n    at async /usr/src/app/bundle.js:240:527909\n    at async $4t._createWorkspace (/usr/src/app/bundle.js:453:426549)\n    at async $4t.doWorkspaceOperation (/usr/src/app/bundle.js:453:428206)\n    at async /usr/src/app/bundle.js:453:425576\n    at async $4t.exec (/usr/src/app/bundle.js:453:425737)"},"level":"error","message":"error","timestamp":"2024-11-12T04:48:51.865Z","workspace":"w-cosmin-omnis-6732c6a4-554176d846-f8cc32","workspaceName":"Omnis"}
{"level":"error","message":"---CREATE-FAILED---------","region":"","time":300288,"timestamp":"2024-11-12T04:48:51.867Z","version":{"major":0,"minor":6,"patch":355},"workspace":"w-cosmin-omnis-6732c6a4-554176d846-f8cc32","workspaceName":"Omnis"}
----------------------------------------
Errors in logs for pod: mongodb-6d55cc4cc9-znv4l
{"t":{"$date":"2024-11-12T00:04:55.237+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7294M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],"}}
{"t":{"$date":"2024-11-12T01:35:30.348+00:00"},"s":"I",  "c":"EXECUTOR", "id":22988,   "ctx":"conn11","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL handshake received but server is started without SSL support"},"remote":"5.101.0.66:60000","connectionId":11}}
{"t":{"$date":"2024-11-12T01:43:04.676+00:00"},"s":"I",  "c":"EXECUTOR", "id":22988,   "ctx":"conn12","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL handshake received but server is started without SSL support"},"remote":"52.189.74.154:50062","connectionId":12}}
{"t":{"$date":"2024-11-12T17:41:01.850+00:00"},"s":"I",  "c":"EXECUTOR", "id":22988,   "ctx":"conn46","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL handshake received but server is started without SSL support"},"remote":"45.84.89.3:63211","connectionId":46}}
----------------------------------------
Errors in logs for pod: account-55c7d486b4-p7dvn
{"err":{"message":"Not enough or too many segments","stack":"Error: Not enough or too many segments\n    at jwt_decode (/usr/src/app/bundle.js:17661:15)\n    at decodeToken (/usr/src/app/bundle.js:17820:50)\n    at decodeToken (/usr/src/app/bundle.js:145760:52)\n    at selectWorkspace (/usr/src/app/bundle.js:145785:28)\n    at /usr/src/app/bundle.js:147119:22\n    at /usr/src/app/bundle.js:183611:33\n    at _MeasureMetricsContext.with (/usr/src/app/bundle.js:15521:25)\n    at /usr/src/app/bundle.js:183608:45\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"},"level":"error","message":"error","status":{"code":"platform:status:InternalServerError","params":{},"severity":"ERROR"},"timestamp":"2024-11-12T03:14:02.103Z"}
{"err":{"message":"Not enough or too many segments","stack":"Error: Not enough or too many segments\n    at jwt_decode (/usr/src/app/bundle.js:17661:15)\n    at decodeToken (/usr/src/app/bundle.js:17820:50)\n    at decodeToken (/usr/src/app/bundle.js:145760:52)\n    at selectWorkspace (/usr/src/app/bundle.js:145785:28)\n    at /usr/src/app/bundle.js:147119:22\n    at /usr/src/app/bundle.js:183611:33\n    at _MeasureMetricsContext.with (/usr/src/app/bundle.js:15521:25)\n    at /usr/src/app/bundle.js:183608:45\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"},"level":"error","message":"error","status":{"code":"platform:status:InternalServerError","params":{},"severity":"ERROR"},"timestamp":"2024-11-12T03:24:12.674Z"}
----------------------------------------
Errors in logs for pod: transactor-8d4cb865d-nxxc6
{"level":"error","message":"invalid token","timestamp":"2024-11-12T00:18:15.309Z"}
----------------------------------------
cosmic@Revenger Devs %

Any ideas what to look into is appreciated.

uhlhosting avatar Nov 12 '24 19:11 uhlhosting

@uhlhosting Check my PR

0xtejas avatar Nov 13 '24 05:11 0xtejas

when will this PR be merged?

uhlhosting avatar Nov 21 '24 13:11 uhlhosting