Pixie Self hosted 400 on Login

[nicolastakashi]

Describe the bug I used this Pixie tutorial to set up a Self Hosted Environment and I have all the Pods up and running healthy as you can see.

When I access https://dev.withpixie.dev/ I can see the login page as the image below.

When I click on the Login button I got an error.

I tried to find anything related but I couldn't, does anyone have an idea what is this?

To Reproduce Steps to reproduce the behavior:

1. Install pixie sefl hosted
2. Click on login
3. See error

Expected behavior Be able to log in on self hosted environment

Screenshots If applicable, add screenshots to help explain your problem. Please make sure the screenshot does not contain any sensitive information such as API keys or access tokens.

Logs Please attach the logs by running the following command:

./px collect-logs

App information (please complete the following information):

• Pixie version: Latest
• K8s cluster version: 1.23
• Node Kernel version
• Browser version: Chrome Mac OS

Additional context Add any other context about the problem here.

[htroisi]

@nicolastakashi could you try clearing out the cookies for this page and reloading it?

[nicolastakashi]

Hi, @htroisi still the same issue. I tried also using the anonymous mode on Chrome.

[jadiaheno]

Same behaviour here. Used git release release/cloud/prod/1668540618 for a working version. No ingress files in that one.

[aimichelle]

Hi all! We just made a release with a bugfix to address an issue where the admin user wasn't getting created properly. Would you all mind confirming that you no longer run into this in the latest release? Make sure you pull the latest from the repo, and walk through the instructions for deploying the cloud again.

Do note, you should clear both cookies and session storage, without clearing both of those you will hit the securecookie error.

[harishkashyap569]

@aimichelle i’m still facing this issue with the latest code. When I looked at the cloud proxy logs it says “104: connection reset by peer”

[aimichelle]

Hi @harishkashyap569 , would you mind sharing the logs from your kratos and hydra pods as well? Since this is an issue that occurs when you click "login" it seems to be related to those auth modules.

[harishkashyap569]

@aimichelle PFA logs.zip

[harishkashyap569]

@aimichelle I can confirm the UI works for version release/cloud/prod/1683050579 but not if I deploy with the latest tagged images.

[rex422]

I also encountered the same problem, about the response 400 error, did you find the reason for this part?

[FuriousBottle]

Hi, I'm also seeing this same problem, with very similar logs to @harishkashyap569 with the current latest version. I'm in an air-gapped environment, and I've tried with both the air-gapped installation guide, and modifying the kustomize files within the self-hosted guide to point to my private repository.

With both approaches I'm getting the 400 response. I haven't been able to work out what combination of image versions release/cloud/prod/1683050579 should use to be able to put them in my private repository. Any chance anyone can shed some light?

[aakosarev]

Hi, I'm also seeing this same problem, with very similar logs to @harishkashyap569 with the current latest version. I'm in an air-gapped environment, and I've tried with both the air-gapped installation guide, and modifying the kustomize files within the self-hosted guide to point to my private repository.

With both approaches I'm getting the 400 response. I haven't been able to work out what combination of image versions release/cloud/prod/1683050579 should use to be able to put them in my private repository. Any chance anyone can shed some light?

Hi! Did you manage to solve this problem?

[aakosarev]

Hi, @htroisi still the same issue. I tried also using the anonymous mode on Chrome.

Hi! Did you manage to solve this problem?

[FuriousBottle]

Hi @aakosarev, I ended up using the images below in the installation, which worked for me: "ghcr.io/pixie-io/cloud-config_manager_server_image:0.1.6" "ghcr.io/pixie-io/cloud-cron_script_server_image:0.1.6" "ghcr.io/pixie-io/cloud-indexer_server_image:0.1.6" "ghcr.io/pixie-io/cloud-metrics_server_image:0.1.6" "ghcr.io/pixie-io/cloud-plugin/load_db:0.1.6" "ghcr.io/pixie-io/cloud-plugin_server_image:0.1.6" "ghcr.io/pixie-io/cloud-profile_server_image:0.1.6" "ghcr.io/pixie-io/cloud-project_manager_server_image:0.1.6" "ghcr.io/pixie-io/cloud-proxy_server_image:0.1.6" "ghcr.io/pixie-io/cloud-scriptmgr_server_image:0.1.6" "ghcr.io/pixie-io/cloud-vzconn_server_image:0.1.6" "ghcr.io/pixie-io/cloud-vzmgr_server_image:0.1.6" "docker.elastic.co/eck/eck-operator:1.8.0@sha256:61d809705b3d65710d12e78a7e76cc2ff195fbd3bb3986d4df2fb80ebd29956a" "envoyproxy/envoy:v1.12.2@sha256:b36ee021fc4d285de7861dbaee01e7437ce1d63814ead6ae3e4dfcad4a951b2e" "gcr.io/pixie-oss/pixie-dev-public/curl:multiarch-7.87.0@sha256:f7f265d5c64eb4463a43a99b6bf773f9e61a50aaa7cefaf564f43e42549a01dd" "gcr.io/pixie-oss/pixie-dev-public/elasticsearch:7.6.0-patched1@sha256:f734909115be9dba66736c4b7356fd52da58b1ffdb895ba74cb5c2fca2b133dd" "gcr.io/pixie-oss/pixie-prod/vizier-deps/nats:2.9.19-scratch@sha256:5de59286eb54ead4d4a9279846098d4097b9c17a3c0588182398a7250cde1af9" "oryd/hydra:v1.9.2-alpine@sha256:faa6ca02e77e0a08f66bfa7470a5e06d80e6e68c9c35410c65a4ea7b501aea61" "oryd/hydra:v1.9.2-sqlite@sha256:61771c706934e1ffd66f86700a28a294ce4ed150fbf30cc131710924271a5871" "oryd/kratos:v0.10.1@sha256:fdcfac3da3b64e619af553451607e1ab00160e59860bb19ec145cdc6f6f9c41d" "postgres:14-alpine@sha256:446abaf8831c54f57212c0ae52f5df84e69eeb2767e2376d07bed9c9742b1243"

[aakosarev]

Hi @aakosarev, I ended up using the images below in the installation, which worked for me: "ghcr.io/pixie-io/cloud-config_manager_server_image:0.1.6" "ghcr.io/pixie-io/cloud-cron_script_server_image:0.1.6" "ghcr.io/pixie-io/cloud-indexer_server_image:0.1.6" "ghcr.io/pixie-io/cloud-metrics_server_image:0.1.6" "ghcr.io/pixie-io/cloud-plugin/load_db:0.1.6" "ghcr.io/pixie-io/cloud-plugin_server_image:0.1.6" "ghcr.io/pixie-io/cloud-profile_server_image:0.1.6" "ghcr.io/pixie-io/cloud-project_manager_server_image:0.1.6" "ghcr.io/pixie-io/cloud-proxy_server_image:0.1.6" "ghcr.io/pixie-io/cloud-scriptmgr_server_image:0.1.6" "ghcr.io/pixie-io/cloud-vzconn_server_image:0.1.6" "ghcr.io/pixie-io/cloud-vzmgr_server_image:0.1.6" "docker.elastic.co/eck/eck-operator:1.8.0@sha256:61d809705b3d65710d12e78a7e76cc2ff195fbd3bb3986d4df2fb80ebd29956a" "envoyproxy/envoy:v1.12.2@sha256:b36ee021fc4d285de7861dbaee01e7437ce1d63814ead6ae3e4dfcad4a951b2e" "gcr.io/pixie-oss/pixie-dev-public/curl:multiarch-7.87.0@sha256:f7f265d5c64eb4463a43a99b6bf773f9e61a50aaa7cefaf564f43e42549a01dd" "gcr.io/pixie-oss/pixie-dev-public/elasticsearch:7.6.0-patched1@sha256:f734909115be9dba66736c4b7356fd52da58b1ffdb895ba74cb5c2fca2b133dd" "gcr.io/pixie-oss/pixie-prod/vizier-deps/nats:2.9.19-scratch@sha256:5de59286eb54ead4d4a9279846098d4097b9c17a3c0588182398a7250cde1af9" "oryd/hydra:v1.9.2-alpine@sha256:faa6ca02e77e0a08f66bfa7470a5e06d80e6e68c9c35410c65a4ea7b501aea61" "oryd/hydra:v1.9.2-sqlite@sha256:61771c706934e1ffd66f86700a28a294ce4ed150fbf30cc131710924271a5871" "oryd/kratos:v0.10.1@sha256:fdcfac3da3b64e619af553451607e1ab00160e59860bb19ec145cdc6f6f9c41d" "postgres:14-alpine@sha256:446abaf8831c54f57212c0ae52f5df84e69eeb2767e2376d07bed9c9742b1243"

Thank you very much!

[yuval0499]

Hey, any update on that? Deployed pixie air gapped, currently the login page gives me 400. I suspect from logs it's try to communicate with the Kratos service on http, and the Kratos not responding cause he accept only https.

*My cloud proxy service is ClusterIP and not LoadBalancer, using nginx ingress to expose it.

Iv'e followed the instructions carefully and modified any required value, including the parameters that in the production readiness page.

Anyone solved this kind of behavior?

[aakosarev]

hi, I used this release and everything worked: release/cloud/prod/1683050579

пн, 30 окт. 2023 г., 13:27 yuval0499 @.***>:

Hey, any update on that? Deployed pixie air gapped, currently the login page gives me 400. I suspect from logs it's try to communicate with the Kratos service on http, and the Kratos not responding cause he accept only https.

*My cloud proxy service is ClusterIP and not LoadBalancer, using nginx ingress to expose it.

Iv'e followed the instructions carefully and modified any required value, including the parameters that in the production readiness page.

Anyone experienced this behavior?

— Reply to this email directly, view it on GitHub https://github.com/pixie-io/pixie/issues/704#issuecomment-1784893796, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARWE3ROECXUSJCRS42L55ADYB56IRAVCNFSM6AAAAAAUHVYRLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUHA4TGNZZGY . You are receiving this because you were mentioned.Message ID: @.***>

[yuval0499]

hi, I used this release and everything worked: release/cloud/prod/1683050579 пн, 30 окт. 2023 г., 13:27 yuval0499 @.*>: … Hey, any update on that? Deployed pixie air gapped, currently the login page gives me 400. I suspect from logs it's try to communicate with the Kratos service on http, and the Kratos not responding cause he accept only https. My cloud proxy service is ClusterIP and not LoadBalancer, using nginx ingress to expose it. Iv'e followed the instructions carefully and modified any required value, including the parameters that in the production readiness page. Anyone experienced this behavior? — Reply to this email directly, view it on GitHub <#704 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARWE3ROECXUSJCRS42L55ADYB56IRAVCNFSM6AAAAAAUHVYRLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUHA4TGNZZGY . You are receiving this because you were mentioned.Message ID: @.>

Thank you! Have you used the same images as the latest version ? The image list on that tag referring to the latest, so it's hard to tell which images was the latest when that tag released. https://storage.googleapis.com/pixie-dev-public/cloud/1683050579/pixie_cloud.tar.gz

[aakosarev]

wherever the tag is specified :latest - replace with :1683050579

:)

пн, 30 окт. 2023 г., 14:55 yuval0499 @.***>:

hi, I used this release and everything worked: release/cloud/prod/1683050579 пн, 30 окт. 2023 г., 13:27 yuval0499 @.*

>: … <#m_-3210006685203968843_> Hey, any update on that? Deployed pixie air gapped, currently the login page gives me 400. I suspect from logs it's try to communicate with the Kratos service on http, and the Kratos not responding cause he accept only https. My cloud proxy service is ClusterIP and not LoadBalancer, using nginx ingress to expose it. Iv'e followed the instructions carefully and modified any required value, including the parameters that in the production readiness page. Anyone experienced this behavior? — Reply to this email directly, view it on GitHub <#704 (comment) https://github.com/pixie-io/pixie/issues/704#issuecomment-1784893796>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARWE3ROECXUSJCRS42L55ADYB56IRAVCNFSM6AAAAAAUHVYRLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUHA4TGNZZGY https://github.com/notifications/unsubscribe-auth/ARWE3ROECXUSJCRS42L55ADYB56IRAVCNFSM6AAAAAAUHVYRLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUHA4TGNZZGY . You are receiving this because you were mentioned.Message ID: @.>

Thank you! Have you used the same images as the latest version ? The image list on that tag referring to the latest, so it's hard to tell which images was the latest when that tag released.

https://storage.googleapis.com/pixie-dev-public/cloud/1683050579/pixie_cloud.tar.gz

— Reply to this email directly, view it on GitHub https://github.com/pixie-io/pixie/issues/704#issuecomment-1785029396, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARWE3RMX2MCDNY3WSFRM6M3YB6IS5AVCNFSM6AAAAAAUHVYRLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBVGAZDSMZZGY . You are receiving this because you were mentioned.Message ID: @.***>