grist-core icon indicating copy to clipboard operation
grist-core copied to clipboard

Published 20 hours ago • verified publisher icon gristlabs

Suddenly unable to access our self-hosted documents

Open asitemade4u opened this issue 2 years ago • 17 comments

We have self-hosted Grist since its availability on docker. One or two weeks ago, it has become impossible for us to connect to our Grist server; the error is something we had encountered at the very inception of Grist:

Something went wrong
There was an error: cannot find user.

I have changed nothing to the docker-compose.yml file:

version: "3"

services:
  grist-kuq:
    image: gristlabs/grist
    container_name: grist
    restart: unless-stopped
    volumes:
      - /home/stsinc/dkr/gsk/dta:/persist
    ports:
      - 8484:8484
    environment:
      - APP_DOC_URL=https://grist.[our-internal-domain, has not changed]
      - APP_HOME_URL=https://[our-internal-domain, has not changed]
      - DEBUG=0
      - GRIST_EXPERIMENTAL_PLUGINS=true
      - GRIST_FORWARD_AUTH_LOGOUT_PATH=/signed-out
      - GRIST_HIDE_UI_ELEMENTS=helpCenter,billing,templates,multiSite,multiAccounts
      - GRIST_HOST=0.0.0.0
      - GRIST_ORG_IN_PATH=false
      - GRIST_PAGE_TITLE_SUFFIX=_blank
      - GRIST_PROXY_AUTH_HEADER=Remote-Email
      - GRIST_SANDBOX_FLAVOR=unsandboxed
      - GRIST_SESSION_SECRET=[the secret , has not changed]
      - GRIST_SINGLE_ORG=docs
      - GRIST_SUPPORT_ANON=false
      - TYPEORM_DATABASE=[database name, has not changed]
      - TYPEORM_HOST=[the URL to the database, has not changed]
      - TYPEORM_PASSWORD=[database password, has not changed]
      - TYPEORM_PORT=5432
      - TYPEORM_TYPE=postgres
      - TYPEORM_USERNAME=docker
      - TZ=America/New_York
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

# For SWAG
networks:
  default:
    name: lsio
    external: true

Here is what the container log says during the session:

Welcome to Grist.
In quiet mode, see http://localhost:8484 to use.
For full logs, re-run with DEBUG=1
Setting up database...
Database setup complete.
2023-11-30 11:18:34.847 - info: == Grist version is 1.1.8 (commit unknown)
2023-11-30 11:18:34.850 - debug: skipping incomplete language ar (set GRIST_OFFER_ALL_LANGUAGES if you want it)
2023-11-30 11:18:34.850 - debug: skipping incomplete language cs (set GRIST_OFFER_ALL_LANGUAGES if you want it)
2023-11-30 11:18:34.854 - debug: skipping incomplete language fa (set GRIST_OFFER_ALL_LANGUAGES if you want it)
2023-11-30 11:18:34.858 - debug: skipping incomplete language ja (set GRIST_OFFER_ALL_LANGUAGES if you want it)
2023-11-30 11:18:34.859 - debug: skipping incomplete language nl (set GRIST_OFFER_ALL_LANGUAGES if you want it)
2023-11-30 11:18:34.864 - debug: skipping incomplete language th (set GRIST_OFFER_ALL_LANGUAGES if you want it)
2023-11-30 11:18:34.870 - info: Loading empty config because /persist/config.json missing
2023-11-30 11:18:34.870 - warn: GRIST_PROXY_AUTH_HEADER is deprecated; interpreted as a synonym of GRIST_FORWARD_AUTH_HEADER
2023-11-30 11:18:34.887 - warn: did not find an appropriately named example workspace in deployment
2023-11-30 11:18:34.891 - info: No plugins found in directory: /grist/.grist/plugins
2023-11-30 11:18:34.895 - info: Found 2 valid plugins on the system
2023-11-30 11:18:34.895 - debug: PLUGIN builtIn/core -- /grist/plugins/core
2023-11-30 11:18:34.896 - debug: PLUGIN bundled/grist-bundled -- /grist/node_modules/@gristlabs/grist-widget/dist/plugins/grist-bundled
2023-11-30 11:18:34.899 - info: Server timeouts: keepAliveTimeout 305000 headersTimeout 306000
2023-11-30 11:18:34.901 - info: server(home,docs,static) available at 0.0.0.0:8484
2023-11-30 11:18:34.908 - warn: Failed to create GoogleAuth endpoint: GOOGLE_CLIENT_SECRET is not defined
2023-11-30 11:18:34.912 - info: Server timeouts: keepAliveTimeout 305000 headersTimeout 306000
2023-11-30 11:18:34.912 - info: pluginServer available at 0.0.0.0:45205
2023-11-30 11:18:34.913 - info: == appRoot: /grist
2023-11-30 11:18:34.913 - info: == i18:namespace: client,server
2023-11-30 11:18:34.913 - info: == docsRoot: /persist/docs
2023-11-30 11:18:34.913 - info: == defaultBaseDomain: grist.[our-domain]
2023-11-30 11:18:34.913 - info: == instanceRoot: /persist
2023-11-30 11:18:34.913 - info: == tag: unknown
2023-11-30 11:18:34.914 - info: == database: postgres://[ip of Postgres server]:5432/grist
2023-11-30 11:18:34.914 - info: == userRoot: /grist/.grist
2023-11-30 11:18:34.914 - info: == bundledRoot: /grist/node_modules/@gristlabs/grist-widget/dist
2023-11-30 11:18:34.914 - info: == loginMiddlewareComment: forward-auth
2023-11-30 11:18:34.914 - info: == docWorkerId: testDocWorkerId_8484
2023-11-30 11:18:34.914 - info: == pluginUrl: http://0.0.0.0:45205/
2023-11-30 11:18:34.914 - info: == willServePlugins: true
2023-11-30 11:18:34.914 - info: == grist.access.supportEmail: [email protected] [default] [GRIST_SUPPORT_EMAIL]
2023-11-30 11:18:34.914 - info: == grist.access.listPublicSites: false [default] [GRIST_LIST_PUBLIC_SITES]
2023-11-30 11:18:34.914 - info: == grist.integrations.sql.timeout: 1000 [default] [GRIST_SQL_TIMEOUT_MSEC]
2023-11-30 11:18:34.914 - info: == grist.integrations.allowedWebhookDomains: - [ALLOWED_WEBHOOK_DOMAINS]
2023-11-30 11:18:34.914 - info: == grist.integrations.proxy: - [GRIST_HTTPS_PROXY]
2023-11-30 11:18:34.914 - info: == grist.locale.offerAllLanguages: - [GRIST_OFFER_ALL_LANGUAGES]
2023-11-30 11:18:34.914 - info: == grist.login.system.forwardAuth.header: Remote-Email [GRIST_PROXY_AUTH_HEADER]
2023-11-30 11:18:34.914 - info: == grist.login.system.forwardAuth.active: true
2023-11-30 11:18:34.914 - info: == grist.login.system.forwardAuth.logoutPath: /signed-out [GRIST_FORWARD_AUTH_LOGOUT_PATH]
2023-11-30 11:18:34.914 - info: == grist.login.system.forwardAuth.loginPath: /auth/login [default] [GRIST_FORWARD_AUTH_LOGIN_PATH]
2023-11-30 11:18:34.914 - info: == grist.login.skipSession: - [GRIST_IGNORE_SESSION]
2023-11-30 11:18:34.914 - info: == grist.login.forced: - [GRIST_FORCE_LOGIN]
2023-11-30 11:18:34.914 - info: == grist.externalStorage.minio.bucket: - [GRIST_DOCS_MINIO_BUCKET]
2023-11-30 11:18:34.914 - info: == grist.externalStorage.disable: - [GRIST_DISABLE_S3]
2023-11-30 11:18:34.914 - info: == grist.externalStorage.active: false
2023-11-30 11:18:34.915 - info: activity docCount=15, orgCount=1, orgInGoodStandingCount=1, userCount=6, userWithLoginCount=6
2023-11-30 11:19:26.739 - debug: Auth[GET]: grist.[our-domain] / customHostSession=, method=GET, host=grist.[our-domain], path=/, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:26.861 - debug: Auth[GET]: grist.[our-domain] / customHostSession=, method=GET, host=grist.[our-domain], path=/, org=docs, [email protected], userId=1, altSessionId=wxcyoLWymz7fKJ8hnVricQ
2023-11-30 11:19:26.873 - debug: Auth[GET]: grist.[our-domain] /session/access/active customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/active, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:26.877 - debug: Auth[GET]: grist.[our-domain] /session/access/all customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/all, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:26.929 - debug: Auth[GET]: grist.[our-domain] /orgs/0/workspaces customHostSession=, method=GET, host=grist.[our-domain], path=/orgs/0/workspaces, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:31.016 - debug: Auth[GET]: grist.[our-domain] /session/access/all customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/all, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:31.028 - debug: Auth[GET]: grist.[our-domain] /session/access/active customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/active, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:32.973 - debug: Auth[GET]: grist.[our-domain] / customHostSession=, method=GET, host=grist.[our-domain], path=/, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:33.050 - debug: Auth[GET]: grist.[our-domain] /session/access/all customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/all, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:33.052 - debug: Auth[GET]: grist.[our-domain] /session/access/active customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/active, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:19:33.088 - debug: Auth[GET]: grist.[our-domain] /orgs/0/workspaces customHostSession=, method=GET, host=grist.[our-domain], path=/orgs/0/workspaces, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:20:42.445 - debug: Auth[GET]: grist.[our-domain] /welcome/home customHostSession=, method=GET, host=grist.[our-domain], path=/welcome/home, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:20:42.466 - debug: Auth[GET]: grist.[our-domain] / customHostSession=, method=GET, host=grist.[our-domain], path=/, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:20:42.566 - debug: Auth[GET]: grist.[our-domain] /session/access/all customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/all, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:20:42.569 - debug: Auth[GET]: grist.[our-domain] /session/access/active customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/active, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:20:42.614 - debug: Auth[GET]: grist.[our-domain] /orgs/0/workspaces customHostSession=, method=GET, host=grist.[our-domain], path=/orgs/0/workspaces, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:20:57.164 - debug: Auth[GET]: grist.[our-domain] /session/access/all customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/all, org=docs, [email protected], userId=1, altSessionId=[session ID]
2023-11-30 11:20:57.173 - debug: Auth[GET]: grist.[our-domain] /session/access/active customHostSession=, method=GET, host=grist.[our-domain], path=/session/access/active, org=docs, [email protected], userId=1, altSessionId=[session ID]

What have I done wrong or is it a regression?

asitemade4u avatar Nov 30 '23 16:11 asitemade4u

Hi @asitemade4u - from the mention of GRIST_PROXY_AUTH_HEADER, this may be from changes in https://github.com/gristlabs/grist-core/commit/3210eee24f72dd9e975a40316a175e1de6e18090 (cc @dsagal). You may be able to get the behavior you liked by setting also GRIST_IGNORE_SESSION=1 so that the headers are respected on all paths. Sorry for the change of behavior. There's an explanation here: https://github.com/gristlabs/grist-core/issues/750#issuecomment-1811082115

paulfitz avatar Nov 30 '23 16:11 paulfitz

@paulfitz , you're great, as always!

asitemade4u avatar Nov 30 '23 16:11 asitemade4u

But unfortunately, having added the instruction to the docker-compose.yml file, re-loaded it, and cleaned the browser cache, I still get the same error....

asitemade4u avatar Nov 30 '23 16:11 asitemade4u

Hmm, and @asitemade4u you are confident the Remote-Email header is being set by whatever is in front of Grist? @dsagal, any ideas what could be up?

paulfitz avatar Nov 30 '23 16:11 paulfitz

So, I have tried reverting to older versions of the image.

  • docker pull gristlabs/grist:1.1.7 -- NOK
  • docker pull gristlabs/grist:1.1.6 -- OK

So I am reassured nothing has been overwritten. Take your time and please let me know what went wrong with our install...

asitemade4u avatar Nov 30 '23 16:11 asitemade4u

I have spoken too fast... With version 1.1.6 I see all documents in the dashboard but when I click on each of their tiles, I get this error:

You can try reloading the document, or using recovery mode. Recovery mode opens the document to be fully accessible to owners, and inaccessible to others. It also disables formulas. [Loading chunk 966 failed. (error: https://grist.qtpl.net/v/unknown/966.bundle.js)]

And the recovery mode loops to no avail. Then I arrive again on an empty dashboard...

Now being really worried.

asitemade4u avatar Nov 30 '23 16:11 asitemade4u

Loading chunk 966 failed means there's a file the browser is failing to load somehow, looks to be https://..../v/unknown/966.bundle.js

You could try clearing your cache again or using a different browser just to be sure it isn't a cached problem.

paulfitz avatar Nov 30 '23 16:11 paulfitz

Same with version 1.1.5. What is obvious is that the error is about the user: except for version 1.1.6 which kind of works, the user on the upper left corner of dashboard is @guest instead of the real user's name.

asitemade4u avatar Nov 30 '23 17:11 asitemade4u

PHEW ! You were right @paulfitz , after cleaning the cache, version 1.1.6 works fine and can access the documents. THANKS!

asitemade4u avatar Nov 30 '23 17:11 asitemade4u

Do users sign in using Basic Auth? With the addition of GRIST_IGNORE_SESSION=1, version 1.1.7 should work too, and more sensibly.

It is important that your setup controls the Remote-Email header for all requests that go to Grist, i.e. your server should not pass through this header (if present in the original request) to any Grist endpoint, but should set its own value. (This is from the issue that @paulfitz linked to, https://github.com/gristlabs/grist-core/issues/750#issuecomment-1811082115.)

For a bit more info: before 1.1.7, one visit with Basic Auth would save a session cookie and keep you signed in. But with Basic Auth, there is no good way to sign out, except clearing cookies AND making the browser forget its Basic Auth credentials. Starting with 1.1.7, Grist will only set a session cookie if you configure GRIST_FORWARD_AUTH_LOGIN_PATH. For a setup like Basic Auth, session cookies are not needed and make things worse, so it requires setting GRIST_IGNORE_SESSION=1.

Both options are explained a bit better here: https://github.com/gristlabs/grist-core#forward-authentication-variables.

dsagal avatar Nov 30 '23 18:11 dsagal

@dsagal thanks for the explanation. However, I must admit it goes beyond my computer security skills. To make things simpler for me, what docker-compose.yml instructions would you recommend in/remove from our configuration above for a local-only Grist server?

asitemade4u avatar Nov 30 '23 19:11 asitemade4u

And currently we all enter Grist without authentication at all. For some reason we do NOT have to enter our credentials and yet we are brought directly to our own private documents

asitemade4u avatar Nov 30 '23 19:11 asitemade4u

Hmm, do you know which piece of your setup sets the header Remote-Email (confgured in your docker-compose.yml as GRIST_PROXY_AUTH_HEADER=Remote-Email) ?

dsagal avatar Nov 30 '23 20:11 dsagal

What I can tell you is that we are using LinuxServer's SWAG, which is based upon NGINX. All our self-hosted applications are served under HTTPS. Here is the NGINX config used in SWAG:

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name grist.*;
    include /config/nginx/ssl.conf;
    client_max_body_size 0;

    location / {
        include /config/nginx/internal.conf;
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app grist;
        set $upstream_port 8484;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;
    }
}

asitemade4u avatar Nov 30 '23 20:11 asitemade4u

I would guess the relevant settings (i.e. those that I'd hope would mention Remote-Email) may be in /config/nginx/proxy.conf that's included from the snippet you shared.

dsagal avatar Dec 04 '23 07:12 dsagal

I am having the same issue. I am using traefik-forward-auth, and v1.1.7 was a breaking change that I fixed with GRIST_IGNORE_SESSION. This morning I pulled the latest version, and now it seems not to be accepting my Remote-Email. I get the following error:

2023-12-06 12:25:42.536 - debug: Auth[GET]: grist.{MY_DOMAIN}.net /session/access/all customHostSession=, method=GET, host=grist.{MY_DOMAIN}.net, path=/session/access/all, org=docs, [email protected], userId=1, altSessionId=undefined

Here's the snippet of my grist environment from docker compose:

environment:
      - DEBUG=1
      - GRIST_DATA_DIR=/docs
      - APP_DOC_INTERNAL_URL=https://localhost:8484
      - APP_DOC_URL=https://grist.$DOMAINNAME_CLOUD_SERVER
      - APP_HOME_URL=https://grist.$DOMAINNAME_CLOUD_SERVER
      - GRIST_SINGLE_ORG=docs
      - GRIST_ORG_IN_PATH=false
      - GRIST_SUPPORT_ANON=false
      - GRIST_IGNORE_SESSION=1
      - GRIST_DOMAIN=https://grist.$DOMAINNAME_CLOUD_SERVER
      - GRIST_FORWARD_AUTH_HEADER=X-Forwarded-User
      - GRIST_FORCE_LOGIN=true
      #- GRIST_FORWARD_AUTH_LOGIN_PATH=/_oauth
      #- GRIST_FORWARD_AUTH_LOGOUT_PATH=/_oauth/logout
      - GRIST_DEFAULT_EMAIL=$MY_EMAIL
      - GRIST_SESSION_COOKIE=grist.$DOMAINNAME_CLOUD_SERVER

jaytrickett1 avatar Dec 06 '23 17:12 jaytrickett1

Sorry to hear this. Could you share your traefik-forward-auth configuration? It may help to look at https://github.com/gristlabs/grist-omnibus, which uses traefik-foward-auth with dex for logins, and is known to work.

Incidentally, the message you shared isn't by itself a sign of problem. It just means that the request is made by a user who isn't logged in.

Also, I notice that you mention "Remote-User" in your message as the header, but the config mentions "X-Forwarded-User". The GRIST_FORWARD_AUTH_HEADER should match the header set by traefik-forward-auth.

dsagal avatar Dec 07 '23 04:12 dsagal