Embedded app cannot complete OAuth process, and custom session storage documentation

[jt274]

Issue summary

App authentication broken after updating to the "new" way of authenticating embedded apps, according to the Shopify tutorial here: https://shopify.dev/tutorials/build-a-shopify-app-with-node-and-react

The previous tutorial produced a working app.

Error: Cannot complete OAuth process. No session found for the specified shop url:

Additionally, the tutorial utilizes MemorySessionStorage, and tells you not to use it. The following page provides a vague explanation of CustomSessionStorage, but does not give enough detail for a working example: https://github.com/Shopify/shopify-node-api/blob/main/docs/issues.md#notes-on-session-handling

The app in question produces the error with MemorySessionStorage.

Expected behavior

App should authenticate once, and store the session so no further authentication is required.

Tutorials should document a fully working CustomSessionStorage example, and explain how to properly access the shopOrigin parameter throughout the React app with cookies no longer active.

Actual behavior

App re-authenticates on almost every page refresh, or displays an error: Cannot complete OAuth process. No session found for the specified shop url:. This also may produce console errors for Graphql requests such as "invalid token < in JSON"

server.js file below:

require('isomorphic-fetch');
const dotenv = require('dotenv');
const Koa = require('koa');
const next = require('next');
const { default: createShopifyAuth } = require('@shopify/koa-shopify-auth');
const { verifyRequest } = require('@shopify/koa-shopify-auth');
const { default: Shopify, ApiVersion, SessionStorage } = require('@shopify/shopify-api');
const Router = require('koa-router');
const axios = require('axios').default;

dotenv.config();

Shopify.Context.initialize({
  API_KEY: process.env.SHOPIFY_API_KEY,
  API_SECRET_KEY: process.env.SHOPIFY_API_SECRET,
  SCOPES: process.env.SHOPIFY_API_SCOPES.split(","),
  HOST_NAME: process.env.SHOPIFY_APP_URL.replace(/https:\/\//, ""),
  API_VERSION: '2021-01',
  IS_EMBEDDED_APP: true,
  SESSION_STORAGE: new Shopify.Session.MemorySessionStorage(),
});

const port = parseInt(process.env.PORT, 10) || 3000;
const dev = process.env.NODE_ENV !== 'production';
const app = next({ dev });
const handle = app.getRequestHandler();

const ACTIVE_SHOPIFY_SHOPS = {};

app.prepare().then(() => {
  const server = new Koa();
  const router = new Router();
  server.keys = [Shopify.Context.API_SECRET_KEY];

  server.use(
    createShopifyAuth({
      accessMode: 'online',
      async afterAuth(ctx) {
        const { shop, accessToken, scope } = ctx.state.shopify;
        ACTIVE_SHOPIFY_SHOPS[shop] = scope;

        //Store accessToken in database

        ctx.redirect(`/?shop=${shop}`);
      },
    }),
  );

  router.post("/graphql", verifyRequest(), async (ctx, next) => {
    await Shopify.Utils.graphqlProxy(ctx.req, ctx.res);
  });

  const handleRequest = async (ctx) => {
    await handle(ctx.req, ctx.res);
    ctx.respond = false;
    ctx.res.statusCode = 200;
  };

  router.get("/", async (ctx) => {
    const shop = ctx.query.shop;

    if (ACTIVE_SHOPIFY_SHOPS[shop] === undefined) {
      ctx.redirect(`/auth?shop=${shop}`);
    } else {
      await handleRequest(ctx);
    }
  });

  router.get("(/_next/static/.*)", handleRequest);
  router.get("/_next/webpack-hmr", handleRequest);
  router.get("(.*)", verifyRequest(), handleRequest);

  server.use(router.allowedMethods());
  server.use(router.routes());

  server.listen(port, () => {
    console.log(`> Ready on http://localhost:${port}`);
  });
});

_app.js file below:

import React from 'react';
import App from 'next/app';
import Head from 'next/head';
import { AppProvider } from '@shopify/polaris';
import { Provider, Context } from "@shopify/app-bridge-react";
import { authenticatedFetch } from "@shopify/app-bridge-utils";
import '@shopify/polaris/dist/styles.css';
import translations from '@shopify/polaris/locales/en.json';
import ClientRouter from '../components/ClientRouter';
import ApolloClient from 'apollo-boost';
import { ApolloProvider } from 'react-apollo';
import '../uptown.css';

class MyProvider extends React.Component {
  static contextType = Context;

  render() {
    const app = this.context;

    const client = new ApolloClient({
      fetch: authenticatedFetch(app),
      fetchOptions: {
        credentials: "include",
      },
    });

    return (
      <ApolloProvider client={client}>
        {this.props.children}
      </ApolloProvider>
    );
  }
}

class MyApp extends App {
  render() {
    const { Component, pageProps, shopOrigin } = this.props;
    const config = { apiKey: process.env.NEXT_PUBLIC_SHOPIFY_API_KEY, shopOrigin: shopOrigin, forceRedirect: true };
    return (
      <React.Fragment>
        <Head>
          <title>My App</title>
          <meta charSet="utf-8" />
        </Head>
        <Provider config={config}>
          <ClientRouter />
          <AppProvider i18n={translations}>
            <MyProvider>
              <Component {...pageProps} />
            </MyProvider>
          </AppProvider>
        </Provider>
      </React.Fragment>
    );
  }
}

MyApp.getInitialProps = async ({ ctx }) => {
  return {
    shopOrigin: ctx.query.shop,
  }
}

export default MyApp;

[paulomarg]

Hey @jt274, thanks for this. To answer your questions:

1. I've pasted your code into a test app and it is working as expected. My steps were:

• Replaced both files with yours and run the server
• Go into my /admin/apps page on Shopify, click on my app
• App redirects me to top level, performs OAuth
• I come back to /admin/apps/my-app, page loads

Are you doing anything differently to run into the error you mention?

2. We're actually in the process of adding a concrete example of CustomSessionStorage using Redis (https://github.com/Shopify/shopify-node-api/pull/129) to the documentation! As stated in the docs, the library provides the MemorySessionStorage class as a way to allow devs to quickly set up their apps, because we can't know which storage method is being used.

[jt274]

@paulomarg Thanks - the documentation from that PR is the best so far.

I guess another question I have is about the session in general. In the previous Node/React tutorial, I don't remember anything about sessions. The libraries seemed to do all of it for you.

I am not familiar with Redis. In the example, using Redis, it would store the session data locally? I'm not seeing from the example how the session is originally generated and passed to the storeCallback, or how to get the id to pass to loadCallback or deleteCallback. And I'm using regular JS, not Typescript.

[paulomarg]

In the previous iteration, we used koa-session to store the session, which simply stored the entire session in a cookie (as per their docs - note how they use a similar pattern for custom storage options). Unfortunately, because modern browsers are making it more difficult to use cookies within iframes, that option wouldn't work for embedded apps.

Our package still handles all of the logic behind sessions, and CustomSessionStorage is just a way for apps to tell us how they want to load / store / delete their sessions - the library handles creating the actual sessions and calling those methods, so apps don't have to worry about it.

Hope this helps clear things up for you and others who end up here from searches!

[jt274]

@paulomarg I understand about the cookies, just not sure how the sessions all worked. Makes sense.

So using the CustomSessionStorage and Redis example, you don't have to pass anything to the callback functions (i.e. the session or id)?

I will try to get it working. If I do I'm willing to post the code.

[jt274]

@paulomarg I am slowly getting somewhere. I've stored the session data to my database of choice, and am able to load it back. But when I do I get an error in the loadCallback method: InternalServerError: Expected return to be instanceof Session, but received instanceof Object.

If I do either one of these (where session is the returned session that's been loaded):

return new Session(session);

var newSession = new Session(session.id);
newSession = {...newSession, ...session};
return newSession;

I get this error: InternalServerError: Mismatched data types provided: string and undefined

The session is storing the id, shop, state, and isOnline parameters.

[ilugobayo]

@jt274 I think I'm at the same point as you.

I've implemented my CustomSessionStorage which is basically just inserting the session ID and the stringified session body in a database record. So the storeCallback function works as expected, but whenever the app wants to load a session I get the same error InternalServerError: Expected return to be instanceof Session, but received instanceof Object.

This is currently my loadCallback function:

async loadCallback(id) {
  try {
    var reply = await dbGetHelper.getSessionForShop(id);
    if (reply) {
      return JSON.parse(reply);
    } else {
      return undefined;
    }
  } catch (err) {
    // throw errors, and handle them gracefully in your application
    throw new Error(err)
  }
 }

Maybe I'm doing it wrong after reading the following comment from @paulomarg here: So you can just return JSON.parse from your loadSession callback and we'll convert it to a proper Session object for you.

I want to know if that's the right way to do it, or maybe I just misunderstood it.

Also, I've been having a hard time with the ACTIVE_SHOPIFY_SHOPS object, it's not clear to me whether the CustomSessionStorage replaces it, or you have to use it as well along with the session storage since the code in the tutorial includes the following comment:

// Storing the currently active shops in memory will force them to re-login when your server restarts. You should
// persist this object in your app.

When and where should we set/update this object?

I have some other questions about custom routes (i.e. navigation elements in embedded app, can't display the pages on them) and verifyRequest, but I think I should find another issue more related to that.

[jt274]

I was finally able to store and load a session during the auth with some messy code in the loadCallback:

var newSession = new Session(session.id)
    newSession.shop = session.shop;
    newSession.state = session.state;
    newSession.scope = session.scope;
    newSession.expires = session.expires; //initially undefined
    newSession.isOnline = session.isOnline; //initially undefined
    newSession.accessToken = session.accessToken; //initially undefined
    newSession.onlineAccessInfo = session.onlineAccessInfo; //initially undefined

    return newSession;

The Session object must first be initialized with only the session ID, then have the other parameters added (which are only shop, state and scope initially). After going through the auth, it created a second new session object of this format:

accessToken: "TOKEN"
expires: "2021-03-12T06:45:01.259Z"
id: "store-name.myshopify.com_12345678901"
isOnline: true
onlineAccessInfo: {

account_number: 1
associated_user: {

account_owner: true
collaborator: false
email: "[email protected]"
email_verified: true
first_name: "Bob"
id: 12345678901
last_name: "Smith"
locale: "en"

}

associated_user_scope: "write_script_tags,write_inventory,read_products"
expires_in: 86396
session: "LONG ID NUMBER"

}
scope: "write_script_tags,write_inventory,read_products"
shop: "store-name.myshopify.com"
state: "01234567890123"

I now continue to have graphql errors, such as Network error: Unexpected token B in JSON at position 0 or token <, etc.

[ilugobayo]

@jt274 are you using online or offline access token? I'm currently using offline tokens, should I use online tokens as well?

Also, that info about the owner, did you generate it or was autogenerated?

[jt274]

@ilugobayo Can't find it now, but docs somewhere mentioned using online tokens for apps in the user facing admin, and offline were for a database backend that makes requests without the user interface.

When going through the Oauth, it initially creates a token in the database that only has the shop, state, id, and scope parameters. Then a couple seconds later in the Oauth process, it modifies the token to have all of that information above.

Now I'm figuring out why it appears I've created and loaded tokens but graphql requests no longer work.

[kato-takaomi-ams]

I also don't understand how to choose access-mode. My understanding is as following table.is it correct?

!Edited!

• My first understanding

access-mode	online	offline
user	customer(use storefront)	merchant(use shopify-admin or backend-system)

• My current understanding

user/access-mode	online	offline	sessionid
customer(use storefront)	use	not use	uuidv4()
merchant(use shopify-admin)	use	not use	${shop}_${userId}
merchant(use backend-system) e.g. external system cooperation	not use	use	offline_${shop}

[ilugobayo]

@jt274 it's really weird, in my case, the starting session object has only id, shop, state and isOnline, that's what is stored in the database at first, I modified my CustomSessionStorage to update the record in the database if a record already exists, this because at some point, after receiving the first session object, I receive another session object, now with id, shop, state, isOnline, accessToken and scope, but my loadCallback keeps loading the a session without the two last elements, I don't know why even though the record is actually updated in the database.

[ilugobayo]

@kato-takaomi-ams maybe that's where I'm failing? I'm only using the offline access mode, I'm not sure if I should be using online as well, I mean, I have an UI for the merchant, but most of what my app does requires offline access mode since its features involve several request to the Shopify Admin API

[paulomarg]

Hey all, let me see if I can shed a bit of light here.

The storeCallback function is expected to update existing sessions. The reason behind it being called twice is that it stores the session once when OAuth begins, mostly so it can store the state - we check that when completing the OAuth process to make sure the request actually matches the OAuth process that we start, for extra security.

We've added support for loadCallback to return the result of a JSON.parse of a stored session so you don't have to build the object from scratch, but that change will only be available upon our next release.

As far as online vs offline sessions go, you can read up on how they work in our documentation. Essentially, online tokens are best suited for cases where users are directly interacting with your app, whereas offline tokens work better for background tasks, since they don't expire and are tied to the shop as a whole.

Hope this helps!

[ilugobayo]

@paulomarg thanks for this comment, I just have some questions about this:

• Do you know why, even though I update the session in my database, the loadCallback keeps loading the "first version" of the session? Maybe I'm not promisifyng?

• My app performs tasks where the merchant interacts but it also has various webhooks that perform tasks whenever these are triggered, should I be using both access modes? Or using offline covers both scenarios? If I should use both of them, can you illustrate me on how I should create/handle them?

[jt274]

@ilugobayo I am not sure if this helps, but my app has the merchant interacting with it, as well as back-end things that happen with webhooks.

I am using the online session for the merchant interaction, but I've also stored the accessToken in the database under the shop name when the app is installed or reloaded. The back end looks this up and uses the accessToken for the back end requests to Shopify (no "session" from the package needed). Hope that makes sense.

[ilugobayo]

@jt274 thanks, I think using online access mode should be better then.

I actually do the same with the accessToken, whenever the app is installed, I store the accessToken associated to the shop domain (xxx.myshopify.com), my only concern is, wouldn't be a problem if by some reason, the accessToken is updated, then a webhook gets triggered just in time to try to use an invalid one? Do you think that scenario is possible or am I thinking way too much?

By the way, do you store the accessToken and the sessions in the same table? Also, do you store everything that comes with the session, do you do it as a stringified JSON as well?

[jt274]

@ilugobayo I suppose that could happen? I have it update the accessToken whenever the app is loaded. I will try to look into how long the accessToken persists and when it would be updated. But I don't recall seeing anywhere that it does update - I thought it was similar to an API key.

I am using Firestore, and store the token/app information in one collection, and sessions in another. When I store the sessions, I put in everything that is given. Since Firestore already is JSON essentially, I just write the session object to it and the fields are mapped automatically. Configuring it to merge data, if it provides a new session a new document is created. If it provides a previous session the document is updated with any new data.

[ilugobayo]

@jt274 interesting, I think this is just getting clearer for me.

Now I just need to figure out why even if my storeCallback updates the session in my database, loadCallback keeps getting the "first phase" of the session; that is really weird, I don't know if I'm doing something wrong when storing/loading

Also, do you perform any checks on scope changes? If so, do you do it in afterAuth?

[jt274]

@ilugobayo Have not worked on the scope changes yet, that also seems new. Still some smaller things like that to get done.

I think mine was doing something similar, but making sure the expires object was a Date seems to have fixed it so far. See here: #65

It does, however, create two sessions still. It creates one with a random id, populates all of it, then creates a second identical (minus slightly later expires timestamp) named store-name.myshopify.com_1234567890. It then appears to use the shop named session from then on. I'm not sure what the first session is all about.

[paulomarg]

Couple of comments:

• @ilugobayo all of those callbacks should be promisified, so it could be that your app is moving forward before it actually updates the session.

• Online and offline tokens work the same way in how they allow access to Shopify, with a few key differences for offline ones:

  • They never expire (so they're less secure)
  • Every user for that shop will have the same level of access, so you may want to avoid offline tokens if you need to limit access for certain users
  • Apps should not allow users to execute GraphQL mutations on behalf of your app (e.g. using the GraphQL proxy) with an offline token, as queries from offline tokens are expected to be coming directly from the app.
  • All of that said, it's still recommended that apps use online tokens for direct user interactions, and offline ones for background tasks.

• The first session (random id) is the cookie-based OAuth session that happens at the top level (so App Bridge and therefore JWT are unavailable). When using embedded apps, the second session is the one that will be loaded from the JWT token on subsequent requests (since cookies are unavailable once the embedded app loads). Unfortunately, we need both as neither solution works for both OAuth and 'regular' usage.

[ilugobayo]

@jt274 Ok ok, this is just getting more and more interesting.

I will test all this with online access mode, since everything I was doing until now was using offline access mode.

Thank you so much for taking time to answer my questions, I might bother you once again if I find some more issues, I hope you can help me then once again.

@paulomarg

• Yes, I think that's where I'm failing, I thought I was promisifying the callbacks, but it seems I'm not.
• I'm moving to use online tokens then, it really seems like the better option here.
• I see, that makes a lot of sense now; it's just weird that in offline access mode, the id is not random, the id is always offline_store-name.myshopify.com, the only difference is that at first, the session doesn't include the accessToken nor the scope, maybe that's why I was so confused.

Thanks for making it clearer!

[jt274]

@paulomarg I think this issue is mostly resolved, but one more question.

Is the online/offline token you're referring to the accessToken parameter of the session object, and is its expire time the expires parameter in the session object? And if so, if your app uses a back end and user interface, should you use an offline token to make sure that the accessToken does not expire suddenly when the back end needs to make a request?

Because before all these auth changes, I simply stored the accessToken and had the back end use it whenever it made a request.

[ilugobayo]

@jt274 exactly my concern. That's why my first approach was to use offline access mode, what if the merchant doesn't open the app in more than the time it took the accessToken to expire, I know it's really hard for this to happen, but I see it as a possible scenario, then every request to the Admin API in the back end would fail, am I right?

[jt274]

@ilugobayo I am not sure. It appears the onlineAccessInfo expires_in is 24 hours, but I think that is the session. Then there is the expires parameter, which seems to be a time stamp of the current time, that is not in the future... So I'm still unsure about the expiration of the actual accessToken.

[jt274]

Think I've got my answer: https://shopify.dev/concepts/about-apis/authentication#api-access-modes

[ilugobayo]

@jt274 I was about to link you that. As you can see, the scenario where the accessToken becomes invalid could be possible. I was thinking if it was possible to use both modes though, but I think it would become too messy to handle.

[jt274]

@ilugobayo I think offline is the way to go.

I've attempted to switch to offline and have new issues. :)

During auth, a session named offline_store-name.myshopify.com is created. It is then loaded with the callback four times...and then the callback tries a fifth time with a non-offline session name (no offline_ prefix). Of course this session is not found, and it throws the error Cannot proxy query. No session found.

I have set accessMode: 'offline' in createShopifyAuth and both instances in server.js of verifyRequest. But it is still looking at some point for an online token after reading the offline token four times in a row.

[ilugobayo]

@jt274 using offline access mode with MemorySessionStorage works as expected to me, you can try that to know if everything is ok, if that's the case then you might have something wrong in your callback functions.

My problem is what I mentioned above, that my loadCallback function doesn't get the updated session, probably because of what @paulomarg mentioned, haven't been able to fix it yet but once I do it, I think I won't have more issues in that regard.

Did you notice that the session id is the same from the start? Not getting a random id like in the online access mode.

By the way, I forgot to ask before, are you persisting the ACTIVE_SHOPIFY_SHOPS object as well? If so, how are you handling it?

[ilugobayo]

• All of that said, it's still recommended that apps use online tokens for direct user interactions, and offline ones for background tasks.

@paulomarg does this mean it is possible to use both modes? If so, can you provide a brief example on how to implement it?

[jt274]

@ilugobayo Frankly, I'm not even sure what the ACTIVE_SHOPIFY_SHOPS is even supposed to be or do. Is it supposed to be a list of every single shop on all accounts that have installed your app? Little documentation even mentioning it, other than to say "persist it" and "delete it when the app is uninstalled".