Add a transformer like trpc

[mattiavitturi]

Is your feature request related to a problem? Please describe. Date objects get converted to string

Describe the solution you'd like Use superjson as a transformer (or any other alternative)

Describe alternatives you've considered For the moment you need to manually convert to superjson and then send the event / parse back on the handler

Additional context some api like serve(inngest, [fns], { transformer: superjson })

[jpwilliams]

@mattiavitturi Yes! We've discussed this a few times to combat JSON serialization being pretty rubbish; Date, as you've found, is usually the first annoyance people run into.

blitz-js/superjson would be preferred due to its continually-growing support in popular JS frameworks, though MongoDB Extended JSON has been another candidate due to its multi-language support for the future.

A user-defined transformer is a great idea. There are a couple of challenges to review:

• Data may arrive in an unexpected format when initially adding/removing the transformer, as the event data and function runs are not necessarily running the same code.

  We could perhaps assume that if the chosen transformer throws or returns undefined then we should try the regular JSON methods as a backup. This solves adding the transformer, but detecting removal might be trickier.

• Typing for step returns currently uses a Jsonfiy<> type to show that Date becomes a string, etc. I think the most reasonable approach here is to be lazy and assume that if a transformer is specified then step return will be exactly the same as it would be in-process. This likely won't actually be the case (e.g. a User object for a database), but requiring that transformers also provide a type for transformation seems like a stretch.

Would love to get your feedback on the above, @mattiavitturi. 🙂 This is a great suggestion to remove a fair amount of boilerplate.

[mattiavitturi]

Sorry for the late reply

Maybe the simplest but not the best solution could be that if you choose to use a transformer then events are valid if and only if the parsing succeed, otherwise they are discarded.

Now that I think about it trpc-openapi fully ignores the transformer on input and output and gives to zod the raw json decoded object.

An option could be to opt-in to transformation on a per-step basis, personally, it's a chore but totally manageable.

Another one is to provide an optional metadata field under the hood, iirc superjson outputs { data: {}, meta: {}} or something similar, but in your case data is the real data of the event and meta the event metadata for the transformer. This in conjunction with the one above would be safe for steps that don't use a transformer, as it would return Jsonify<T> and T in case of a transformer.

For our start-up we are using 100% typescript so I can accept to "tie" myself with a transformer.

Being fully optional gives the responsibility of the user to choose wisely and consider the pros/cons

[jpwilliams]

@mattiavitturi Thanks for this thoughtful reply!

---

For our start-up we are using 100% typescript so I can accept to "tie" myself with a transformer.

💯 - This is the case for many folks in the Jamstack/serverless space, so I agree that giving the user the responsibility is reasonable here.

---

Another one is to provide an optional metadata field under the hood, iirc superjson outputs { data: {}, meta: {}} or something similar, but in your case data is the real data of the event and meta the event metadata for the transformer.

Yeah, I did want to try leaning into this, but the names feel way too likely to be keys someone would define on their own instead of some ___hidden_dont_use__this meta key that we could embed silently.

---

Broadly I think, as you say, allowing the user to control this and manage the consequences themselves does seem reasonable. Regarding Jsonify<T>, we could use the signature of whatever deserialize function is being passed as the transform class/object and infer this if it's something explicit (i.e. not unknown or any) so that users can also customize that in the future.

I will write up an internal spec for this to get feedback from the rest of the team and see what we think. Transforming step data is an easy win as that's all very likely to be in a single application. In contrast, event data might be a future foot gun as those everlasting facts are forever tied to a particular transformer.

[jpwilliams]

Transferring this to the SDK repo as it'll likely be a change there.

[mcky]

The Jsonify types thing has bitten us on this project, so I'm aiming to have a crack at creating a superjson middleware. Would anyone be able to point me in the direction of how to do it in a way that's going to be type-safe for the consuming function, without it knowing anything about the superjson layer?

This tweet mentions superjson so I figured someone might have tried it so far.

I don't see the transformer thing on the roadmap, so I presume it's not coming in the near term and we'll want to implement this user-side

[jpwilliams]

@mcky Transformation is definitely something we'd like to support easily. Typing is the difficulty, for sure. As you can see above, for superjson this might be acceptable in the form of just removing the Jsonify<> wrapper if a transformer is specified, though other transformers may add data or want to transform the type in different ways.

You can see this consideration is still waiting in the code. 😄 https://github.com/inngest/inngest-js/blob/main/packages/inngest/src/components/InngestStepTools.ts#L351-L354

The runtime transformation itself is possible right now with something small.

new InngestMiddleware({
  name: "Step Transformation Example",
  init: () => ({
    onFunctionRun: () => ({
      transformInput: ({ steps }) => ({
        steps: steps.map((step) => ({
          ...step,
          data: step.data && superjson.parse(step.data),
        })),
      }),
      transformOutput: ({ result }) => ({
        result: {
          data: result.data && superjson.stringify(result.data),
        },
      }),
    }),
  }),
});

The above posts discuss some methods of the user being able to provide a type that transforms data from one state (its native Inngest form) to another (superjson, or anything else).

I'd love to know your thoughts on how you might like to express this within middleware, as well as if taking off the Jsonify<> wrapper would be a reasonable short-term solution we can roll out for you.

[benank]

Thanks for this solution. I just tried this out (slightly modified), and it works pretty well, although I've run into a few issues.

1. Transform output doesn't like to be parsed since it's an object, so I had to modify the output slightly.

            transformOutput: ({ result }) => ({
                result: {
                    data: result.data
                }
            })

2. Steps still return the Jsonified output, which means that after parsing them, the data will be in the json field, even though the intellisense says otherwise. I had to create a helper to parse it for me.

        const sd = (i: any) => JSON.parse(i)['json'];

        const poll: MalaPoll = sd(
            await step.run('get-poll', async () => {
                return await db.getPoll(event.data.pollId);
            })
        );

The any type on the helper is also intended, but not very good for typescript. Looking forward to better solutions in the future for this!

[hiramhuang]

Here is the workaround for my project. I don’t know the details of how the Inngest JS SDK works internally, but this implementation works fine for me.

If you spot any bugs in the implementation, please let me know.

Don't forget to update the if expression in the function trigger you're trying to apply.

superjson-middleware.ts

import { InngestMiddleware, type MiddlewareRegisterReturn } from 'inngest';
import SuperJSON from 'superjson';

/**
 * Use the original type for the hook.
 * From the function's perspective, the input and output remain unchanged.
 *
 * This implementation is referenced from the @inngest/middleware-encryption package.
 * @see https://github.com/inngest/inngest-js/blob/67c6c580aa194465991093e35aacbb454e0f62fb/packages/middleware-encryption/src/stages.ts#L235C6-L235C19
 */
type SendEventHook = NonNullable<MiddlewareRegisterReturn['onSendEvent']>;

/**
 * Use the original type for the hook.
 * From the function's perspective, the input and output remain unchanged.
 *
 * This implementation is referenced from the @inngest/middleware-encryption package.
 * @see https://github.com/inngest/inngest-js/blob/67c6c580aa194465991093e35aacbb454e0f62fb/packages/middleware-encryption/src/stages.ts#L233
 */
type FunctionRunHook = NonNullable<MiddlewareRegisterReturn['onFunctionRun']>;

/**
 * Use this middleware to automatically serialize and deserialize data with SuperJSON.
 *
 * ---
 *
 * We must "serialize" the data instead of "stringify" it because Inngest cannot handle the data as a string.
 * Otherwise, this error will be thrown:
 * ```
 * Error: Inngest API Error: 400 json: cannot unmarshal string into Go struct field Event.data of type map[string]interface {}
 * ```
 *
 * This implementation is referenced from the @inngest/middleware-encryption package.
 * @see https://github.com/inngest/inngest-js/blob/cc6cc995372549c69f5130e485777e35cd03991a/packages/middleware-encryption/src/middleware.ts#L130-L194
 *
 * The Inngest team might offer built-in middleware for this in the future.
 * @see https://github.com/inngest/inngest-js/issues/160
 * @see https://github.com/inngest/inngest-js/issues?q=jsonify
 * @see https://github.com/inngest/inngest-js/issues?q=superjson
 */
export const superjsonMiddleware = () =>
  new InngestMiddleware({
    name: 'Inngest SuperJSON Middleware',
    init: () => {
      const onSendEvent: SendEventHook = () => ({
        transformInput: ({ payloads }) => ({
          payloads: payloads.map((payload) => ({
            ...payload,
            ...(!!payload.data && {
              data: SuperJSON.serialize(payload.data),
            }),
          })),
        }),
      });

      const onFunctionRun: FunctionRunHook = () => ({
        transformInput: ({ ctx, steps }) => ({
          steps: steps.map((step) => ({
            ...step,
            ...(!!step.data && {
              data: SuperJSON.deserialize(step.data),
            }),
          })),
          ctx: {
            event: {
              ...ctx.event,
              ...(!!ctx.event.data && {
                data: SuperJSON.deserialize(ctx.event.data),
              }),
            },
            events: ctx.events.map((event) => ({
              ...event,
              ...(!!event.data && {
                data: SuperJSON.deserialize(event.data),
              }),
            })),
          },
        }),
        transformOutput: (ctx) => {
          if (!ctx.step) return undefined;

          return {
            result: {
              ...ctx.result,
              ...(!!ctx.result.data && {
                data: SuperJSON.serialize(ctx.result.data),
              }),
            },
          };
        },
      });

      return {
        onSendEvent,
        onFunctionRun,
      };
    },
  });

utils/run.ts

import type { Inngest, StepOptionsOrId } from 'inngest';
import type { createStepTools } from 'inngest/components/InngestStepTools';

/**
 * This is a helper function to override the `step.run` function to return the correct type.
 * The Inngest SDK uses the `Jsonify<T>` type on the return value of the `step.run` function.
 * This function helps cast the return value to the correct type.
 *
 * Important: Use this function only when you're using the `superjsonMiddleware`.
 *
 * @see https://github.com/inngest/inngest-js/issues/160
 * @see https://github.com/inngest/inngest-js/issues?q=jsonify
 * @see https://github.com/inngest/inngest-js/issues?q=superjson
 */
export const run =
  <TClient extends Inngest.Any>(step: ReturnType<typeof createStepTools<TClient>>) =>
  async <T extends () => unknown>(idOrOptions: StepOptionsOrId, fn: T) =>
    (await step.run(idOrOptions, fn)) as T extends () => Promise<infer U>
      ? Awaited<U extends void ? null : U>
      : ReturnType<T> extends void
        ? null
        : ReturnType<T>;

Example

inngest.createFunction(
  ...,
  ...,
  async ({ step }) => {
    // Just replace `step.run` with `run(step)`
    const stepReturnedValue = run(step)('returns-date', () => new Date());

    // No type error anymore
    const dateExpected: Date = stepReturnedValue;

    // You can verify the returned value
    console.log('typeof stepReturnedValue', typeof stepReturnedValue)
  }
)