v2.3.0 causes mongo transaction issue "Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]"

[ssyberg]

Link to reproduction

No response

Describe the Bug

Apologies in advance, I'm not going to be able to provide many details here. I'm experiencing this in a large project I can't share and I have very little insight into what's causing it. What I do know is my unit/integration tests are working and passing on my project on 2.2.1, when I upgrade to 2.3.0 with no other changes, my seeds break with this error during a locale create call.

This only occurs in the seed script, calling create via local API. Creating a record in this same collection via admin works fine.

/Users/seth/dev/backend/node_modules/mongodb/src/transactions.ts:168
    throw new MongoRuntimeError(
          ^
MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
    at Transaction.transition (/Users/seth/dev/backend/node_modules/mongodb/src/transactions.ts:168:11)
    at commandHandler (/Users/seth/dev/backend/node_modules/mongodb/src/sessions.ts:740:27)
    at /Users/seth/dev/backend/node_modules/mongodb/src/sessions.ts:810:7
    at /Users/seth/dev/backend/node_modules/mongodb/src/utils.ts:462:14
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  [Symbol(errorLabels)]: Set(0) {}
}

And seeing this response on the local API create call:

{"ok":0,"code":251,"codeName":"NoSuchTransaction","$clusterTime":{"clusterTime":{"$timestamp":"7307627979285200993"},"signature":{"hash":"AAAAAAAAAAAAAAAAAAAAAAAAAAA=","keyId":0}},"operationTime":{"$timestamp":"7307627979285200993"}}

To Reproduce

Due to the complexity of my project and the relative lack of details in the error, I really have no way to isolate this issue or provide a simplified reproduction. My best guess is this is related to something happening in a hook executing during create. The one other thing that's relevant is that this problem does not occur on the first 10 or so collection seeds which all complete as expected.

Payload Version

2.3.0

Adapters and Plugins

db-mongodb

[bsides]

Same error here on a fresh install, using mongodb atlas and payload-demo, also reported here: https://github.com/payloadcms/payload/issues/4072

[davelsan]

The error message is slightly different than the one in #4072, though.

[TRANSACTION_COMMITTED] to [TRANSACTION_ABORTED]
// versus
[TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]

I was not having the problem after I updated. I'm going to check again though, I've been developing locally since then. I'm using a mongodb+srv connection with retryWrites=true&w=majority options. In case it helps.

[bsides]

Thanks, I’m going to try those

I was not having the problem after I updated. I'm going to check again though, I've been developing locally since then. I'm using a mongodb+srv connection with retryWrites=true&w=majority options. In case it helps.

[ssyberg]

I've tried adding retryWrites=true&w=majority but no change. We're stuck on 2.1 for the moment 😓

[ssyberg]

I have a little extra info here @DanRibbens upon closer inspection my seed script successfully creates one record on the collection in question (posts) in English, it then runs update in a loop and creates a translation for each locale available (in my case that's ~8 locales. All that executes without issue and I see the record and translations in the CMS. It then attempts to create a second post in English and that's when I get this error. I'm still investigating what about posts is different from the 12 collections this error doesn't occur on, it must be something happening in a hook that specific to posts but why this works on 2.1 and not 2.3 is confounding. I'm really hoping something about how transactions changed between those two versions will be obvious to someone!

[DanRibbens]

I'm trying so hard to recreate the issues in this thread without success.

I was able to get some errors when seeding the public-demo against an atlas connection with transactions. This makes sense as the seeding function is creating and updating documents synchronously in a way that it shouldn't work. That said, I was getting write conflict errors which are expected with how it is currently written.

I think there is a documentation gap we need to cover. Ideally when you're doing your document changes like the seeding in public-demo you have two options:

1. Wrap all your local api calls in one transaction assigned to the req and pass it through to every operation. This way document changes occur in one session and will not have any write conflicts with (unless some other source is also actually conflicting).
2. await all local api calls so that they are asynchronously changing the database

This has been done throughout the core of payload and plugins and the /templates. It seems I also need to update our payload-demo.

The only known issue I've come across has to do with multiple synchrounous operations that error abortTransaction at the same time will cause a 500 and there is a PR and some discussion ongoing into that one.

I don't see what has changed from 2.1 that would cause this looking at the changes released.

I have a branch that will allow passing transactionOptions or disabling transaction usage altogether for mongodb. I don't want that to be the fix though.

Thanks for your patience while I figure things out.

[ssyberg]

I've got a bit more data here from my custom "payload shell", active transaction number -1 doesn't seem good! Also just to confirm I'm not doing any explicit transaction stuff at all. No accessing req.transactionID or proactively committing/aborting.

payload> r = await payload.create({collection:'posts',locale:'en',data})
Uncaught:
MongoServerError: Given transaction number 1 does not match any in-progress transactions. The active transaction number is -1
    at Connection.onMessage (/Users/seth/dev/****/backend/node_modules/mongodb/src/cmap/connection.ts:449:20) {
  ok: 0,
  code: 251,
  codeName: 'NoSuchTransaction',
  '$clusterTime': {
    clusterTime: new Timestamp({ t: 1702056495, i: 185 }),
    signature: {
      hash: new Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: 0
    }
  },
  operationTime: new Timestamp({ t: 1702056495, i: 185 }),
  [Symbol(errorLabels)]: Set(1) { 'TransientTransactionError' }
}
payload> Uncaught:
MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
    at /Users/seth/dev/****/backend/node_modules/mongodb/src/sessions.ts:810:7
    at commandHandler (/Users/seth/dev/****/backend/node_modules/mongodb/src/sessions.ts:740:27)
    at Transaction.transition (/Users/seth/dev/****/backend/node_modules/mongodb/src/transactions.ts:168:11) {
  [Symbol(errorLabels)]: Set(0) {}
}
payload> Uncaught:
MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
    at /Users/seth/dev/****/backend/node_modules/mongodb/src/sessions.ts:810:7
    at commandHandler (/Users/seth/dev/****/backend/node_modules/mongodb/src/sessions.ts:740:27)
    at Transaction.transition (/Users/seth/dev/****/backend/node_modules/mongodb/src/transactions.ts:168:11) {
  [Symbol(errorLabels)]: Set(0) {}
}

[ssyberg]

I've gone through and confirmed every create and update is awaited and passing req along.

[ssyberg]

I need to switch gears here, but I have been able to reliably reproduce it and more importantly reliably avoid it. I can't say what I've found is super promising or helpful, but here it is.

I commented out data generators for my posts seeds and one by one uncommented and re-run seeds. What I determined was when I generated data for a relationship field I got the transaction error. If I commented out that generator, the error went away (I repeated this many times to make sure this was in fact triggering it). It's also worth noting I experimented a bit with some other relationship fields and some of them always work and some of them never work. I can't find any pattern, some relationships to upload collections work, others do not, some relationships to other non-upload collections work, others do not. I was even able to reproduce it in admin and this is where it gets even weirder:

I kept clicking Save over and over and the transaction id was always off by 1 and I kept getting that error, and then on the 9th time it saved successfully! It only happens on new posts, if I select that seed file on any other existing post I never get an error on save...

[mvdve]

Also running into this issue with Payload 2.3.1 running on AWS app runner with DocumentDB. Our app runner instance stops/crash/restart with the following backtrace:

MongoServerError: Given transaction number 859 does not match any in-progress transactions.
at Connection.onMessage (/home/node/node_modules/mongodb/lib/cmap/connection.js:231:30)
at MessageStream.<anonymous> (/home/node/node_modules/mongodb/lib/cmap/connection.js:61:60)
at MessageStream.emit (node:events:514:28)
at processIncomingData (/home/node/node_modules/mongodb/lib/cmap/message_stream.js:125:16)
at MessageStream._write (/home/node/node_modules/mongodb/lib/cmap/message_stream.js:33:9)
at writeOrBuffer (node:internal/streams/writable:392:12)
at _write (node:internal/streams/writable:333:10)
at Writable.write (node:internal/streams/writable:337:10)
at Socket.ondata (node:internal/streams/readable:766:22)
at Socket.emit (node:events:514:28)
/home/node/node_modules/mongodb/lib/transactions.js:120
throw new error_1.MongoRuntimeError(`Attempted illegal state transition from [${this.state}] to [${nextState}]`);
MongoRuntimeError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
at Transaction.transition (/home/node/node_modules/mongodb/lib/transactions.js:120:15)
at commandHandler (/home/node/node_modules/mongodb/lib/sessions.js:478:33)
at /home/node/node_modules/mongodb/lib/sessions.js:530:9
at /home/node/node_modules/mongodb/lib/utils.js:348:66
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[Symbol(errorLabels)]: Set(0) {}
}
Node.js v18.17.0

Will update this post if i find more.

[ssyberg]

@DanRibbens I believe I've found the culprit (at least in our setup). I've been able to avoid this by disabling filterOptions on the relationship field, which indicates to me that the associated filter/validation/find is somehow not part of the session/transaction. That's the best I can guess at this point!

[mvdve]

Could not find a stable version so went the nuclear option and disabled transactions all together on version 2.4.0. Seems to be stable.

[xHomu]

Could not find a stable version so went nuclear option and disabled transactions all together on version 2.4.0. Seems to be stable.

How do you disable transactions altogether? I can't seem to find it in the docs.

[mvdve]

How do you disable transactions altogether? I can't seem to find it in the docs.

This is not yet an option, but there is a PR (#4467) in the works.

I patched @payloadcms/db-mongodb/dist/index.js with the following:

- const _beginTransaction = require("./transactions/beginTransaction");
- beginTransaction: _beginTransaction.beginTransaction,
+ beginTransaction: undefined,

[ssyberg]

I've got a little more data here, I was able to bypass some of the most persistent errors here by disabling some filterOptions during my seed script (not great since we use the seeds for our unit tests) and what I've found now I think is fairly revealing of the underlying issue (and is in line with what we've seen thus far). Every time I run my seeds now it fails on a different collection! To me this indicates there's a race condition such that depending on when some database operations completes this will or will not get triggered. I increasingly think this is on reads, not writes - possibly that reads/finds are losing the transaction somewhere along the line, or a find references a transaction that a write has already committed and hence the transaction mismatch?

[ssyberg]

FWIW @mvdve this PR is useful for unblocking us, but feels like there is still an underlying issue?

[jnicewander]

A support rep at MongoDB Atlas pointed me to this related issue with the MongoDB Node.js driver. The rep said it was actively being looked into.

[DanRibbens]

Thanks for sharing that link @jnicewander!

Given that information, it makes sense to me that we would just swallow the error with try/catch so that any errors from calling abortTransaction can be ignored. You would get the error response that caused the abort to fail rather than 500 something went wrong because of this meaningless error from the node.js mongodb driver.

What do we think?

[mvdve]

abortTransaction does not throw any other errors? According to the documentation the abortTransaction function only does a single retry on failure so it would be preferable to catch and log this error to prevent debugging issues until it has been resolved.

[franckmartin]

I have the same problem when I run my seed script. About 5% of MongoDB transactions fail randomly. I temporarily fixed the problem by disabling all filterOptions in my relationship fields, but this is not a lasting solution.

Configuration : payload 2.6 & db-mongodb 1.1.

[fr3fou]

Is there a workaround that doesn't involve removing filterOptions? They're crucial to ensuring data integrity for relationships

[DanRibbens]

I've still not been pointed to a reproducable repo for these transactions issues. Can somebody share their filterOptions code that leads to this problem?

Reviewing our code for filterOptions, I could see there being a problem if the query is having an error resulting in an abort before the rest of the transaction, but I'm not very eager to make changes here without understand the problem more.

[franckmartin]

Hi @DanRibbens

Really nothing special with the filterOptions which are problematic :

{
    name: "heroHeaders",
    label: {en: "Hero headers", fr: "Bandeaux"},
    type: "relationship",
    relationTo: "hero-headers",
    hasMany: true,
    required: true,
    filterOptions: {_status: {equals: "published"}},
}

{
    name: "animation",
    label: {en: "Animation", fr: "Animation"},
    type: "upload",
    relationTo: animationsCollectionSlug,
    filterOptions: {mimeType: {equals: "video/mp4"}},
    required: required,
}

[ssyberg]

I don't know about others situations/projects, but it would be pretty difficult for me to create a reproduction. I think this only happens with a certain level of complexity/relationships/hooks that would be very difficult to simulate. I know that isn't helpful, but it's not for lack of trying! I also haven't been able to reproduce in reasonably simple sandboxes, but I'm certain there's an issue 😆

I think you're instinct here @DanRibbens is correct re an error/abort during transaction. I'm hesitant to speak to this because I have very cursory understanding of node process management / workers / tasks (or even the payload hook system) but I think the single biggest development difficulty in Payload is lack of transparency in errors. Again I have no idea if that is fundamental to node async tasks so not necessarily a knock on Payload, but there does seem like there's a huge swatch of errors like this that just get swallowed and if they were easily surfaced could significantly improve debugging.

[DanRibbens]

@ssyberg, that is good feedback as a general matter. Payload should have try/catch in a great many more places, especially when we call functions straight from the config. As a result, the developer can't easily identify bugs stemming from their own config, plugins or in Payload resulting in the error.

Thanks everyone for the patience on transaction related issues. I would really like to finally move past the transactions issues.

I'm currently wrapping the filterOptions backend code in a try/catch. Also, because https://jira.mongodb.org/browse/NODE-5804 was labeled wont fix from the MongoDB team, we just need to prevent this on our end. For that I'm preventing duplicate calls to abort the same transaction. That way if this happens it won't throw another error which buries the actual error in the callstack.

Yesterday I opened 2 PRs that should help if you were running into transaction issues while using plugin-search https://github.com/payloadcms/payload/pull/5068 or plugin-form-builder https://github.com/payloadcms/payload/pull/5069.

It hasn't been a clear path with different errors being reported that are never reproducable. Occasionally we do see write conflict errors from the test suites in CI. This has been mostly improved with seeding changes to be less async, though we're still seeing that a little.

I want to show progress here even though the issue hasn't been closed, we are not ignoring anyone! :)

[ssyberg]

Thanks @DanRibbens - your efforts are much appreciated, I know how nebulous this one is!

[franckmartin]

@DanRibbens I just sent you an invitation to a repository, so you can reproduce. Just do npm run seed https://github.com/franckmartin/payload-mongo-transaction

[DanRibbens]

@DanRibbens I just sent you an invitation to a repository, so you can reproduce. Just do npm run seed franckmartin/payload-mongo-transaction

I got past the sanitize error you had on the master branch. When I ran seed, I can see that your scripts create collection docs concurrently in a map. If you change your code to await prior create operations, I suspect you will be able to avoid all the retry logic you've added in your createCollectionItems function. This SO sums it up https://stackoverflow.com/questions/68924106/error-given-transaction-number-does-not-match-in-mongodb-and-nodejs

Let me know if you have any other questions!

[franckmartin]

Hi @DanRibbens

Thank you very much for your quick answer.

Sorry but I don't get you. I'm using a map to get my seed data (ex. getFishersData) but it's synchronous and don't write to Payload. And my Payload writing function createCollectionItems is using a for loop rather than a map, and is awaiting each Promise to resolve before the next create operation. So I don't think that the transaction errors are related to this SO thread.

Did I miss something ? 🤔

[iamacup]

@denolfe @DanRibbens i was previously getting these errors on project startup for 2.11.0, moving to 2.11.1 has fixed the startup errors but i still am able to produce them when loading a collection in the admin panel.

I have a video here: https://drive.google.com/file/d/1kR3DK3kdb0YyUcPwKxqppjBfpVS8iQAa/view?usp=sharing - do you have any guidance on how it might be possible to debug? Like the OP this is quite deep in the project logic - for me however this collection has no hooks or any logic.

I also have no filterOptions in the project at all.

(please note in the video, these errors are only when running with a replicaset, they do not happen if i just run mongo basic)

91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
[16:10:24] ERROR (payload): Error: Given transaction number 11 does not match any in-progress transactions. The active transaction number is 10
    at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:12)
    at new MongoServerError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:185:12)
    at onMessage (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/cmap/connection.js:256:50)
    at emit (native)
    at processIncomingData (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/cmap/message_stream.js:34:18)
    at _write (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/cmap/message_stream.js:82:26)
    at writeOrBuffer (node:stream:2596:27)
    at <anonymous> (node:stream:2579:40)
    at ondata (node:stream:2101:22)
    at emit (native)
    at addChunk (node:stream:1940:22)
    at readableAddChunk (node:stream:1894:30)
    at data (node:net:52:9)
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81
91 |  * @privateRemarks
92 |  * mongodb-client-encryption has a dependency on this error, it uses the constructor with a string argument
93 |  */
94 | class MongoError extends Error {
95 |     constructor(message) {
96 |         super(MongoError.buildErrorMessage(message));
             ^
error: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED]
      at new MongoError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:96:9)
      at new MongoDriverError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:168:9)
      at new MongoRuntimeError (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/error.js:205:9)
      at transition (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/transactions.js:120:15)
      at commandHandler (/Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:478:13)
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/sessions.js:530:9
      at /Users/tompickard/Desktop/ek-stuff/source/ek-nexus/node_modules/mongodb/lib/utils.js:348:81