🐛 BUG: Import an D1 exported DB cannot be imported in local - HashIndex detected hash table inconsistency

[lakano]

Which Cloudflare product(s) does this pertain to?

D1

What versions are you using?

3.108.0

What operating system and version are you using?

Arch Linux

Please provide a link to a minimal reproduction

No response

Describe the Bug

Hi,

After an export of our D1 database on remote ( wrangler d1 export dbname --remote --output export.sql ), I'm trying to import it on local ( wrangler d1 execute --local --file export.sql ), but I got a lot of errors.

After theses long list of errors I also got a generic « ✘ [ERROR] internal error », and none of the INSERT request are executed. The file contain ~3000 INSERT with each request could expand until 12000 characters each (JSON content), if I split the files in multiples files of ~200 INSERT, I could be able to import all the file, wittout the « ✘ [ERROR] internal error » but still with the repeated errors « HashIndex detected hash table inconsistency ».

Please provide any relevant error logs

kj/table.c++:57: error: HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.
stack: ; kj::getStackTrace() = /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2b4cc7d /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2b4868c /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2b4b0ab /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2277fc0 /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@39d1109 /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@39d0edd /workers/node_modules/@cloudflare/workerd-linux-64/bin/workerd@38358f5

[mrankine]

I am also seeing this, after upgrading from 3.103.2 to 3.109.1.

[petebacondarwin]

Would it be possible to provide an import file that causes this error so that we can investigate?

[lakano]

Unfortunately I'm not allowed by the company. I'll try to find time to create a dumb requests of JSON fields with a lot of the content, and duplicate this query to reach the same massive SQL file, to see if I can reproduce it and then to attach it here to help you to debug. I'm pretty sure it's caused by some buffer / limit problems.

[mrankine]

Here's an import file which causes many HashIndex errors. Multiple INSERTs of the same values don't appear to trigger the error, at least in some quick testing.

hash_errors.sql.txt

[emily-shen]

@mrankine I've just tried to execute that file and it seems to work fine without issues :(

I'm testing with a clean new database, wondering if you're running this on a local db with stuff in it already, or if there are any other details of your setup that i might need to reproduce this?

[mrankine]

@emily-shen I created a fresh project and fresh database and can reproduce the error.

My setup is Debian 12 bookworm running in WSL2 on Windows.

• Linux *** 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64 GNU/Linux
• Windows 11 Home 24H2

Has wrangler picked up a new version of capnproto recently? Maybe something to do with this commit?

Maybe @lakano has more clues?

[mrankine]

Just confirming that, in that repo, downgrading Wrangler to 3.103.2 resolves the error.

[petebacondarwin]

Hmm... so on MacOS 15.3.1:

pnpm wrangler d1 execute prod-d1-tutorial --local --file hash_errors.sql 

 ⛅️ wrangler 3.112.0 (update available 3.114.1)
---------------------------------------------------------

🌀 Executing on local database prod-d1-tutorial (e2f51fce-c9fe-408b-88f3-79bed5a07e01) from .wrangler/state/v3/d1:
🌀 To execute on your remote database, add a --remote flag to your wrangler command.
🚣 7155 commands executed successfully.

[mrankine]

Just confirming this is still an issue in Wrangler 4. Are you able to reproduce this on a Linux machine?

[petebacondarwin]

OK I have recreated it in our Windows CI: https://github.com/cloudflare/workers-sdk/actions/runs/13885240243/job/38849070736?pr=8513#step:5:3838

[lrapoport-cf]

hey @vy-ton :) can someone on the d1 team please take a look at this?

[brewaa]

I can confirm that this is an issue for me also using Wrangler 4 with the latest version of cloudflare@latest.

I'm using:

npm create cloudflare@latest -- my-nuxt-app --framework=nuxt

I apply my migrations to my DB as usual , the schema is there as expected but running an SQL script that worked in previous versions results in the following errors (hundreds of):

kj/table.c++:57: error: HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.
stack: ; kj::getStackTrace() = 7ff6584c7927 7ff65885f8c1 7ff65885abff 7ff65885e144 7ff657fe588e 7ff657ec32d0 7ff657ec302a 7ff657ec2e1a 7ff6598aac39 7ff6b98c90ba 12a00000010 12a00a97c80 9 12a00000760 12a17ff0568

Downgrading to Wrangler 3.103.2 does resolve the hash issue but if you had scaffolded a project using npm create cloudflare@latest which installs Wrangler 4 and run npm run cf-typegen and followed the instructions to uninstall "@cloudflare/workers-types": "^4.20250124.3" , then downgraded to overcome the original hash issue, the project will fail to build/preview properly because the file referenced in .wrangler\deploy\config.json does not exist ({"configPath":"..\\..\\dist\\_worker.js\\wrangler.json"}

This is the error after downgrade, the downgrade being required, because there is definitely a problem with Wrangler 4:

 There is a deploy configuration at ".wrangler\deploy\config.json".

  But the redirected configuration path it points to, "dist\_worker.js\wrangler.json", does not
  exist.

Hope that helps and someone can get eyes on this soon.

[arthurgailes]

Also having this problem with Windows, wrangler v4.5. downgrading resolved the issue.

[brewaa]

Also having this problem with Windows, wrangler v4.5. downgrading resolved the issue.

I'd be interested to know how you overcame the configuration changes required to downgrade from v4.

Are you using Cloudflare C3 to scaffold a Nuxt or similar project?

Did you have the same problem I mentioned where the file referenced in .wrangler\deploy\config.json does not exist ({"configPath":"..\\..\\dist\\_worker.js\\wrangler.json"}?

[mrankine]

@vy-ton @lrapoport-cf any update on this?

[lakano]

Hi, Sorry for my late feedback.

Here's an import file which causes many HashIndex errors. Multiple INSERTs of the same values don't appear to trigger the error, at least in some quick testing.

hash_errors.sql.txt

Thank you for this .sql file, I got the same error with your hash_errors.sql file too, I suppose workers-sdk developers could uses to debug and fix this issue.

As reported when I've created this ticket, this bug occurred under wrangler 3.108.0 ( GNU/Linux Arch - kernel 6.13.8-zen1 ) I'v tried to go back to 3.103.2 and I can confirm this also solve this issue for me, for this hash_errors.sql file but also for my initial export.sql file. ( Thank you @mrankine to dig and found this previous wrangler version that works! )

If tried all the next releases, and it's works until 3.106.0 included, the next release available for public seems to be the 3.107.2 where the bug appear. But in the release details ( https://github.com/cloudflare/workers-sdk/releases/tag/wrangler%403.107.2 ) the only information is about a revert patch from the version 3.107, and this version wasn't published on npmjs.

Then I've tried the last current release, 4.8.0 and the bug is still present but with an extra reference information : ✘ [ERROR] internal error; reference = vi5r7at8elidlhtfv1je6c75

Hope theses informations could help devs to fix this annoying bug.

[caprolactam]

Looks like this only happens with packages that depend on miniflare >=3.20250129.0 (e.g. wrangler >=3.107.1, @cloudflare/vitest-pool-workers >=0.6.10).

It is because the following code works with [email protected], but fails with [email protected].

Something probably went wrong in the code changes from [email protected] to [email protected], as noted here.

[mrankine]

Looking at the changelog for [email protected], this commit could be related to the hash errors we are seeing: Use arm64/x86_64 ISA extensions to compute CRC32C by @fhanau

[fhanau]

Looking at the changelog for [email protected], this commit could be related to the hash errors we are seeing: Use arm64/x86_64 ISA extensions to compute CRC32C by @fhanau

Is CRC32C being used in the given range of changes though? Support for it was only made available shortly before the commit and it is available through the DigestStream API, not sure if that is being used in the D1 API.

[philippscheer]

I'm also having this problem. Running wrangler on an old version was a workaround for me:

npx [email protected] d1 execute mydb --local --file="./catalog/products.sql"

[vy-ton]

this is on our plan to address this quarter, hope to start in May

[brewaa]

Just want to add some more findings in the hope it might help.

I downgraded to:

"nitro-cloudflare-dev": "^0.1.5",
"wrangler": "^3.87.0"

I reset my database:

• Removed all tables
• Applied my initial migration

When I try to import a 41mb file using this command:

npx wrangler d1 execute DBNAME --file .\tmp\export.sql --local

I receive the error:

kj/table.c++:103: warning: detected excessive collisions in hash table; is your hash function OK?; entryCount = 2897; collisionCount = 25405; kj::getStackTrace() = 7ff6160f0fb2 7ff61690162c 7ff6168fd698 7ff616901294 7ff6161b4a5e 7ff615eb3a00 7ff615eb3763 7ff615eb355a 7ff6177a4579 7ff6777d4152 31b00000068 31b00a57494 9 31b00000790 31b11ac8278 624e 31b123ca2bc 31b123ca45c 31b123ca4e0

But the data is actually imported successfully.

I tested the same process again. Reset the database and attempt the execute from file using these versions:

"nitro-cloudflare-dev": "^0.2.2",
"wrangler": "^4.14.1"

The error output from [email protected] is much more verbose so I'm just posting a few lines of which there were hundreds:

kj/table.c++:57: error: HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.
stack: ; kj::getStackTrace() = 7ff644e15a07 7ff6451cc791 7ff6451c78df 7ff6451cb014 7ff6449b3d3e 7ff64488bf60 7ff64488bcc4 7ff64488baba 7ff645e00339 7ff6a5e49061 c900000010 c900a97340 9 c900000760 c919a63094
kj/table.c++:57: error: HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.
stack: ; kj::getStackTrace() = 7ff644e15a07 7ff6451cc806 7ff6451c78df 7ff6451cb014 7ff6449b3d3e 7ff64488bf60 7ff64488bcc4 7ff64488baba 7ff645e00339 7ff6a5e49061 c900000010 c900a97340 9 c900000760 c919a63094
kj/table.c++:57: error: HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.

But the data import was unsuccessful.

I then downgraded to:

"nitro-cloudflare-dev": "^0.2.2",
"wrangler": "^3.87.0"

Despite the error: kj/table.c++:103: warning: detected excessive collisions in hash table; is your hash function OK?; entryCount = 2897; collisionCount = 25405; kj::getStackTrace() = 7ff642ea0fb2 7ff6436b162c 7ff6436ad698 7ff6436b1294 7ff642f64a5e 7ff642c63a00 7ff642c63763 7ff642c6355a 7ff644554579 7ff6a455a7d2 33600000068 33600a57494 9 33600000790 336118e8ab8 624e 33612326114 336123262b4 33612326338

The data is imported successfully.

I still receive the error with [email protected] but data is still inserted correctly.

Just to be sure I uninstalled [email protected] and upgraded to [email protected], reset the database again and the import worked successfully without error?!

These are my devDependencies:

"devDependencies": {
		"@cloudflare/workers-types": "^4.20250506.0",
		"nitro-cloudflare-dev": "^0.2.2",
		"nitropack": "^2.11.11",
		"wrangler": "^3.105.1"
	}

Hope that helps.

[jedidja]

this is on our plan to address this quarter, hope to start in May

Awesome! I'm also experiencing this. Can confirm https://github.com/cloudflare/workers-sdk/issues/8153#issuecomment-2830527966 works as well - thanks @f1ps1!

[lakano]

this is on our plan to address this quarter, hope to start in May

Hi @vy-ton Can you share any new ETA about this issue please?

[matrs]

It's really an important bug and after months there is nothing fixed. This bug basically doesn't allow me to reproduce a remote database locally in a straight forward way. Imagine if Cloudflare didn't have hundreds of thousands of paying users, how this would work.

✘ [ERROR] HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.

  stack: ; kj::getStackTrace() =
/node_modules/wrangler/node_modules/@cloudflare/workerd-linux-64/bin/workerd@25501e6
/node_modules/wrangler/node_modules/@cloudflare/workerd-linux-64/bin/workerd@254b55c
...

[lakano]

this is on our plan to address this quarter, hope to start in May

Hi @vy-ton This bug haven't been addressed in May, nor in Q2 as planned, nor in Q3, and we are now in Q4, can you give us an new ETA please?

[taojoe]

It looks like a memory leak problem, because when using the following code to import sql, the first few items are normal, then the above information is reported, and then the following error occurs:

[cause]: Error: connect ECONNREFUSED 127.0.0.1:36341 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1637:16) { errno: -111, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 36341 } }

async function main() {
	// Read the SQL file
  const sqlContent = await fspromises.readFile(SQL_PATH, 'utf-8');

  // Split the file by '\n\n' to get individual SQL statements
  const sqlStatements = sqlContent.split('\n\n').filter(stmt => stmt.trim().length > 0);

  const proxy = await getPlatformProxy<Env>();
  const { env } = proxy;
  const D1 = env.DB;

  // Execute each SQL statement using D1
  for (const sql of sqlStatements) {
  let sql0=sql;
  if (sql0.trim().startsWith('CREATE TABLE')) {
  	sql0 = sql0.replace(/\n/g, ' ').trim();
  }
    try {
      await D1.exec(sql0);
      console.log('Executed SQL statement successfully.');
    } catch (error) {
      console.error('Error executing SQL statement:', error);
      // Continue with next statement or decide to stop
    }
  }

  await proxy.dispose();
}

[mrankine]

Yes, I am also hitting this error in local development when trying to exec larger SQL statements using the Workers D1 binding. So it's not just wrangler execute that's affected.

@vy-ton @petebacondarwin is there any update on this?

[0] kj/table.c++:57: error: HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.
...
[0] *** Received signal #11: Segmentation fault
[0] stack: /home/[..]/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2d1978f /home/foo/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2d14466 /home/foo/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@2d18a28 /home/foo/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@1f771e2 /home/foo/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@371a0be /home/foo/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3719d90 /home/foo/node_modules/.pnpm/@[email protected]/node_modules/@cloudflare/workerd-linux-64/bin/workerd@3584875

[taojoe]

In the testing environment, this issue can be temporarily resolved by importing SQL using the sqlite command.

[treeder]

This is still an issue. You can use this instead though in the meantime (I'm surprised it's not using this under the hood?):

sqlite3 .wrangler/state/v3/d1/miniflare-D1DatabaseObject/43f2xxxbbc25.sqlite < ./db.sql