cnpmcore
cnpmcore copied to clipboard
使用接口同步包的时候,会报错 ER_DUP_ENTRY
在使用接口同步一个包时,会报 Duplicate entry 的错误。 接口:http://127.0.0.1:7001/-/package/async-validator/syncs 接口参数 body:{ "force": true, "skipDependencies": false } 错误信息:
2022-10-09 15:46:35,355 ERROR 47296 [-/127.0.0.1/39ad8b40-4794-11ed-a306-2b9d8248ddf2/7848494.761ms SCHEDULE /__schedule?path=/Users/admin/work/cnpmcore/app/port/schedule/SyncPackageWorker.js&interval=60000&type=worker&env=&disable=false&immediate=false] nodejs.ER_DUP_ENTRYError: ER_DUP_ENTRY: Duplicate entry '63426607d20c8db8c0450c98-3.0.0-beta.20' for key 'package_versions.uk_package_id_version'
at Query.Sequence._packetToError (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
at Query.ErrorPacket (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/sequences/Query.js:79:18)
at Protocol._parsePacket (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Protocol.js:291:23)
at Parser._parsePacket (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Parser.js:433:10)
at Parser.write (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Parser.js:43:10)
at Protocol.write (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Protocol.js:38:16)
at Socket.<anonymous> (/Users/admin/work/cnpmcore/node_modules/mysql/lib/Connection.js:88:28)
at Socket.<anonymous> (/Users/admin/work/cnpmcore/node_modules/mysql/lib/Connection.js:526:10)
at Socket.emit (node:events:527:28)
at addChunk (node:internal/streams/readable:315:12)
--------------------
at Protocol._enqueue (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Protocol.js:144:48)
at PoolConnection.query (/Users/admin/work/cnpmcore/node_modules/mysql/lib/Connection.js:198:25)
at /Users/admin/work/cnpmcore/node_modules/leoric/src/drivers/mysql/index.js:90:18
at new Promise (<anonymous>)
at MysqlDriver.query (/Users/admin/work/cnpmcore/node_modules/leoric/src/drivers/mysql/index.js:89:21)
at MysqlDriver.cast (/Users/admin/work/cnpmcore/node_modules/leoric/src/drivers/abstract/index.js:45:23)
at Spell.ignite (/Users/admin/work/cnpmcore/node_modules/leoric/src/spell.js:454:37)
at Spell.then (/Users/admin/work/cnpmcore/node_modules/leoric/src/spell.js:471:17)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
code: "ER_DUP_ENTRY"
errno: 1062
sqlMessage: "Duplicate entry '63426607d20c8db8c0450c98-3.0.0-beta.20' for key 'package_versions.uk_package_id_version'"
sqlState: "23000"
index: 0
sql: "INSERT INTO `package_versions` (`gmt_create`, `gmt_modified`, `package_id`, `package_version_id`, `version`, `abbreviated_dist_id`, `manifest_dist_id`, `tar_dist_id`, `readme_dist_id`, `publish_time`) VALUES ('2022-10-09 15:46:35.353', '2022-10-09 15:46:35.353', '63426607d20c8db8c0450c98', '63427c3ed20c8db8c0455151', '3.0.0-beta.20', '63427c3ed20c8db8c0455150', '63427c3ed20c8db8c045514e', '63427c33d20c8db8c045510f', '63427c3ed20c8db8c045514f', '2020-07-09 00:45:45.537')"
taskId: "63422f631be21f52ca8681d0"
name: "ER_DUP_ENTRYError"
pid: 47296
hostname: admindeMac-mini.local
存在并发同步主键冲突,目前会自动重试最终一致性。
这个还是有点问题,可能在并发的时候出一些数据错误。得做成抢占式,同一时间一个包只能有一个任务在同步。
https://github.com/cnpm/cnpmcore/blob/master/app/core/service/PackageSyncerService.ts#L350
做了一些优化来减少发生概率:
- 同步任务创建时,如果任务还没有执行,任务会进行合并。
- 引入了 cork 机制,默认等包同步完成后统一触发 change 事件,确保下游请求时任务已更新。
现在还依赖 db 插入来做锁,理论上概率很小。