cnpmcore icon indicating copy to clipboard operation
cnpmcore copied to clipboard

使用接口同步包的时候,会报错 ER_DUP_ENTRY

Open laoboxie opened this issue 2 years ago • 3 comments

在使用接口同步一个包时,会报 Duplicate entry 的错误。 接口:http://127.0.0.1:7001/-/package/async-validator/syncs 接口参数 body:{ "force": true, "skipDependencies": false } 错误信息:

2022-10-09 15:46:35,355 ERROR 47296 [-/127.0.0.1/39ad8b40-4794-11ed-a306-2b9d8248ddf2/7848494.761ms SCHEDULE /__schedule?path=/Users/admin/work/cnpmcore/app/port/schedule/SyncPackageWorker.js&interval=60000&type=worker&env=&disable=false&immediate=false] nodejs.ER_DUP_ENTRYError: ER_DUP_ENTRY: Duplicate entry '63426607d20c8db8c0450c98-3.0.0-beta.20' for key 'package_versions.uk_package_id_version'
    at Query.Sequence._packetToError (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
    at Query.ErrorPacket (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/sequences/Query.js:79:18)
    at Protocol._parsePacket (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Protocol.js:291:23)
    at Parser._parsePacket (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Parser.js:433:10)
    at Parser.write (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Parser.js:43:10)
    at Protocol.write (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Protocol.js:38:16)
    at Socket.<anonymous> (/Users/admin/work/cnpmcore/node_modules/mysql/lib/Connection.js:88:28)
    at Socket.<anonymous> (/Users/admin/work/cnpmcore/node_modules/mysql/lib/Connection.js:526:10)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12)
    --------------------
    at Protocol._enqueue (/Users/admin/work/cnpmcore/node_modules/mysql/lib/protocol/Protocol.js:144:48)
    at PoolConnection.query (/Users/admin/work/cnpmcore/node_modules/mysql/lib/Connection.js:198:25)
    at /Users/admin/work/cnpmcore/node_modules/leoric/src/drivers/mysql/index.js:90:18
    at new Promise (<anonymous>)
    at MysqlDriver.query (/Users/admin/work/cnpmcore/node_modules/leoric/src/drivers/mysql/index.js:89:21)
    at MysqlDriver.cast (/Users/admin/work/cnpmcore/node_modules/leoric/src/drivers/abstract/index.js:45:23)
    at Spell.ignite (/Users/admin/work/cnpmcore/node_modules/leoric/src/spell.js:454:37)
    at Spell.then (/Users/admin/work/cnpmcore/node_modules/leoric/src/spell.js:471:17)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
code: "ER_DUP_ENTRY"
errno: 1062
sqlMessage: "Duplicate entry '63426607d20c8db8c0450c98-3.0.0-beta.20' for key 'package_versions.uk_package_id_version'"
sqlState: "23000"
index: 0
sql: "INSERT INTO `package_versions` (`gmt_create`, `gmt_modified`, `package_id`, `package_version_id`, `version`, `abbreviated_dist_id`, `manifest_dist_id`, `tar_dist_id`, `readme_dist_id`, `publish_time`) VALUES ('2022-10-09 15:46:35.353', '2022-10-09 15:46:35.353', '63426607d20c8db8c0450c98', '63427c3ed20c8db8c0455151', '3.0.0-beta.20', '63427c3ed20c8db8c0455150', '63427c3ed20c8db8c045514e', '63427c33d20c8db8c045510f', '63427c3ed20c8db8c045514f', '2020-07-09 00:45:45.537')"
taskId: "63422f631be21f52ca8681d0"
name: "ER_DUP_ENTRYError"
pid: 47296
hostname: admindeMac-mini.local

laoboxie avatar Oct 09 '22 07:10 laoboxie

存在并发同步主键冲突,目前会自动重试最终一致性。

fengmk2 avatar Feb 13 '23 13:02 fengmk2

这个还是有点问题,可能在并发的时候出一些数据错误。得做成抢占式,同一时间一个包只能有一个任务在同步。

killagu avatar Feb 13 '23 14:02 killagu

https://github.com/cnpm/cnpmcore/blob/master/app/core/service/PackageSyncerService.ts#L350

做了一些优化来减少发生概率:

  1. 同步任务创建时,如果任务还没有执行,任务会进行合并。
  2. 引入了 cork 机制,默认等包同步完成后统一触发 change 事件,确保下游请求时任务已更新。

现在还依赖 db 插入来做锁,理论上概率很小。

elrrrrrrr avatar Jun 11 '23 17:06 elrrrrrrr