TDengine
TDengine copied to clipboard
taosdata
@tdengine/[email protected] REST API concurrently inserting enough rows throws error "connect ENOBUFS"
Bug描述
@tdengine/[email protected] REST API 并发迁移40张sqlite数据表，每张表有1千万条数据，当前仅使用了每张表的1000条数据进行并发插入，仅相当于40 * 1000的并发量，每次验证只有约15张表完成了1000条的插入任务，其余的都发生了报错：
超级表
CREATE STABLE s (f_ts timestamp, f_value bigint, f_open float, f_upper float, f_lower float, f_close float, f_price float, f_high float, f_low float) TAGS (f_code NCHAR(10), f_name NCHAR(10));
自动创建表插入数据
INSERT INTO db_quant.d_000659 USING db_quant.d (f_code, f_name) TAGS ('000659', '000659') (f_ts, f_price, f_amount, f_turnover, f_count, f_type) VALUES ('2024-01-24 11:12:21', 2.15, 2, 430, 1, 1)
INSERT INTO db_quant.d_000659 USING db_quant.d (f_code, f_name) TAGS ('000659', '000659') (f_ts, f_price, f_amount, f_turnover, f_count, f_type) VALUES ('2024-01-24 11:12:30', 2.15, 4, 860, 2, 1)
迁移脚本
TD.ts
import { connect, options, type TDengineCursor } from '@tdengine/rest'
let database: TDengineCursor
export async function TD () {
if (!database) {
options.path = '/rest/sql'
options.host = 'td1'
options.port = 6041
const conn = connect(options)
database = conn.cursor()
}
return database
}
index.ts
import { readDirectoryFiles, targetDir } from './read'
import Database from 'better-sqlite3'
import { TD } from './TD'
import path from 'path'
async function update (file) {
const startTime = performance.now()
console.log(`Updating database in ${file}`)
const db = new Database(file)
const stm = db.prepare('SELECT * FROM t_stocks_details limit 1000')
const taos = await TD()
const requestMap = stm.all().map(async (item: any) => {
const insertSql = `INSERT INTO db_quant.d_${item.code} USING db_quant.d (f_code, f_name) TAGS ('${item.code}', '${item.code}') (f_ts, f_price, f_amount, f_turnover, f_count, f_type)
VALUES ('${[item.date.replace(/\//g, '-'), item.time].join(' ')}', ${item.price}, ${item.amount}, ${item.turnover}, ${item.count}, ${item.type})`
console.log({ insertSql })
return taos.query(insertSql)
})
const allSettled = await Promise.allSettled(requestMap)
const result = JSON.stringify({
db: path.basename(file),
takes: performance.now() - startTime,
allSettled
})
console.log(`Updated database in ${file}`, result)
return result
}
async function main () {
console.log({ targetDir })
const files = await readDirectoryFiles(targetDir)
return Promise.allSettled(files.map(async file => update(file)))
}
main().then(console.log).catch(console.error)
执行命令
ts-node index.ts
成功日志：
{"status":"fulfilled","value":{"_code":0,"_rows":1,"_column_meta":[{"columnName":"affected_rows","typeName":"INT","size":4}],"_data":[[1]]}},
失败日志：
{"status":"rejected","reason":{"message":"request to http://td1:6041/rest/sql failed, reason: connect ENOBUFS 172.30.192.35:6041 - Local (undefined:undefined)","type":"system","errno":"ENOBUFS","code":"ENOBUFS"}}
预期行为
- 完成全部迁移工作。
截屏日志
环境
客户端环境
设备名称 DESKTOP-E61ENE7
处理器 Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz 4.00 GHz
机带 RAM 64.0 GB
设备 ID 9BD07667-B3AF-4D3D-81A1-CF8BDEF96AE0
产品 ID 00391-70000-00000-AA073
系统类型 64 位操作系统, 基于 x64 的处理器
笔和触控 没有可用于此显示器的笔或触控输入
版本 Windows 10 专业工作站版
版本号 22H2
安装日期 2022/6/20
操作系统内部版本 19045.4170
体验 Windows Feature Experience Pack 1000.19054.1000.0
客户端nodejs环境
$ node -v
v18.13.0
服务器ubuntu环境
root@DESKTOP-E61ENE7:~# lshw
desktop-e61ene7
description: Computer
width: 64 bits
capabilities: smp vsyscall32
*-core
description: Motherboard
physical id: 0
*-memory
description: System memory
physical id: 1
size: 32GiB
*-cpu
product: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
vendor: Intel Corp.
physical id: 2
bus info: cpu@0
version: 6.94.3
width: 64 bits
capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves md_clear flush_l1d arch_capabilities
configuration: microcode=4294967295
*-scsi:0
description: SCSI storage controller
product: Virtio console
vendor: Red Hat, Inc.
physical id: 3
bus info: pci@1eab:00:00.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: scsi msix bus_master cap_list
configuration: driver=virtio-pci latency=64
resources: iomemory:90-8f iomemory:90-8f iomemory:90-8f irq:0 memory:9ffe00000-9ffe00fff memory:9ffe01000-9ffe01fff memory:9ffe02000-9ffe02fff
*-virtio0 UNCLAIMED
description: Virtual I/O device
physical id: 0
bus info: virtio@0
configuration: driver=virtio_console
*-display
description: 3D controller
product: Microsoft Corporation
vendor: Microsoft Corporation
physical id: 4
bus info: pci@5eaa:00:00.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: bus_master cap_list
configuration: driver=dxgkrnl latency=0
resources: irq:0
*-generic
description: System peripheral
product: Virtio file system
vendor: Red Hat, Inc.
physical id: 0
bus info: pci@e5ea:00:00.0
version: 01
width: 64 bits
clock: 33MHz
capabilities: msix bus_master cap_list
configuration: driver=virtio-pci latency=64
resources: iomemory:e0-df iomemory:e0-df iomemory:c0-bf irq:0 memory:e00000000-e00000fff memory:e00001000-e00001fff memory:c00000000-dffffffff
*-virtio1 UNCLAIMED
description: Virtual I/O device
physical id: 0
bus info: virtio@1
configuration: driver=virtiofs
*-pnp00:00
product: PnP device PNP0b00
physical id: 5
capabilities: pnp
configuration: driver=rtc_cmos
*-scsi:1
physical id: 6
logical name: scsi0
*-disk:0
description: SCSI Disk
product: Virtual Disk
vendor: Linux
physical id: 0.0.0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: 1.0
size: 388MiB
capabilities: extended_attributes large_files huge_files extents ext2 initialized
configuration: ansiversion=5 filesystem=ext2 logicalsectorsize=512 sectorsize=512 state=clean
*-disk:1
description: Linux swap volume
product: Virtual Disk
vendor: Msft
physical id: 0.0.1
bus info: scsi@0:0.0.1
logical name: /dev/sdb
version: 1
serial: 8ae81d30-4fa9-4f3e-9e6d-cda34fc9a8c7
size: 8GiB
capacity: 8GiB
capabilities: swap initialized
configuration: ansiversion=5 filesystem=swap logicalsectorsize=512 pagesize=4096 sectorsize=4096
*-disk:2
description: EXT4 volume
product: Virtual Disk
vendor: Linux
physical id: 0.0.2
bus info: scsi@0:0.0.2
logical name: /dev/sdc
logical name: /
logical name: /mnt/wslg/distro
logical name: /snap
version: 1.0
serial: 51b56668-88d9-4fdd-ac8d-befdbad73e8d
size: 1TiB
capabilities: journaled extended_attributes large_files huge_files dir_nlink recover 64bit extents ext4 ext2 initialized
configuration: ansiversion=5 created=2024-02-22 01:41:55 filesystem=ext4 lastmountpoint=/distro logicalsectorsize=512 modified=2024-03-30 20:43:07 mount.fstype=ext4 mount.options=rw,relatime,discard,errors=remount-ro,data=ordered mounted=2024-03-30 20:43:07 sectorsize=4096 state=mounted
*-usbhost:0
product: USB/IP Virtual Host Controller
vendor: Linux 5.15.146.1-microsoft-standard-WSL2 vhci_hcd
physical id: 1
bus info: usb@1
logical name: usb1
version: 5.15
capabilities: usb-2.00
configuration: driver=hub slots=8 speed=480Mbit/s
*-usbhost:1
product: USB/IP Virtual Host Controller
vendor: Linux 5.15.146.1-microsoft-standard-WSL2 vhci_hcd
physical id: 2
bus info: usb@2
logical name: usb2
version: 5.15
capabilities: usb-3.00
configuration: driver=hub slots=8 speed=5000Mbit/s
*-network
description: Ethernet interface
physical id: 3
logical name: eth0
serial: 00:15:5d:1c:44:71
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=hv_netvsc driverversion=5.15.146.1-microsoft-standard-W duplex=full firmware=N/A ip=172.30.192.35 link=yes multicast=yes speed=10Gbit/s
root@DESKTOP-E61ENE7:~#
TDengine版本
root@DESKTOP-E61ENE7:~# taos
Welcome to the TDengine Command Line Interface, Client Version:3.2.2.0
Copyright (c) 2023 by TDengine, all rights reserved.
******************************** Tab Completion ************************************
* The TDengine CLI supports tab completion for a variety of items, *
* including database names, table names, function names and keywords. *
* The full list of shortcut keys is as follows: *
* [ TAB ] ...... complete the current word *
* ...... if used on a blank line, display all supported commands *
* [ Ctrl + A ] ...... move cursor to the st[A]rt of the line *
* [ Ctrl + E ] ...... move cursor to the [E]nd of the line *
* [ Ctrl + W ] ...... move cursor to the middle of the line *
* [ Ctrl + L ] ...... clear the entire screen *
* [ Ctrl + K ] ...... clear the screen after the cursor *
* [ Ctrl + U ] ...... clear the screen before the cursor *
**************************************************************************************
Server is Community Edition.
taos>
