DiceDB
Inconsistent `mSet`: Connection Reset
Steps to reproduce
Hi, I am trying to dump some data on dicedb instance hosted on GCP via NodeJS + Redis
Node Version: 20 DiceDB Version: Latest (checked out from main branch 2 days ago) Node Redis Package: https://www.npmjs.com/package/redis - 4.7.0
Steps:
- Get data from DB (Attached the data) table1.csv table2.csv
- Iterate over it and dump to redis in batches via mSet
Code:
// Initialize objects to store tickets and sales orders
const tickets = {};
const salesOrders = {};
// Iterate over each row in the result set
for (const row of ticketrows) {
// Create a ticket key using the uniqueReferenceID
const ticketKey = `${row.uniqueReferenceID}`;
// Trim the ticket data to only include the necessary fields
const currentTicket = {
tkt: ticketKey,
bvID: row.batchVariantID,
bvName: row.batchVariantName,
value: row.productValue,
};
const ticketData = {
so: row.salesOrderID,
qty: 1,
tkts: [currentTicket],
};
// Store the ticket data as a JSON string in the tickets object
tickets[ticketKey] = JSON.stringify(ticketData);
}
// Iterate over each row in the result set
for (const row of sorows) {
// Create a ticket key using the uniqueReferenceID
const ticketKey = `${row.uniqueReferenceID}`;
// Trim the ticket data to only include the necessary fields
const currentTicket = {
tkt: ticketKey,
bvID: row.batchVariantID,
bvName: row.batchVariantName,
value: row.productValue,
};
const ticketData = {
so: row.salesOrderID,
qty: 1,
tkts: [currentTicket],
};
// Create a sales order key using the salesOrderID and batchVariantID
const salesOrderKey = `${row.salesOrderID}`;
// Check if the sales order already exists in the salesOrders object
if (salesOrders[salesOrderKey] !== undefined) {
// Increment the quantity and add the ticket to the sales order
let tempSalesOrder = JSON.parse(salesOrders[salesOrderKey]);
tempSalesOrder.qty++;
tempSalesOrder.tkts.push(currentTicket);
salesOrders[salesOrderKey] = JSON.stringify(tempSalesOrder);
} else {
salesOrders[salesOrderKey] = JSON.stringify(ticketData);
}
}
// Get the count of tickets and sales orders
let ticketCount = Object.keys(tickets).length;
let salesOrderCount = Object.keys(salesOrders).length;
console.log('Active tickets:', ticketCount);
console.log('Active sales orders:', salesOrderCount);
// Dump tickets to Redis in batches
if (ticketCount > 0) {
const batchSize = 10;
for (let i = 0; i < ticketCount; i += batchSize) {
const batch = Object.entries(tickets).slice(i, i + batchSize);
await redis.mSet(batch);
}
console.log('Dumped active tickets to Redis');
console.log('Dumped all active tickets to Redis in bulk');
} else {
console.log('No active tickets to dump');
}
// Dump sales orders to Redis in batches
if (salesOrderCount > 0) {
const batchSize = 5;
for (let i = 0; i < salesOrderCount; i += batchSize) {
const batch = Object.entries(salesOrders).slice(i, i + batchSize);
await redis.mSet(batch);
}
console.log('Dumped active sales orders to Redis');
} else {
console.log('No active sales orders to dump');
}
// Return true to indicate successful completion
return true;
3 out of 5 times it works.
Other times I get a connection error
DiceDB logs from server:
1:38PM INF FLUSHDB called args={}
1:38PM INF FLUSHDB called args={}
1:39PM INF FLUSHDB called args={}
1:39PM WRN connection timeout
Expected output
The expected output when the above set of commands (maybe when run on Redis)
Works all the time.
Observed output
The observed output when the above set of commands when run on DiceDB
WRN connection timeout
Expectations for resolution
This issue will be considered resolved when the following things are done
- changes in the
dicecode to meet the expected behavior - addition of relevant test case to ensure we catch the regression
You can find the tests under the integration_tests directory of the dice repository and the steps to run are in the README file. Refer to the following links to set up DiceDB and Redis 7.2.5 locally
Thanks for reporting this, we're looking into it. Will keep this thread posted.
@JyotinderSingh can u plz assign this issue to me
Feel free to look into this, but fyi, a few other team members are also investigating this. Please report your findings here.
@maplenk , quick question, could you provide any info regarding the infra for this setup? do you have both the server and dicedb running on the same machine or different ones?
Hey, Installed dicedb on Ubuntu 22 on GCP Connecting to it remotely via mac
@maplenk I noticed that dice-db instance where you are facing this issue is using the older server. We have recently moved our server to new multi-threaded RESP server. Can you please re-run and let us know?
Hi Just updated and now cannot seem to connect Getting: ERR: Multiple commands not supported
Please share the exact command and the command format.
Running MSET works fine through CLI.
FYI, we support RESP2 format.
Logs from server where dicedb is running:
root@redis:~/dice# go run main.go
2024-11-26T16:44:28+05:30 INF Field Logging.LogDir failed validation: dirpath
2024-11-26T16:44:28+05:30 INF Setting default value for Logging.LogDir to: /tmp/dicedb
██████╗ ██╗ ██████╗███████╗██████╗ ██████╗
██╔══██╗██║██╔════╝██╔════╝██╔══██╗██╔══██╗
██║ ██║██║██║ █████╗ ██║ ██║██████╔╝
██║ ██║██║██║ ██╔══╝ ██║ ██║██╔══██╗
██████╔╝██║╚██████╗███████╗██████╔╝██████╔╝
╚═════╝ ╚═╝ ╚═════╝╚══════╝╚═════╝ ╚═════╝
2024-11-26T16:44:28+05:30 INF starting DiceDB version=0.1.0
2024-11-26T16:44:28+05:30 INF running with port=7379
2024-11-26T16:44:28+05:30 INF running with cores=1
2024-11-26T16:44:28+05:30 INF running with shards=1
2024-11-26T16:44:28+05:30 INF running with watch=false
2024-11-26T16:44:28+05:30 INF running with profiling=false
2024-11-26T16:44:28+05:30 INF running with persistence=true
2024-11-26T16:44:44+05:30 INF Stopping io-thread id=W-15532-1
2024-11-26T16:44:44+05:30 INF Stopping io-thread id=W-16105-2
2024-11-26T16:44:45+05:30 INF Stopping io-thread id=W-16721-3
2024-11-26T16:44:46+05:30 INF Stopping io-thread id=W-17536-4
2024-11-26T16:44:46+05:30 INF Stopping io-thread id=W-18408-5
2024-11-26T16:44:47+05:30 INF Stopping io-thread id=W-19213-6
2024-11-26T16:44:48+05:30 INF Stopping io-thread id=W-20015-7
2024-11-26T16:44:49+05:30 INF Stopping io-thread id=W-20879-8
2024-11-26T16:45:45+05:30 INF Stopping io-thread id=W-77035-9
2024-11-26T16:45:46+05:30 INF Stopping io-thread id=W-77664-10
2024-11-26T16:45:46+05:30 INF Stopping io-thread id=W-78284-11
2024-11-26T16:45:47+05:30 INF Stopping io-thread id=W-78999-12
@maplenk I am unable to look into the issue further. Sorry about the delay. Have asked @lucifercr07 for help here...