Potential bug when updating json DB

[adame21]

Hi, So I've been using this package a few days now building a system. I'm using this package to "cache" SQL table data so I can use it locally later. It's supposed to update the "cache" weekly, or when the app is restarted.

What the problem is that sometimes when the json file already exists it fails to push the new data giving the error of: "Can't Load Database" And the inner error of: "SyntaxError: Unexpected end of JSON input"

This gets resolved if I delete the existing files and run it again, but if that's the only fix currently I cant use this very useful package.

Some code example: (this runs everytime the app is started for several tables)

let dataToCache = await knex.raw(`SELECT * FROM [BTS].[dbo].[BTSStatusConv] ORDER BY [BTSStatusConv].SystemID , [BTSStatusConv].StatusID`);

for (let i = 0; i < dataToCache.length; i++) {

 db.push(`/${dataToCache[i].SystemID}_${dataToCache[i].StatusID}`, dataToCache[i]);

}

Am I using this package wrong? Is it an actual bug?

Thanks

[Belphemur]

Hello,

It's an interesting usage of the lib.

Basically the lib is complaining the json file got corrupted between reboot of the app.

Can you share the content of the json file?

How did you configure the jsonDB?

---

From: adam [email protected] Sent: Wednesday, September 18, 2019 06:52 To: Belphemur/node-json-db Cc: Subscribed Subject: [Belphemur/node-json-db] Potential bug when updating json DB (#97)

Hi, So I've been using this package a few days now building a system. I'm using this package to "cache" SQL table data so I can use it locally later. It's supposed to update the "cache" weekly, or when the app is restarted.

What the problem is that sometimes when the json file already exists it fails to push the new data giving the error of: "Can't Load Database" And the inner error of: "SyntaxError: Unexpected end of JSON input"

This gets resolved if I delete the existing files and run it again, but if that's the only fix currently I cant use this very useful package.

Some code example: (this runs everytime the app is started for several tables)

let dataToCache = await knex.raw(SELECT * FROM [BTS].[dbo].[BTSStatusConv] ORDER BY [BTSStatusConv].SystemID , [BTSStatusConv].StatusID);

for (let i = 0; i < dataToCache.length; i++) {

db.push(/${dataToCache[i].SystemID}_${dataToCache[i].StatusID}, dataToCache[i]);

}

Am I using this package wrong? Is it an actual bug?

Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[adame21]

Hi, Thanks for responding! the config looks like this:

const { JsonDB } = require('node-json-db')
const { Config } = require('node-json-db/dist/lib/JsonDBConfig')
var cachePath = require('path').join(__dirname, '../', '/cache/')
const db = new JsonDB(new Config(cachePath + getVariableValue("BTS_QUERY_RECORD_JSON_DB_NAME"), true, true, '/'));

I'll just state that getVariableValue only brings an env var for the name of the json file which is basically a string.

and the json looks like this: (this is just a partial piece of it, the full thing is 700+ entries)

{
    "1_01": {
        "SystemID": 1,
        "StatusID": "01",
        "StatusDesc": "התקבל",
        "AgentsStatusDesc": "התקבל",
        "CustomerStatusDesc": "התקבל",
        "InternalStatusDesc": null,
        "StatusGroupID": null,
        "IVRStatusDesc": "התקבל IVR"
    },
    "1_1": {
        "SystemID": 1,
        "StatusID": "1",
        "StatusDesc": "העברת בעלות",
        "AgentsStatusDesc": "התקבלה",
        "CustomerStatusDesc": null,
        "InternalStatusDesc": null,
        "StatusGroupID": null,
        "IVRStatusDesc": null
    },
    "1_10": {
        "SystemID": 1,
        "StatusID": "10",
        "StatusDesc": "ניסיון",
        "AgentsStatusDesc": "",
        "CustomerStatusDesc": null,
        "InternalStatusDesc": null,
        "StatusGroupID": null,
        "IVRStatusDesc": null
    },
    "1_11": {
        "SystemID": 1,
        "StatusID": "11",
        "StatusDesc": "עבר למיכון",
        "AgentsStatusDesc": "בטיפול",
        "CustomerStatusDesc": "בעבודה",
        "InternalStatusDesc": null,
        "StatusGroupID": null,
        "IVRStatusDesc": null
    }
}

I kept testing and noticed it only happens sometimes, I can run it 10 times and it updates properly, every once in a while (once or twice during the whole 8 hour work day) the error happens.

[olignyf]

@adame21 but the json data above seems valid? It passes okay through a json validator. Did you eventually figured it out?

[psipherious]

I know this is a pretty old-issue but I came up with an workaround people might find useful.

I started to experience this same issue, no idea why/what caused it as I wasn't writing a lot of data. Perhaps I wasn't asyncing things properly, I don't really know, but I had it set to save automatically on every push. And every so often something would get messed up and the database would become 0 bytes.

When the db.json exists but has 0 data in it, it's considered invalid json, as when this library first initializes a brand new database it set the file to '{}' - which makes it valid JSON, otherwise if you try to connect it crashes out with an error that it couldn't read the database (due to invalid JSON).

For my use-case it isn't terribly important if the data gets blown away when this happens because my server basically will re-create/repopulate all the data when needed upon a server reboot. Since this is the case, I just needed to fix the 0 bytes file back to valid JSON.

So I wrote a very basic JSON validator and then extended the JsonDB class with some added functions that will validate it, and if it fails, it'll repopulate the db.json with a fresh '{}'. Then I can use it again.

For context, I am running v2.2.0.

In my "extension" I created a .read() and .write() which are just wrappers for .getData() & .push() - .read() uses .getObjectDefault() instead of .getData() so that I can return undefined in the event the dataPath doesn't exist.

There is also a variable failDefault which you can change if you want it to return something other than 'undefined', such as an actual error code or something more descriptive like fail:retrieval or what have you.

common.js:

import fs from 'fs';    // needed for importJSON()
import { JsonDB, Config } from 'node-json-db';

class jsonDB extends JsonDB {
  constructor(dbPath, autoSave = true, humanReadable = false, separator = '/') {
    let dbFile = importJson(dbPath);
    if (!dbFile) {
      console.error('Database is corrupted, resetting to empty');
      fs.writeFileSync(dbPath, '{}');
    }
    super(new Config(dbPath, autoSave, humanReadable, separator));
    this.failDefault = undefined;
    dbFile = undefined;   // no longer needed, set undefined for garbage collection
  }

  async write(datPath, data) {
    const result = await this.push(datPath, data);
    return result;
  }

  async read(datPath, nonObj = false) {    // you can use "getData" if you use db.read(dataPath, true);
    let result;
    if (nonObj) {
      result = await this.getData(datPath);
    } else {
      result = await this.getObjectDefault(datPath, this.failDefault);
    }
    return result;
  }
}

// Simple Json file importer with error checking
export function importJson(filePath) {
  try {
    if (!fs.existsSync(filePath)) {           // file does not exist: exit
      console.error('File not found: ' + filePath);
      process.exit(1);
    }
    const data = fs.readFileSync(filePath);   // read it
    const json = JSON.parse(data);            // parse it
    return json;
  } catch (err) {                             // handle JSON parse and file read errors
    console.error('Failed to parse JSON from file: ' + filePath);
    console.error(err.message);
    // process.exit(1);
    return false;
  }
}

Usage:

import { fileURLToPath } from 'url';
import path from 'path';
import { jsonDB } from './common.js';

// Establish ability to use __dirname in ESM
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);           // these do not include trailing slash

// Connect the database
const db = new jsonDB(`${__dirname}/db.json`, true, false, '/');

// Use the database
db.write('/test/path', 'hello world');
const result = db.read('test/path');

console.info(result);