Location of datafile?

[steveparker-1984]

I'd like my datafile to be portable. I have the following:

var Datastore = require('nedb')
  , db = new Datastore({ filename: 'datafile.nedb', autoload: true });

Data is definitely persisted, but I can't locate this file anywhere. I've tried setting a more explicit path but the file isn't written to that location. How can I ship a data store with my app this way?

[nia1048596]

I just figured out that when using some bundling tools like webpack, its default target is browser, which means the nedb will use IndexedDB instead (see browser-specific/lib/storage.js below for details).

In nedb's package.json:

  "browser": {
    "./lib/customUtils.js": "./browser-version/browser-specific/lib/customUtils.js",
    "./lib/storage.js": "./browser-version/browser-specific/lib/storage.js"
  },

The browser field tells webpack to use browser-specific version of storage.js instead, which will not create any files. The solution is to change the target to, in my case, electron-main in webpack.config.js.

Hope this helps.

[JamesMGreene]

Good tip, @nia1048596.

[JamesMGreene]

When using NeDB on Node.js, I've always used an absolute file path to be certain where my datafiles will be stored (and they are always in the spot I specify).

However, if your file path is not absolute, it should be placed relative to the current working directory of the Node executable process itself (typically meaning the directory your user/program was "located in" when you launched the Node executable).

[steveparker-1984]

I've been trying to use angular 2 with electron to build an app, which uses webpack but seems not to allow configuration of webpack, so I think my issue is as described by @nia1048596. I've tried numerous approaches to getting around this but nothing seems to give me anything close to a reasonable workflow. Going to rethink the angular approach.

[JamesMGreene]

@steveparker-kcom: Seems like it is possible to configure based on some blog posts I've seen.

See the "Configuring Webpack" section (and following sections) of this article: https://semaphoreci.com/community/tutorials/setting-up-angular-2-with-webpack

[FossPrime]

If you run into this while trying to run NeDB in browser with webpack but don't see IndexedDB in the Dev tools... close the browser tab and open it again... dev tools won't always show new IndexedDB stores

You do not need to point directly at the browserversion with webpack if you intend to use it in the browser with indexedDB. This will suffice:

ES Modules version:

import NeDB from  'nedb';

cjs:

const NeDB = require('nedb');

[VincentMore]

We had this trouble (electron, ng4, webpack) and I think the fix was to add this in our webpak config:

module.exports = { ... , target: 'electron-renderer' };

I think this was to solve the location/import for electron and nedb. I don't know it it requires anything else but our app work pretty well!

On Oct 17, 2017 10:28 AM, "Steven Parker" [email protected] wrote:

I've been trying to use angular 2 with electron to build an app, which uses webpack but seems not to allow configuration of webpack, so I think my issue is as described by @nia1048596 https://github.com/nia1048596. I've tried numerous approaches to getting around this but nothing seems to give me anything close to a reasonable workflow. Going to rethink the angular approach.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/louischatriot/nedb/issues/531#issuecomment-337249012, or mute the thread https://github.com/notifications/unsubscribe-auth/ASq6cfHNkG0-yykKQ_fKm6esAimv8Yrpks5stLl5gaJpZM4PxuBQ .

[steveparker-1984]

Based on @VincentMore comment I figured I might have more luck with angular 4. Starting off with the quickstart below and updating the webpack as described gives me the desired result.

https://github.com/maximegris/angular-electron

Thanks all

[YoonTso]

try to create database in the main.js, it will force nedb to create a local file as database, not in the indexDB, it works for me

[Prathik-Jain]

I am working on angular 6 - Suggestions? @steveparker-kcom @nia1048596 @JamesMGreene @rayfoss @VincentMore @NEOTSO

[digimezzo]

Well, figuring this one out, has been a tremendous waste of time on my Electron project. In case other novice Electron developers stumble upon this issue, here's some centralized information. I'm using TypeScript, so pardon the syntax differences:

Nedb let's you create a new auto-loading Datastore using this call:

let db = new Datastore({ filename: 'path/to/datafile', autoload: true });

It appears, however, that this command is only accurate when performed from the main process (for new Electron developers, this is usually your main.ts or main.js file).

If you perform the Datastore creation command from a class which is executed in the renderer process (any class which is executed in the BrowserWindow), nedb ignores the datafile you provided and creates your database in IndexedDB instead. You'll find your database in your application's "userData" directory in a subdirectory \IndexedDB\http....leveldb. In my case it was (on Linux):

\home\myUser\.config\myAppName\IndexedDB\http_localhost_4200.indexeddb.leveldb

If you really want nedb to create and use the database file that you provide during Datastore creation, you must create AND access the data file (add, remove,... documents) from the main process. In my case, I worked around this issue by:

1. Creating the data file from the main process (in main.ts):

let collectionsDb: Datastore = new Datastore({ filename: path.join(app.getPath("userData"), "Collections.db"), autoload: true });

2. Putting the db variable in a global variable in the main process (in main.ts):

const globalAny:any = global;

globalAny.collectionsDb = collectionsDb;

3. Accessing the global db variable from the renderer process by calling the global db variable:

import { remote } from 'electron';

private db = remote.getGlobal('collectionDb');

[bjbk]

@digimezzo Your solution above did indeed create the file. However, and I may have not properly optimized things, my application is very slow now. My db file is very small (currently 272 KB) and even though I have autoload set to true, performance is noticeably slower. Any tips on improving it? I'm using vue.js and have put the db on the Vue instance as noted here: electron-vue.

// datastore.js
import Datastore from 'nedb'
import path from 'path'
import { app } from 'electron'

export default new Datastore({
  filename: path.join(app.getPath('userData'), '/data.db'),
  autoload: true
})

// main.js MAIN PROCESS
import db from './datastore'

const globalAny = global
globalAny.db = db
db.ensureIndex({ fieldName: 'type' }, (err) => {
  // If there was an error, err is not null
  if (err) {
    return err
  }
})

// Renderer
import { remote } from 'electron'
Vue.prototype.$db = remote.getGlobal('db')

[acjay]

This was really frustrating. I spent a whole day trying to figure out why none of my data was being persisted. It's not actually documented that the storage engine is swapped silently when you import the main module from a browser process. And as I'm using nedb-promises, the issue was hidden a step further away.

I don't think it's a good idea to force loading the database in the main process. Hoping there's a way to fix this.

[digimezzo]

@bjbk and @acjay Loading the DB from the main process is indeed a very bad idea. I learned from experience. It causes serious performance issues. For this reason, I switched to Loki.js, which allows creating the DB file anywhere you wish, even when created from a renderer process. You can see how I used it in my Knowte project at https://github.com/digimezzo/knowte-electron

[acjay]

I ended up just forking this project and knocking those lines out of package.json. Loki looks like a totally reasonable option, although it seems to be more suited to in-memory transformations with explicit syncing to disk. I believe nedb differs by asynchronously writing to disk and then calling the continuation you pass when the data is persisted. Please correct me if i'm wrong! But my needs are pretty basic, so I may explore switching if it comes to it.

[Almarane]

Hello,

I have the same issue. I'm using nedb with Electron and Angular 8. I can't save, and I can't load. I did as acjay and removed the two lines, but I have to manually do it each time I pull and install my code on a new computer.

Do you know if this issue will be fixed soon ?

[ThatAnton]

@acjay were you ever able to find a better solution than forking this library and modifying package.json?

[acjay]

I'm not sure it's a "better" solution, but I found a workaround that strikes more directly at the heart of the problem: https://acjay.com/2019/04/19/an-epic-webpack-mystery/

[ThatAnton]

This works great @acjay! And thank you for the write up on the whole process. I recently upgraded from an older version of webpack and this issue popped up. Its been driving me crazy for a while now. While a plugin for a specific package isn't ideal, its far better than forking and modifying the package. Thanks again!

[ArisAgeha]

finally I found the store file. you could use process moniter to checkout which file is using by nedb. for example, you could use software process explorer or process monitor to do this. and for linux you could use lsof command.

in my case, I finally find it in C:\Users\${username}\AppData\Roaming\${projectName}\IndexedDB (windows10)

[Manwe-777]

I'm not sure it's a "better" solution, but I found a workaround that strikes more directly at the heart of the problem: https://acjay.com/2019/04/19/an-epic-webpack-mystery/

Had this problem today and this fixed it, THANKS!

ps, this could very well be a package on its own? or maybe an extension of nedb itself, as I can see its a fairly common problem. @louischatriot

[Hariprabu94]

I just figured out that when using some bundling tools like webpack, its default target is browser, which means the nedb will use IndexedDB instead (see browser-specific/lib/storage.js below for details).

In nedb's package.json:

  "browser": {
    "./lib/customUtils.js": "./browser-version/browser-specific/lib/customUtils.js",
    "./lib/storage.js": "./browser-version/browser-specific/lib/storage.js"
  },

The browser field tells webpack to use browser-specific version of storage.js instead, which will not create any files. The solution is to change the target to, in my case, electron-main in webpack.config.js.

Hope this helps.

Hi, I did same thing. Even its not created any database files. Am using vue electron builder.. so i changed path "./lib/storage.js": "../vue-cli-plugin-electron-builder/lib/webpackConfig.js". Help me pls

[znjameswu]

@digimezzo Your solution above did indeed create the file. However, and I may have not properly optimized things, my application is very slow now. My db file is very small (currently 272 KB) and even though I have autoload set to true, performance is noticeably slower. Any tips on improving it?

This performance regression is caused by inter-process communications because you are using object proxy of db provided by the remote package.

I have found a fix for this performance issue. I think this method is non-intrusive enough.

First register the ipcMain in the main process

// main.ts
(global as any).ipcMain = ipcMain;

Then write a function to transfer data from main process to renderer process.

// transfer.ts
import { IpcMain, ipcRenderer, remote } from 'electron';

function dec2hex(dec: any) {
  return dec < 10 ? '0' + String(dec) : dec.toString(16);
}

function generateId(len?: number) {
  var arr = new Uint8Array((len || 40) / 2);
  window.crypto.getRandomValues(arr);
  return Array.from(arr, dec2hex).join('');
}

export function transferValueFromMainToRenderer<T>(
  value: Promise<T>
): Promise<T> {
  const id = generateId();
  const channel = 'transfer-value-' + id;

  return new Promise((resolve, reject) => {
    (remote.getGlobal('ipcMain') as IpcMain).once(channel, (event) =>
      value.then((v) => event.reply(channel, v))
    );
    ipcRenderer.once(channel, (event, arg) => resolve(arg));
    ipcRenderer.send(channel, {});
  });
}

Then whenever you feel you are going to do a lot of operations on results returned from db, wrap it in transferValueFromMainToRenderer

transferValueFromMainToRenderer(db.users.find<any>())

In my case, I successfully reduced execution time of business logic from 1000+ms down to 12.16 ms

---

Note: any objects transfered by this function is effectively serialized and then deserialized, which means they will lose their prototype information and their member functions.

[robrow]

If someone needs to get this running with Vue CLI 4 and electron-builder:

Paste this plugin code top most in your vue.config.js file:

const fixNedbForElectronRenderer = {
  apply(resolver) {
    resolver
      .getHook("beforeDescribed-relative")
      .tapAsync(
        "FixNedbForElectronRenderer",
        (request, resolveContext, callback) => {
          if (!request.descriptionFileData.name === "nedb") {
            return callback()
          }
 
          let relativePath
          if (
            request.path.startsWith(
              resolver.join(request.descriptionFileRoot, "lib/storage")
            )
          ) {
            relativePath = "lib/storage.js"
          } else if (
            request.path.startsWith(
              resolver.join(
                request.descriptionFileRoot,
                "lib/customUtils"
              )
            )
          ) {
            relativePath = "lib/customUtils.js"
          } else {
            return callback()
          }
 
          const path = resolver.join(
            request.descriptionFileRoot,
            relativePath
          )
          const newRequest = Object.assign({}, request, { path })
          callback(null, newRequest)
        }
      )
  }
};

Last step is to modify the module.exports and add chainWebpack so that it looks something like this:

// vue.config.js
module.exports = {
  chainWebpack: config => {
    config.resolve
      .plugin("fixNedbForElectronRenderer")
      .use(fixNedbForElectronRenderer);
  },
  // rest of your vue.config module
  }
};

For reference the whole vue.config.js file.

// credits to https://acjay.com/2019/04/19/an-epic-webpack-mystery/
const fixNedbForElectronRenderer = {
  apply(resolver) {
    resolver
      .getHook("beforeDescribed-relative")
      .tapAsync(
        "FixNedbForElectronRenderer",
        (request, resolveContext, callback) => {
          if (!request.descriptionFileData.name === "nedb") {
            return callback();
          }

          let relativePath;
          if (
            request.path.startsWith(
              resolver.join(request.descriptionFileRoot, "lib/storage")
            )
          ) {
            relativePath = "lib/storage.js";
          } else if (
            request.path.startsWith(
              resolver.join(request.descriptionFileRoot, "lib/customUtils")
            )
          ) {
            relativePath = "lib/customUtils.js";
          } else {
            return callback();
          }

          const path = resolver.join(request.descriptionFileRoot, relativePath);
          const newRequest = Object.assign({}, request, { path });
          callback(null, newRequest);
        }
      );
  }
};

// vue.config.js
module.exports = {
  chainWebpack: config => {
    config.resolve
      .plugin("fixNedbForElectronRenderer")
      .use(fixNedbForElectronRenderer);
  },
  configureWebpack: {
    devtool: "source-map"
  },
  pluginOptions: {
    electronBuilder: {
      nodeModulesPath: ["./node_modules"],
      nodeIntegration: true // fixes node modules not available in vue components
    },
    i18n: {
      locale: "en",
      fallbackLocale: "en",
      localeDir: "locales",
      enableInSFC: false
    }
  }
};

Credits: https://acjay.com/2019/04/19/an-epic-webpack-mystery/