[*] Charts - Filters using related data break custom charts

[tannerwelsh]

Expected behavior

When creating a custom chart with a filter that is applied to related data (using only the options provided in the UI), I expect the chart to render correctly using only the filtered data.

Actual behavior

The chart breaks, with UI error "Forest cannot render this error." :(

Failure Logs

Successful setup UI:

Then adding a filter using a related document:

... and we see this.

Trying with another related document, using a string-matching query, gives the same error:

Error log from server (from first failed attempt):

OPTIONS /forest/VideoContent?fields%5BVideoContent%5D=_id%2Ctitle%2Cduration%2Corganization_id&fields%5Borganization_id%5D=name&page%5Bnumber%5D=1&page%5Bsize%5D=30&searchExtended=0&sort=title&timezone=-05%3A00 204 0.276 ms - 0
GET /forest/VideoContent?fields%5BVideoContent%5D=_id%2Ctitle%2Cduration%2Corganization_id&fields%5Borganization_id%5D=name&page%5Bnumber%5D=1&page%5Bsize%5D=30&searchExtended=0&sort=title&timezone=-05%3A00 200 23.188 ms - 16326
[forest] 🌳🌳🌳  Unexpected error: $and/$or/$nor must be a nonempty array
MongoError: $and/$or/$nor must be a nonempty array
    at queryCallback (/Users/tannerwelsh/Code/Swayable/swayable-app/node_modules/mongodb-core/lib/cursor.js:223:25)
    at /Users/tannerwelsh/Code/Swayable/swayable-app/node_modules/mongodb-core/lib/connection/pool.js:541:18
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
POST /forest/stats/Response 500 17.612 ms - 77

Error log from server (from second failed attempt):

OPTIONS /forest/Survey?fields%5BSurvey%5D=_id%2Cname%2Corganization_id%2ClaunchAt%2CcloseAt%2Cpretest%2CcompletionCode%2CpreviewLink%2Cstate&fields%5Borganization_id%5D=name&page%5Bnumber%5D=1&page%5Bsize%5D=20&searchExtended=0&sort=-closeAt&timezone=-05%3A00 204 0.167 ms - 0
GET /forest/Survey?fields%5BSurvey%5D=_id%2Cname%2Corganization_id%2ClaunchAt%2CcloseAt%2Cpretest%2CcompletionCode%2CpreviewLink%2Cstate&fields%5Borganization_id%5D=name&page%5Bnumber%5D=1&page%5Bsize%5D=20&searchExtended=0&sort=-closeAt&timezone=-05%3A00 200 489.506 ms - 6619688
[forest] 🌳🌳🌳  Unexpected error: $and/$or/$nor must be a nonempty array
MongoError: $and/$or/$nor must be a nonempty array
    at queryCallback (/Users/tannerwelsh/Code/Swayable/swayable-app/node_modules/mongodb-core/lib/cursor.js:223:25)
    at /Users/tannerwelsh/Code/Swayable/swayable-app/node_modules/mongodb-core/lib/connection/pool.js:541:18
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
POST /forest/stats/Response 500 7.300 ms - 77

Context

• Package Version: 2.7.0
• Express Version: 4.16.2
• Mongoose Version: 5.0.14
• MongoDB Version: 3.6.3

Relevant Schema Code

const ResponseSchema = new Schema({
  survey_id: { type: Schema.Types.ObjectId, required: true, ref: 'Survey' },
  content_id: { type: Schema.Types.ObjectId, required: true, ref: 'Content' },
  startTime: { type: Date, required: true },
  endTime: { type: Date, required: true },
  // ...
})

mongoose.model('Response', ResponseSchema)

// ...

const SurveySchema = new Schema({
  name: { type: String, required: true },
  // ...
})

mongoose.model('Survey', SurveySchema)

// ...

const ContentSchema = new Schema({
  title: { type: String, required: true },
  // ...
})

mongoose.model('Content', ContentSchema)

[arnaudbesnier]

Hi @tannerwelsh, you're right. This is something that work well with all other lianas but not on express-mongoose. Unfortunately this is something we have tried to fix in the past, but it is hard with the current stats implementation (if I remember well).

What we should do as the first step is to disable such filtering option (on related data) for the mongoose projects.

Then the second step could be to try to implement the Live Query feature like we did for other lianas. It would make charts creation more flexible.

[tannerwelsh]

Thanks @arnaudbesnier!