tunguska-reactive-aggregate
tunguska-reactive-aggregate copied to clipboard
robfallows
Exceptions thrown in aggregate.js update() method
I recently modified a pipeline I am using with ReactiveAggregate, and although the results of the pipeline are correct, I often see exceptions being thrown from the update() method in aggregate.js.
It looks like the exception is happening on line 239 of aggregate.js in version 1.3.7 (and 1.3.6).
I see two different exceptions, depending on which object is undefined when the line is hit:
Here the sub._session.collectionViews.get(localOptions.clientCollection) is undefined, so it cannot resolve the documents object:
Exception in setTimeout callback: errorClass [Error]: Cannot read property 'documents' of undefined
at update (packages/tunguska:reactive-aggregate/aggregate.js:267:13)
at packages/tunguska:reactive-aggregate/aggregate.js:279:7
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1257:12)
at packages/meteor.js:555:25
at runWithEnvironment (packages/meteor.js:1320:24) {
path: '',
sanitizedError: errorClass [Error]: tunguska:reactive-aggregate [Error]
at errorClass.<anonymous> (packages/tunguska:reactive-aggregate/aggregate.js:11:27)
at new errorClass (packages/meteor.js:660:17)
at update (packages/tunguska:reactive-aggregate/aggregate.js:267:13)
at packages/tunguska:reactive-aggregate/aggregate.js:279:7
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1257:12)
at packages/meteor.js:555:25
at runWithEnvironment (packages/meteor.js:1320:24) {
isClientSafe: true,
error: 'Error',
reason: 'tunguska:reactive-aggregate',
details: undefined,
errorType: 'Meteor.Error'
},
errorType: 'tunguska:reactive-aggregate'
}
Here the sub._session.collectionViews.get(localOptions.clientCollection).documents.get(doc_id) object is undefined when it tries to read dataByKey object.
Exception in changedobserveChanges callback: errorClass [Error]: Cannot read property 'dataByKey' of undefined
at update (packages/tunguska:reactive-aggregate/aggregate.js:267:13)
at debounce (packages/tunguska:reactive-aggregate/aggregate.js:286:7)
at changed (packages/tunguska:reactive-aggregate/aggregate.js:308:9)
at runWithEnvironment (packages/meteor.js:1320:24)
at packages/meteor.js:1333:14
at packages/mongo/observe_multiplex.js:178:30
at Array.forEach (<anonymous>)
at Function._.each._.forEach (packages/underscore.js:139:11)
at Object.task (packages/mongo/observe_multiplex.js:172:9)
at Meteor._SynchronousQueue.SQp._run (packages/meteor.js:922:16)
at packages/meteor.js:899:12 {
path: '',
sanitizedError: errorClass [Error]: tunguska:reactive-aggregate [Error]
at errorClass.<anonymous> (packages/tunguska:reactive-aggregate/aggregate.js:11:27)
at new errorClass (packages/meteor.js:660:17)
at update (packages/tunguska:reactive-aggregate/aggregate.js:267:13)
at debounce (packages/tunguska:reactive-aggregate/aggregate.js:286:7)
at changed (packages/tunguska:reactive-aggregate/aggregate.js:308:9)
at runWithEnvironment (packages/meteor.js:1320:24)
at packages/meteor.js:1333:14
at packages/mongo/observe_multiplex.js:178:30
at Array.forEach (<anonymous>)
at Function._.each._.forEach (packages/underscore.js:139:11)
at Object.task (packages/mongo/observe_multiplex.js:172:9)
at Meteor._SynchronousQueue.SQp._run (packages/meteor.js:922:16)
at packages/meteor.js:899:12 {
isClientSafe: true,
error: 'Error',
reason: 'tunguska:reactive-aggregate',
details: undefined,
errorType: 'Meteor.Error'
},
errorType: 'tunguska:reactive-aggregate'
}
The busier the server is the more often I see these errors, and I see the dataByKey exception far more than the documents exception.
I recently upgraded meteor from 2.5.1 to 2.5.6, and upgraded the meteor packages I am using, which brought in 1.3.7 of tunguska:reactive-aggregate. Since the line where the undefineds are happening was touched by that release, I tried downgrading to 1.3.6, but I still get the exceptions.
This is the latest code I am using to define the pipeline and use it in ReactiveAggregate:
Meteor.publish('property_stats', function (propertyId) {
// This is the pipeline used to aggregate the propertyStats
const propertyStatsPipeline = [
{
$match: {
propertyId,
type: 'Thermostat',
},
},
{
// Get the device offline alerts (if any) for each deviceId
$lookup: {
from: 'alerts',
let: { deviceId: '$deviceId' },
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: ['$device.id', '$$deviceId'] },
{ $eq: ['$type.id', 'DEVICE_OFFLINE'] },
],
},
},
},
],
as: 'deviceAlert',
},
},
{
$match: {
// keep only the devices with no deviceAlert
deviceAlert: { $size: 0 },
},
},
{
$group: {
// Not entirely clear, but for ReactiveAggregate the group _id cannot be specified
// as null to indicate grouping over all incoming documents. Instead use a unique id string,
// which here is convenient to use the propertyId.
_id: propertyId,
maxTemp: { $max: '$temperature.value' },
minTemp: { $min: '$temperature.value' },
avgTemp: { $avg: '$temperature.value' },
maxRh: { $max: '$humidity.value' },
minRh: { $min: '$humidity.value' },
avgRh: { $avg: '$humidity.value' },
},
},
];
// reactive aggregate
ReactiveAggregate(this, Devices, propertyStatsPipeline, {
clientCollection: 'propertyStats',
noAutomaticObserver: true,
debounceCount: 20,
debounceDelay: 5000,
observers: [
Devices.find({ propertyId, type: 'Thermostat' }),
Alerts.find({ 'property.id': propertyId, 'device.type': 'Thermostat' }),
],
warnings: false,
// debug: true,
// capturePipeline(docs) {
// console.log(docs);
// },
});
});
I do have other pipelines as well, but this is the one that changed when the exceptions started happening. The change was in the $lookup stage, where the conventional localField/foreignField usage was replaced with its own pipeline to limit what documents in the alerts collection were considered.
Any ideas what I am doing wrong?
I'm sure you've already tried this, but given
The change was in the $lookup stage, where the conventional localField/foreignField usage was replaced with its own pipeline to limit what documents in the alerts collection were considered.
Have you reverted this and retested?
Ah, good suggestion. I assumed it was the last thing I changed in that code that was causing the problem, but I reverted to the previous version and it still happens. And yes I did change other things, since I upgraded to meteor 2.5.6 and upgraded a few packages as well.
Let me step back and attempt to methodically upgrade pieces to see if i can identify where the problem appears.
I can say that most of the time the aggregation pipeline is triggered it runs fine. But every so often it throws one of those exceptions, usually the dataByKey complaint (undefined documents.get(doc_id)).
I will report back when I can isolate it better.
I started back at our prior commit on meteor 2.5.1, added my latest pipeline code, and stepped through each upgrade to 2.5.6. Everything worked fine.
It was only when I manually upgraded the tunguska:reactive-aggregate to 1.3.7 that I began seeing the exceptions. Since that is tied to the 2.6 meteor upgrade, it was unnecessary for me to have done that forced upgrade of reactive-aggregate. For now I am leaving it at 1.3.6 while running meteor 2.5.6.
However, I had the impression from the 1.3.7 changes that it was intended to be backward compatible, but maybe that is not the case. In any event, if I had just proceeded with the package upgrades pulled in by the meteor upgrade, all would have been well.
Probably this can be closed
Hmm. I'm not convinced this should be closed. 1.3.7 should indeed be backwards compatible.
@smohanty92 - would you please take a look at this issue and see if it was introduced by your changes? Thank you 🙂
Wanted to give an update on this issue.
We upgraded to Meteor 2.6 today and experienced the same issue with 1.3.7. I can run fine with 1.3.6, but again get those exceptions using 1.3.7.
I also want to mention we use an external MongoDB server running older 4.0.x mongo server, in case its relevant.
It looks like this bug was introduced by @smohanty92's changes. I will have to revert the release code unless he gets back.
Hi @robfallows - getting this error from a very simple aggregation
Meteor server restarted
I20220329-17:17:21.980(-4)? Exception from sub reportTotals id zKi9oCYYijQNzbsa3 Error: every aggregation document must have an _id
I20220329-17:17:21.981(-4)? at update (packages/tunguska:reactive-aggregate/aggregate.js:267:13)
I20220329-17:17:21.981(-4)? at ReactiveAggregate (packages/tunguska:reactive-aggregate/aggregate.js:334:3)
I20220329-17:17:21.981(-4)? at Subscription.
Simple code Meteor.publish("reportTotals", function() { ReactiveAggregate(this, Assignments, [{ $group: { '_id': null, 'adds': { $sum: 'adds' } } }], { clientCollection: "clientReport" }); });
Any suggestions would be appreciated!
- This started recently after upgrading to 2.3+
@frankwo1 I have encountered this before - for some undocumented reason, you cannot specify 'id': null in a group stage with ReactiveAggregate. Instead, use any unique identifier that makes sense for your collection
Thanks @kevpfowler - that works! Just assigned a randomly generated # to it.
Apologies for the delay getting back - I see this is basically a duplicate of #57 - which I have also failed to address 🤦
Utter madness at work at the moment, but I'll try to get both issues closed ASAP.
Thanks for your patience 😄
No Worries - I got a clue from @kevpfowler to solve issue - Thanks @robfallows for all you do for Meteor - I follow several of your published materials - been extremely helpful since I started using Meteor - you can close my issue.
Sorry! I had never gotten an email notification about this mention.
I left my last position and haven't used Meteor in a few months.
I just opened up a PR
Seems to be a simple fix, however I haven't been able to test.
@kevpfowler Is this resolving the warnings?
We're getting this every few hours now on our server since updating to meteor 2.10. Has anyone opened a fork and applied the above PR ? If not I'll do it in the morning.
So I just noticed that another PR, #79 - has been merged, which addressed this but we're running that version (1.3.13) in production. I'm going to update to 1.3.16 and see how it goes