Add support for data diffing

[rbren]

Should be able to detect new/changed items automatically by providing a local file or a database where changes can be tracked.

datafire.monitor('issues', {
  do: github.get('/user/issues'),
  check: item, cb => {
    cb(null, require('issues.json').indexOf(item.id) === -1) 
  },
  stash: item, cb => {
    let issues = require('issues.json')
    issues.push(item.id)
    fs.writeFile('issues.json', JSON.stringify(issues), cb)
  }
})

[coderofsalvation]

shouldnt this be a separate integration? like an action-chain with an actioncalled 'onchange' which only forwards data to another action if there's a diff. btw. have you looked at this:

http://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageBus.html

[coderofsalvation]

just and idea: maybe there's a small js library with all messaging patterns (throttle, onchange etc) on npm?

[rbren]

I've been talking with another integration provider about how best to go about this. Each service will probably have a unique way of pushing new items and changes to items. E.g. one service might have support for webhooks that get called each time a new Widget is added, while another service might need to be polled every X minutes to get a list of Widgets.

The big issue is that often we'll need to maintain some kind of state to track which Widgets have been seen already. In some cases, just keeping a local file with a list of IDs might suffice, but in cases with much larger lists you might need a database to help keep track.

I'm going to open another issue for keeping state across action runs - once that's solved I think this one will be a bit easier.