basecamp3 api pagination

[michaelpalumbo]

Hi There,

When requesting data from my basecamp3 account, I receive only 15 entries per JSON response. I understand this is due to basecamp's pagination rule. However, how do I ensure I receive all 'pages' from BC when using the @datafire/basecamp nodejs package?

[rbren]

Hi there,

Thanks for the report! Looks like we missed the page parameter for Basecamp's list operations. It was just added in v2.0.1. If you update, you should be able to set page=2, page=3 etc until you get an empty array back.

Hope that helps!

[michaelpalumbo]

awesome, thanks so much @bobby-brennan . btw love how active you all are here :D

[michaelpalumbo]

oops, sorry, where would one set the page value in one's script?

[michaelpalumbo]

@bobby-brennan so it looks like some of the calls do not allow for pagination yet? (or perhaps i'm doing it incorrectly:

basecamp.projects.recordings.json.get({
  "type":"Comment",
  "page":1
   }).then(allComments => {
   console.log(allComments)
})

yields:

(node:38731) UnhandledPromiseRejectionWarning: Error: data should NOT have additional properties
    at module.exports.Action.run (/Users/mp/bcViz/node_modules/datafire/src/lib/action.js:91:17)
    at Object.obj.(anonymous function) [as get] (/Users/mp/bcViz/node_modules/datafire/src/lib/integration.js:158:23)
    at Object.toObj.(anonymous function) [as get] (/Users/mp/bcViz/node_modules/datafire/src/lib/integration-instance.js:23:32)
    at Object.<anonymous> (/Users/mp/bcViz/bc3.js:81:35)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:744:10)
(node:38731) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:38731) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[rbren]

No problem!

You could write, e.g.

basecamp.buckets.bucketId.client.approvals.json.get({
  bucketId: "abcd",
  page: 2,
}, context)

If you're using the latest version of Node, you can easily use async/await to accomplish what you need:

async function getApprovals() {
  let approvals = [];
  let newApprovals = null;
  let page = 1;
  while (!newApprovals || newApprovals.length) {
    newApprovals = await basecamp.buckets.bucketId.client.approvals.json.get({page: page++})
    approvals = approvals.concat(newApprovals);
  }
  return approvals;
}

[rbren]

Whoops, looks like we missed recordings.json.get - one sec...

[michaelpalumbo]

ah ok!

[rbren]

OK - should be ready in @datafire/[email protected]. Let me know if you spot any more issues!

[michaelpalumbo]

ok, so I get this error again:

(node:39012) UnhandledPromiseRejectionWarning: Error: data should NOT have additional properties
    at module.exports.Action.run (/Users/mp/bcViz/node_modules/datafire/src/lib/action.js:91:17)
    at Object.obj.(anonymous function) [as get] (/Users/mp/bcViz/node_modules/datafire/src/lib/integration.js:158:23)
    at Object.toObj.(anonymous function) [as get] (/Users/mp/bcViz/node_modules/datafire/src/lib/integration-instance.js:23:32)
    at getApprovals (/Users/mp/bcViz/bc3.js:123:74)
    at Object.<anonymous> (/Users/mp/bcViz/bc3.js:129:13)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
(node:39012) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
(node:39012) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

this error is referring to this line:

    newApprovals = await basecamp.buckets.bucketId.client.approvals.json.get({page: page++})

[michaelpalumbo]

@bobby-brennan it appears to be the same issue, is there something wrong with how the page number is being requested?

[michaelpalumbo]

@bobby-brennan mind if we reopen this issue since this isn't resolved :)

[rbren]

Using your code, I get the error "Error: data should have required property 'bucketId'". Adding bucketId gets me to a 404 error (which I expect since I don't have a basecamp account set).

Are you sure you're on 2.0.2?

[michaelpalumbo]

Yep, I'm using 2.0.2!

[rbren]

Can you share the full script?

What do you see if you run datafire describe basecamp/buckets.bucketId.client.approvals.json.get?

[michaelpalumbo]

Hi @bobby-brennan sorry for taking so long to get back to you, thanks again for all your help. Ok, so here's what I get in response to shell datafire describe basecamp/buckets.bucketId.client.approvals.json.get

buckets.bucketId.client.approvals.json.get

Input
object
  bucketId*:    string

Output
array[object]
  status:       string
  app_url:      string
  approval_status: string
  parent:       object
    url:          string
    app_url:      string
    type:         string
    id:           integer
    title:        string
  title:        string
  url:          string
  replies_url:  string
  created_at:   string
  approver:     object
    attachable_sgid: string
    bio:          null
    name:         string
    title:        string
    admin:        boolean
    created_at:   string
    updated_at:   string
    time_zone:    string
    company:      object
      id:           integer
      name:         string
    avatar_url:   string
    personable_type: string
    owner:        boolean
    email_address: string
    id:           integer
  bucket:       object
    type:         string
    id:           integer
    name:         string
  updated_at:   string
  id:           integer
  content:      string
  creator:      object
    attachable_sgid: string
    bio:          string
    name:         string
    title:        string
    admin:        boolean
    created_at:   string
    updated_at:   string
    time_zone:    string
    company:      object
      id:           integer
      name:         string
    avatar_url:   string
    personable_type: string
    owner:        boolean
    email_address: string
    id:           integer
  due_on:       null
  replies_count: integer
  type:         string
  inherits_status: boolean
  bookmark_url: string
  subject:      string

As for my script:

const fs = require('fs')
const { exec, execSync, spawn, spawnSync, fork } = require('child_process')
const prepend = require('prepend-file')

// retrieve OAuth2. 
var obj =  <REDACTED>

let accessToken = <REDACTED>
let integration =  <REDACTED>
let clientID =  <REDACTED>
let clientSecret =  <REDACTED>
let refreshToken =  <REDACTED>
let redirectURI =  <REDACTED>
let accountID =  <REDACTED>


let basecamp = require('@datafire/basecamp').create({
  access_token: accessToken,
  account_id: accountID
});

// // // comments

async function getComments() {
  let comments = [];
  let newcomments = null;
  let page = 1;
  while (!newcomments || newcomments.length) {
    newcomments = await basecamp.projects.recordings.json.get({page: page++})
    comments = comments.concat(newcomments);
  }
  console.log(comments)

  return comments;

}

console.log(getComments())


async function getApprovals() {
  let approvals = [];
  let newApprovals = null;
  let page = 1;
  while (!newApprovals || newApprovals.length) {
    newApprovals = await basecamp.buckets.bucketId.client.approvals.json.get({page: page++})
    approvals = approvals.concat(newApprovals);
  }
  return approvals;
}

console.log(getApprovals())

[michaelpalumbo]

note both async functions for the pagination return the same error for me:

(node:47983) UnhandledPromiseRejectionWarning: Error: data should NOT have additional properties
    at module.exports.Action.run (/Users/mp/bcViz/node_modules/datafire/src/lib/action.js:91:17)
    at Object.obj.(anonymous function) [as get] (/Users/mp/bcViz/node_modules/datafire/src/lib/integration.js:158:23)
    at Object.toObj.(anonymous function) [as get] (/Users/mp/bcViz/node_modules/datafire/src/lib/integration-instance.js:23:32)
    at getComments (/Users/mp/bcViz/bc3.js:226:59)
    at Object.<anonymous> (/Users/mp/bcViz/bc3.js:235:13)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
(node:47983) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 5)
(node:47983) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:47983) UnhandledPromiseRejectionWarning: Error: data should NOT have additional properties
    at module.exports.Action.run (/Users/mp/bcViz/node_modules/datafire/src/lib/action.js:91:17)
    at Object.obj.(anonymous function) [as get] (/Users/mp/bcViz/node_modules/datafire/src/lib/integration.js:158:23)
    at Object.toObj.(anonymous function) [as get] (/Users/mp/bcViz/node_modules/datafire/src/lib/integration-instance.js:23:32)
    at getApprovals (/Users/mp/bcViz/bc3.js:243:74)
    at Object.<anonymous> (/Users/mp/bcViz/bc3.js:249:13)
    at Module._compile (internal/modules/cjs/loader.js:702:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
    at Module.load (internal/modules/cjs/loader.js:612:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:551:12)
    at Function.Module._load (internal/modules/cjs/loader.js:543:3)
(node:47983) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 6)

[rbren]

Hmm...looks like it's somehow it's still using 2.0.1. What do you see when you run:

cat node_modules/@datafire/basecamp/package.json | grep version

Can you try rm -rf ./node_modules and reinstalling?

[michaelpalumbo]

Ah! it actually reports 'version 2.0.0'!!!

[michaelpalumbo]

hmm, but

rm -rf ./node_modules

wouldn't this also remove other node modules in my repo? is that what I should do?

[rbren]

Yes it would - do you have them saved in package.json? You can also just run rm -rf node_modules/@datafire/basecamp.

But in general I'd suggest using the --save flag when running npm install (which will store it in package.json), so that you can always run rm -rf node_modules && npm install to get a fresh install of all your dependencies. It's like the "turn it off and on again" of nodejs :)