nodejs-bigtable icon indicating copy to clipboard operation
nodejs-bigtable copied to clipboard

Read the data of one property from

Open artman opened this issue 2 years ago • 0 comments

I'm trying to stream a large amount of data for processing using:

table.createReadStream({prefix: `prefix`}).on("data", row => {
  ...
})

I'm receiving a lot of rows, but each of them contains only small amounts of pre-serialized data. I'm having a hard time trying to figure out how to receive only the data and not the full row object. What I receive for each data callback is an object representation of the row, containing Bigtable, Table and Instance metadata:

 Row {
   bigtable: Bigtable {
     customEndpoint: undefined,
     options: {
       BigtableClient: [Object],
       BigtableInstanceAdminClient: [Object],
       BigtableTableAdminClient: [Object]
     },
     api: { BigtableClient: [BigtableClient] },
     auth: GoogleAuth {
       ...AbortSignal
     },
     projectId: '...',
     appProfileId: undefined,
     projectName: 'projects/{{projectId}}',
     shouldReplaceProjectIdToken: true
   },
   table: Table {
     bigtable: Bigtable {
       customEndpoint: undefined,
       options: [Object],
       api: [Object],
       auth: [GoogleAuth],
       projectId: '...',
       appProfileId: undefined,
       projectName: 'projects/{{projectId}}',
       shouldReplaceProjectIdToken: true
     },
     instance: Instance {
       bigtable: [Bigtable],
       id: '...',
       name: '....'
     },
     name: '...',
     id: '...'
   },
   id: '...',
   data: { entity: { properties: [Array] } }
}

Even if this information is not transmitted over the network for every request, it seems wasteful to construct an object for every row when I'm only interested in the data. Any way to omit this metadata and just stream the the data I'm interested in?

artman avatar Sep 05 '22 14:09 artman