Why can't we stream upload to cloudflare workers?

[Gozala]

[yocontra]

@Gozala You might find this useful - this is working for me fine using node, just using superagent to stream a file into IPFS. I think the 100mb limit is there for the workers still (your other issue) but for simply avoiding OOM on the node end it works.

import { NFTStorage, Blob } from 'nft.storage'
import request from 'superagent'
import { URL } from 'url'
import { pipeline, PipelineSource } from 'stream'

const client = new NFTStorage({ token: config.IPFS_STORAGE_KEY })

const storeStream = async (stream: PipelineSource<any>): Promise<string> => {
  const { endpoint, token } = client
  const url = new URL('/upload', endpoint).toString()

  return new Promise((resolve, reject) => {
    const req = request.post(url).set(NFTStorage.auth(token))
    let res: Record<any, any>
    req.once('response', (rez) => res = rez.body)
    pipeline(stream, req, (err: Error) => {
      if (err) return reject(err)
      if (!res?.value?.cid) {
        return reject(new Error(`Invalid response: ${JSON.stringify(res)}`))
      }
      return resolve(res.value.cid)
    })
  })
}

[dchoi27]

@Gozala can we close this?

[Gozala]

@dchoi27 I think this is your call, but this was a meta issue to try and work out if there is the way we could overcome CF 100mb limit. @hugomrdias had multiple calls with them and if I recall correctly they seemed to think we should not have this issue & where going to look into that. I'm not sure what is the status there.

On the other hand if we end up with sessions IDs and rootless CAR uploads we would probably not care about this any more.

[LinmyChi-you]

I've been stuck here for a long time. I want to know what's the reason and how to solve it. I've tried it many times and it still looks like this. Does anyone know how to solve it? My nodejs18.10 and nftup1.02.

[elizabeth-griffiths]

We're closing this issue. If you still need help please open a new issue.