Stream for encoding

[arestov]

Currently we can stream file and decoding it. Is is possible to implement encoding stream?

I want to do this

var cryptoStream = crypto.createHash('sha1');
cryptoStream.on('end', function(){
    console.log(cryptoStream.digest('hex'));
});

var bencStream = new benc.EncodeStream({prop: []});
bencStream.on('data', function(chunk){
    cryptoStream.update(chunk);
});
bencStream.on('end', function(){
    cryptoStream.update(null);
});

[a2800276]

Not sure if I understand your question, but yes the library can encode bencoding. Could you describe what you want to do instead of just posting code?

[feross]

I think he's asking for the ability to do streaming decoding as the bits arrive on the wire.

[a2800276]

The text & sample are pretty clear about encoding not decoding...

Not sure why you'd need a stream interface to encode a Javascript Object to beencoding...? Not sure to what extent this would even be possible.

I have the feeling he wants to stream a datafile somewhere, and as it's being streamed, magically create the beencoded meta-data?

Streaming decoding of deencoded dictonionaries works with the Stream object somebody implemented (I think you, even :).

Or maybe I'm just being obtuse and am missing something entirely obvious...

[arestov]

For example I want to create torrent file, from list of files. The list is big: 2000 files.

To create torrent I must create infoDictionary. (using benc.encode) Then calculate infoHash (sha1).

Both of this operations are heavy, so them can freeze application. So I want to be able to make this operations asynchronously.

The real reason is that https://github.com/mafintosh/torrent-stream completely revalidates torrent infoDictionary before use the torrent. torrent-stream revalidates it by calculating benc.encode and sha1 from scratch. https://github.com/mafintosh/torrent-stream/blob/master/encode-metadata.js

My application automatically search torrents which it uses. And I found torrent which is 2.5 MB, so application freezed for few minutes. I think I will filter out such torrents and I doubt about reasonableness of the revalidation, but anyway asynchronous encoding may be usefull.

[a2800276]

I think I'm still not entirely clear what the problem is. Currently, the encode function takes the infoDictionary to be encoded. I can't really think a a scenario that the infoDict to encode is so huge that it would take more than a negligible amount of time.

As I see it, the bottleneck is probably calculating the sha hash. You could try to pass the Buffer containing the encoded infodict to sha in pieces. I guess you separate the infoDict into pieces and call the various encode methods inside of BEncode directly, but that doesn't really seem viable, at the least, it will require reworking the internals substantially.

If you can post the dict you're trying to encode I can try to reproduce any performance issues you're having.

Finally, from my point of view, it's unecessary to decode and reencode the dict to get the sha1 hash, you could just run the stream through the hash while you're receiving it.