galaxy

galaxy copied to clipboard

All is great, but any suggestions how to implement this use case with galaxy/galaxy-streams: Want to recursively readdir and get file contents with fs.createReadStream, pass to readable object mode stream, provide api on top for middlewares/plugins/pipes, then write with createWriteStream.

Reading directories and files - okey, but how to yield from createReadStream? Or some galaxy-ish way?

I read the tutorial and readme, but can't figure out how to do it..

var galaxy = require('galaxy');
var fs = galaxy.star(require('fs'));

function* doDir(dir, options) {
  yield (yield fs.readdir(dir)).forEachStar(10, function*(file) {
    var fullpath = dir + '/' + file, stat;
    if (options.stats) {
      stat = yield fs.stat(fullpath);
      if (stat.isFile()) {
        // streaming read `filepath`
        // piping, middlewares and etc
      } else if (options.recursive && stat.isDirectory()) {
        yield doDir(fullpath);
      }
    } else {
      // yield some like {file: filepath}
      // cuz readable/transform stream might be with objectMode.
    }
  });
}

galaxy.main(function * () {
  var future = galaxy.spin(doDir(__dirname+'/src'));
  // maybe yield future(), when want to start 
});

I'm new in galaxy approach. :)

Well, I have released another stream package called ez-streams and my intent is to deprecate galaxy-streams in favor of ez-streams. So I'd rather direct you to something that we will be supporting.

ez-streams is not specifically designed for galaxy but it is easy to use with galaxy. Here is how I would write your use case with it:

var ez = require('ez-streams');

// create ez-streams file readers and writers (assuming text).
var reader = ez.devices.file.text.reader(inputFilename);
var writer = ez.devices.file.text.writer(outFilename);

// chain a parser transform, to get a stream of objects, then a filter, to keep only the objects that you want,
// then to a formatter to obtain a stream a text, and then pipe the writer.
var f = reader.transformStar(myParser).filterStar(myFilter).transformStar(myFormatter).pipeStar(writer);
// the pipeStar call returns a function* on which you can yield, if you want to wait for the completion of the pipe operation

// the transformations
function* myParser(reader, writer) {
  // this transform is a nop but shows how you read from the reader and write to the writer:
  var val = yield reader.readStar();
  yield writer.writeStar(val);
}

function* myFilter(obj) {
  return obj.someProp == "keepMe";
}

// you can also use transforms already provided by ez-streams, for example a JSON formatter
var myFormatter = galaxy.star(ez.transforms.json.parser(), 0);

Two important gotchas about this:

• The above code assumes that the ez-streams API has been "starred" (or galaxied). I haven't done it yet because we are not using galaxy directly in our code. So the transformStar, filterStar, pipeStar functions do not exist yet. We are using streamline.js instead of galaxy but streamline.js uses galaxy under the hood, and streamline takes care of the starring / unstarring. If you are interested, I can create a galaxy-ez-streams wrapper for ez-streams (this is easy and fun to do).
• The example above assumes that you will be writing your transforms, filters or that you will be using existing ez-streams transforms (like the json parser above). In theory you should also be able to use standard node middleware in the chain but I haven't tested that scenario and there are probably a few rough edges that will need ironing.

The ez-streams package contains reader/writers for a variety of input/output devices (console, files, http client, http server, mongodb collections, mysql tables, oracle tables, etc.). And you can add your own devices (look at the existing ones for example). The package also contains the most obvious transforms like line/CSV/JSON parsers and formatters but they are a bit simplistic (CSV for example). This is still a bit experimental but we are already using it in our product and you can expect more devices/transforms in the future.

Bruno

Hmm, interesting approach, cool! And yea, i see ez-streams and read your blog posts, but to now, I can't realize how to couple them. Thanks I'll try it.

Am I right, with starring ez-streams you mean var ez = galaxy.star(require('ez-streams'));, or have something more? :)

Hm, nope. Not success with above starring, and not success with var reader/writer = galaxy.star(ez.devices ...) , separately.

It is more complex than just starring the devices. You also have to star all the methods that are installed by reader.decorate, and this is tricky because these are methods, not functions (so this must be bound correctly).

I can create the wrapper but I'm in meetings all day so I will only be able to do at the end of the day.

Also, the filter/transform functions take callbacks and these callbacks have to be "unstarred" by the wrapper. So this is tricky. Not a lot of code to write but tricky.

Okey, I'll keep playing and waiting. Mention me when you ready with solution. :)

@tunnckoCore Hi, I have completed a first pass on the ez-streams API improvements for galaxy. This is described here: https://github.com/Sage/ez-streams#galaxy-support

Important: I have not published yet to NPM and I had to make changes to both packages for this. So if you want to test this, do not get galaxy and ez-streams from NPM. Instead, get the latest code of both repos from GitHub. I will publish to NPM soon but I need to run more tests before doing it.