jonahkagan
Array length hint
It would be useful to have a hint that controlled the length of a randomly generated array. Now it appears to randomly choose between 1 and 5 elements to place in the array, but some situations call for long lists, like 1000 or more. Terrific utility, though, and .... fun to use!
That's a good idea - it think it would be good to use the minItems and maxItems attributes for this (http://json-schema.org/latest/json-schema-validation.html#anchor42). Would you like to try implementing it?
OK, so I took the bait. I did not have node on my OS X machine nor coffee so I started by getting that all set up. While drinking coffee, in fact.
Downloaded the ipsum source and ran it and "express" was not found. Fortunately, I recalled seeing something about npm and realized it was probably very gem like (I am no expert at ruby either). So I did 'nom install' and it appeared to download a healthy collection of things. Reran node bin/server.js and got this:
Buzzs-MacBook-Pro:schematic-ipsum buzz$ node bin/server.js undefined /Users/buzz/tmp/schematic-ipsum/bin/../public Schematic Ipsum listening on port 3000
I can curl http://localhost:3000 -d @f1.json where f1.json is the sample schema on the online site but it returns Bad type: "undefined" and the server output says
Bad type: "undefined" POST / 400 80ms - 21
In fact, ANY content I POST to the server yields the same undefined error.
Any quick tips on debugging this? I am happy to try to add logic to the app and learn coffee and node and all but starting from a running app is a huge leg up. I am less of a framework person and more of an info arch / solutions design person (as the rants on moschetti.org might suggest).
On Nov 29, 2013, at 3:08 AM, Jonah Kagan [email protected] wrote:
That's a good idea - it think it would be good to use the minItems and maxItems attributes for this (http://json-schema.org/latest/json-schema-validation.html#anchor42). Would you like to try implementing it?
— Reply to this email directly or view it on GitHub.
So I did this: app.post("/", function(req, res) { return async.waterfall([ function(done) { console.log("inbound body: " + req.body); console.log( JSON.stringify(req.body, null, 4) );
My client side curl is this:
curl -d '{ "type": "string"}' http://localhost:3000
And this is what gets printed:
inbound body: [object Object] { "{ "type": "string"}": "" }
Yow! The entire input is being parsed into the key and value is blank. There must be some sort of pilot error on my part. Any clues? BTW, I know where the maxItems minItems logic should go and am excited to give that a crack.
On Nov 29, 2013, at 12:50 PM, Buzz Moschetti [email protected] wrote:
OK, so I took the bait. I did not have node on my OS X machine nor coffee so I started by getting that all set up. While drinking coffee, in fact.
Downloaded the ipsum source and ran it and "express" was not found. Fortunately, I recalled seeing something about npm and realized it was probably very gem like (I am no expert at ruby either). So I did 'nom install' and it appeared to download a healthy collection of things. Reran node bin/server.js and got this:
Buzzs-MacBook-Pro:schematic-ipsum buzz$ node bin/server.js undefined /Users/buzz/tmp/schematic-ipsum/bin/../public Schematic Ipsum listening on port 3000
I can curl http://localhost:3000 -d @f1.json where f1.json is the sample schema on the online site but it returns Bad type: "undefined" and the server output says
Bad type: "undefined" POST / 400 80ms - 21
In fact, ANY content I POST to the server yields the same undefined error.
Any quick tips on debugging this? I am happy to try to add logic to the app and learn coffee and node and all but starting from a running app is a huge leg up. I am less of a framework person and more of an info arch / solutions design person (as the rants on moschetti.org might suggest).
On Nov 29, 2013, at 3:08 AM, Jonah Kagan [email protected] wrote:
That's a good idea - it think it would be good to use the minItems and maxItems attributes for this (http://json-schema.org/latest/json-schema-validation.html#anchor42). Would you like to try implementing it?
— Reply to this email directly or view it on GitHub.
It looks like curl -d sends form encoded data using the syntax key=val. If you want to just send a JSON string, you have to add a header like so: curl localhost:3000 -H "Content-Type: application/json" -d '{"type":"string"}'.
Hopefully that will get you on the right track! Sorry for not including instructions about npm install - I've added them now. Also, you can use ./run.sh to run a development server.
It might be useful to add some tests for min/maxItems before you start - then you can just test your implementation using make test instead of curl.
Success! Thanks. OK, now onto making the mods. Ageed with adding the tests first. Test-driven development is choice.
On Nov 29, 2013, at 2:49 PM, Jonah Kagan [email protected] wrote:
"Content-Type: application/json"
In my excitement I put the min/max array aside and change the MAX_ITEMS from 50 100000. The schema file from the online site gets up to n=81 OK. With n=82, something goes bad here (I put in a null check and a hardcode for crutch at the moment):
readFile = function(files, done) { return fs.readFile(_.randomFrom(files), "utf8", function(err, contents) { if(contents == null) { console.log("?!? contents null reading files: " + err); return done(err, [ "Buzz", "M" ]); } else { return done(err, contents.split("\n")); } }); };
The err is #define EMFILE 24 /* Too many open files */
So I am guessing that those data files are open but not closed. Will take a look at that, too.
On Nov 29, 2013, at 4:15 PM, Buzz Moschetti [email protected] wrote:
Success! Thanks. OK, now onto making the mods. Ageed with adding the tests first. Test-driven development is choice.
On Nov 29, 2013, at 2:49 PM, Jonah Kagan [email protected] wrote:
"Content-Type: application/json"
These are the mods I have in the making, now that I learned coffeescript, installed node.js, etc. etc.
- Increase MAX_ITEMS from 50 to 10000000. I need to generate some BIG lists....
- Add minItems/maxItems optional hints to array type to permit varying size arrays (instead of just between 1 and 5)
- Fix existing array logic so that one and only one entry could be created with minItems = 1 and maxItems = 1
- (Brace yourself) Change the readFile function to use fs.readFileSync. The async version for large values of n on the command line would exhaust the file descriptor supply. Initial tests show essentially no change in performance, especially since the same process in the OS is going after the same file and it's probably all in OS cache......
- Add an ipsum hint called counter that auto increments an integer. Very useful for generate large sequences of messages and other ordinal things.
Folks: Due to my struggling with coffeescript and out-of-memory errors stemming from the creation of a big list that gets stringified at the very end, I have abandoned making changes to this codebase and reimplemented it in python. And Java, too, actually. Thanks to Jonah for providing a good path to follow.