Set location of data directory created after first run

[callumprentice]

trafficstars

If I run my app via the command line, the data directory is created in the current working directory and my code works.

The use case I am working on is embedding node in another app where I have little control on the current working directory. When my app starts, the data directory is not generated and copying over a previously generated one to all the obvious places (alongside executable, alongside the main JS file etc.) doesn't work either.

Other than by editing the source, is there a way to specify the location of this directory - something in the options object perhaps?

[Pomax]

Yes and no...

Right now it's partially hardcoded to a ./data dir existing relative to where the the app is run (you could symlink so you don't need to copy it over?), mostly in the code for https://github.com/Pomax/node-flickrapi/blob/master/src/flickr-api-object.js, which is all about what methods the Flickr API supports... it should be possible to refactor the code to default to the ./data dir if no explicit other location is passed in, but I'd have to take a careful look at it.

The code for downsyncing, over at https://github.com/Pomax/node-flickrapi/blob/master/src/handlers/downsync/index.js, does have a provision to specify a custom directory as described in https://github.com/Pomax/node-flickrapi/blob/master/README.md#download-data-from-flickr, and data that was downsynced in this manner can also be loaded in from that custom location again using the loadLocally(location) call described in https://github.com/Pomax/node-flickrapi/blob/master/README.md#use-your-flickr-data-in-another-application

[sebastienbarre]

I'm in a similar situation here. Would love to be able to set ./data to an arbitrary location. Along the same lines, in the interest of performance, it would be great to be able to specify which Flickr API endpoints I'm going to use, and have node-flickrapi only fetch these methods from Flickr, and cache them for future use.

[Pomax]

That feels like a micro optimization (the space taken up by the cached API isn't all that much), but until I get around to making the data dir fully configurable, symlinks should do the trick.

[sebastienbarre]

@Pomax thanks for the quick reply. Sorry, I should have clarified.

1. the space is definitely not an issue, it's the time it takes to retrieve those 200+ methods from the Flickr API. If the tokenOnly() or authenticate() functions could take a methods option, as an array of API calls I'm planning to make (say, ['flickr.people.findByUsername', 'flickr.photos.search')]), then parseMethods in flickr-api-objects.js could filter the others out before calling flickr.reflection.getMethodInfo on each one of them.
2. I'd like to use node-flickrapi in a node CLI of mine, which implies it can be called from any directory in the user's filesystem. Each time the user do so, a new ./data directory is created, with the aforementioned delay, which is a problem at this point. I'd like to be able to set a fixed data directory when I initialize node-flickrapi (I would typically pick a subdir somewhere in the user's HOME/USERPROFILE dir).

Thanks

[Pomax]

Hm, but methods that were previously cached should already be read from disk on subsequent calls, using https://github.com/Pomax/node-flickrapi/blob/master/src/flickr-api-object.js#L88-L94... is that not what it's doing? If so, that's a bug.

Also note that node (unlike PHP) is designed to stay up, so writing that utility as an always running single instance on some port X, and then making the CLI tool an interface to that instance instead, might be an alternative worth thinking about.

That said, it'll be nice to have the data dir be configurable; I'm not sure how much time I'll have to work on this, but if you want to take a stab at finding all the references and switching them to an options.userdir or the like, I wouldn't turn that down =P

[sebastienbarre]

Yes, they are. But the first time is pretty long, since it is calling the Flickr API more than 200 times through flickr.reflection.getMethodInfo. And that first time will occur pretty much every time in my case, because I'm writing a node command-line tool. It is installed globally with npm -g and could be invoked from anywhere. A new ./data directory will be created each and every time if it's not found in the current directory the user is in (let alone the cases where the current directory might not be writable to the user).

This could be solved if:

• one could specify where the data directory should be (I would initialize your library with a fixed dir),
• one could specify which methods are going to be called, and only this subset would be introspected and cached using flickr.reflection.getMethodInfo; or better yet, only call flickr.reflection.getMethodInfo on a per-need basis, i.e. when the method itself is called by the user. Do a round-trip to get the method parameters, cache it in data, then invoke it (and re-use that cache the next time it is called); that would spread the cost of that long, first initialization. Just an idea...

[Pomax]

of those two, I'm not a fan of restricting the API methods that get retrieved, but making the dir dynamic will be worth doing.

[sebastienbarre]

Sounds great. Once I can set data, I can pre-populate it with a subset of the methods I'm planning on using, or use jhnns/rewire to monkey-patch parseMethods -- I bet that could solve my other issue.

[sebastienbarre]

BTW I would certainly write a PR for a methods parameter that would allow for a faster startup, but it's your library, and your design decision ultimately.

[Pomax]

I've been in open source long enough to not care: I am not my code. If someone has an awesome PR that solves a real problem, not just for them but for everyone, I will happily merge that in (although after a PR review of course).