openframeworks
Dependency and compatibility info in addon template
I think the addon template should also have dependency and compatibility info included in some file (xml?) or other automatedly processable form (special strings in the README?). This was discussed a couple of times already, but the discussion never came to a conclusion. Maybe now is a good time to do this.
What info should be included? How should it be included? How do we allow/account for future features like automated addon download by PG (openframeworks/openFrameworks#1233)?
@obviousjim @atduskgreg @ofTheo @ofZach @arturoc: Thoughts?
there was some talk about using homebrew for this, but I kept backing away because it was such a beast. We don't really need configuration/installation -- we just need files to get downloaded to the right place.
I propose something like this:
the readme can have key/value pairs that ofxaddons will recognize and add tags to.
dependency: https://github.com/YCAMInterlab/ofxTimeline dependency: https://github.com/obviousjim/ofxGameCamera ...
Then a download script will go fetch the rest of the required addons. There may be a problem when 2 addons require different forks of the same addon... but that is probably an edge case
Maybe you can also specify target platforms platform: iOS platform: OS X platform: Linux
This developers can state what they are going for.
What do you think @atduskgreg?
On Tue, Jul 24, 2012 at 12:45 AM, Christoph Buchner < [email protected]
wrote:
I think the addon template should also have dependency and compatibility info included in some file (xml?) or other automatedly processable form (special strings in the README?). This was discussed a couple of times already, but the discussion never came to a conclusion. Maybe now is a good time to do this.
What info should be included? How should it be included? How do we allow/account for future features like automated addon download by PG (openframeworks/openFrameworks#1233)?
@obviousjim @atduskgreg @ofTheo @ofZach @arturoc: Thoughts?
Reply to this email directly or view it on GitHub: https://github.com/benben/ofxAddonTemplate/issues/4
- James
I want to investigate some file format that replaces the current libsorder.make +
- different search paths than the usual ones
- additional libraries ( in the system, probably only useful for linux)
- dependencies with other addons
- system packages that need to be installed ( linux only )
- additional code to be included not in src
and probably others, something similar to the current install.xml but optional and plain text so it can be parsed easily from Makefiles. I want to have this for the next release.
this file can be parsed by ofxaddons to look for dependencies and also by the PG or any other tool for downloading online addons with their dependencies
El 23/07/12 22:54, James George escribió:
there was some talk about using homebrew for this, but I kept backing away because it was such a beast. We don't really need configuration/installation -- we just need files to get downloaded to the right place.
I propose something like this:
the readme can have key/value pairs that ofxaddons will recognize and add tags to.
dependency: https://github.com/YCAMInterlab/ofxTimeline dependency: https://github.com/obviousjim/ofxGameCamera ...
Then a download script will go fetch the rest of the required addons. There may be a problem when 2 addons require different forks of the same addon... but that is probably an edge case
Maybe you can also specify target platforms platform: iOS platform: OS X platform: Linux
This developers can state what they are going for.
What do you think @atduskgreg?
On Tue, Jul 24, 2012 at 12:45 AM, Christoph Buchner < [email protected]
wrote:
I think the addon template should also have dependency and compatibility info included in some file (xml?) or other automatedly processable form (special strings in the README?). This was discussed a couple of times already, but the discussion never came to a conclusion. Maybe now is a good time to do this.
What info should be included? How should it be included? How do we allow/account for future features like automated addon download by PG (openframeworks/openFrameworks#1233)?
@obviousjim @atduskgreg @ofTheo @ofZach @arturoc: Thoughts?
Reply to this email directly or view it on GitHub: https://github.com/benben/ofxAddonTemplate/issues/4
The advantage I see with @obviousjim's approach is that this info being in/parsed from the README, we won't have to duplicate it somewhere else, so that it is readily available for both humans and computers. Also, I think it would be easier for addon maintainers to add this to a README than to construct an xml/json/whatever file, and we want to make it as easy as possible, right? or would your file also be just key:value pairs, arturo? What tag set would you propose, @obviousjim? dependencies, target platforms, anything else? I suggest also including OF version compatibility/"testedness" - this could tie in nicely with the ofxaddons "freshness" indicator you are planning.
Arturo's approach would be much more powerful of course, in that we can also store more complicated, "internal", info.
@arturoc: by next release, you mean 0073, right? 'cause 0072 is way late, already... also @arturoc, what's the current status of the install.xml? Do we use it? Will we continue to use it? Where can I find info regarding the expected structure/content (to include in the template)?
The beauty of ofxaddons is, when addons get published/updated (btw, do you crawl all the addons regularly?), we can check if the README or Arturo's file are "valid"/parse correctly, and if not, automatically open an issue via the GH API! Maybe this could later even be extended to testing compliance with the addon template structure...
I would recommend to look on other excellent dependency tools like bundler (ruby) or npm (node.js) which do dependency management exactly the same way we want to do.
npm example written for an addon:
{
"name": "ofxMyCoolAddon",
"description": "a very cool addon which can do a lot of cool things",
"author": "Cool addon author <[email protected]>",
"dependencies": {
"ofxFenster": ">= 1.2.0", // source can be omitted, because it is the same on ofxaddons
"ofxSpecialAddon": "http://some/special/source/path" // can be local, git or http path
},
"openFrameworks": "0071",
"tags": ["working on linux", "experimental", "needs fix on windows"]
}
I think the node management is a little bit smarter, because its pure json and we don't need any further interpreter.
/edit/
If we say every addon needs a package.json or something like that, we can even use it to add tags, we talked about on the ml like "working on linux", "experimental", etc. (I added this to the example above)
ofxaddons app can then easily parse this file or file a issue if an addon doesn't have one or send an automated pull request with a template of this file :)
/edit2/
another advantage on relying on some standardized format is the fact that we are open to further enhancements in some directions because we can use already existing parsers for the dependency management in every possible language.
Where would this file live?
On Tue, Jul 24, 2012 at 4:41 PM, Benjamin Knofe < [email protected]
wrote:
I would recommend to look on other excellent dependency tools like bundler (ruby) or npm (node.js) which do dependency management exactly the same way we want to do.
npm example written for an addon:
{ "name": "ofxMyCoolAddon", "description": "a very cool addon which can do a lot of cool things", "author": "Cool addon author <[email protected]>", "dependencies": { "ofxFenster": ">= 1.2.0", # source can be omitted, because it is the same on ofxaddons "ofxSpecialAddon": "http://some/special/source/path" # can be local, git or http path }, "openFrameworks": "0071" }I think the node management is a little bit smarter, because its pure json and we don't need any further interpreter.
Reply to this email directly or view it on GitHub: https://github.com/benben/ofxAddonTemplate/issues/4#issuecomment-7200670
- James
I don't really like it honestly. I know it's nice to leverage existing technology - but it feels like overkill.
the regular expression to parse something that is on one line and contains one ":" is certainly not a bad wheel to re invent
and the human readable factor + we already have README parsing in ofxAddons is a +2 to keeping it simple and straight forward.
that's my 2円 neway
On Tue, Jul 24, 2012 at 5:45 PM, Christoph Buchner < [email protected]
wrote:
I'd say in the addon root directory.
Reply to this email directly or view it on GitHub: https://github.com/benben/ofxAddonTemplate/issues/4#issuecomment-7201718
- James
+1 for root directory.
@obviousjim Beside how the file is formatted, I think its no overkill to add more (standardized) information to it, because there is need for more meta-information and personally I don't like the proposal for adding marker like snippets cluttering the names/descriptions/issues on github (like the proposal mentioned on the ML). I think it makes more sense to put all this in a file. Also I think if there will come other features, we could add it to the file. Do you agree on storing everything meta in a file and you just don't like the formatting?
I think what James is driving at is that it should be sufficient to keep it in the README, with a simple : as a marker, instead of some elaborately structured file.
This would have the advantage of also being immediately readable and available to the user at the first point-of-entry, i.e. the README. Additionally, keeping info solely in the README would avoid duplication (and maintenance mistakes) of e.g. having the version number or dependencies in several files. So in that regard I tend to agree.
However, the simple scheme already gets more complicated if we get more complicated info (in the future), e.g. multiple dependencies as already above, or multiple attributes for one thing (dependency-version, -url(optional), -name, -minVersion,...), and we then might as well stick to an already well-established and proven syntax.
Maybe a middle ground is to have a metadata section at the end of the README, with something similar to what ben posted above in it?
*** "I want to investigate some file format that replaces the current libsorder.make +
- different search paths than the usual ones
- additional libraries ( in the system, probably only useful for linux)" ***
@arturoc isn't that what install.xml is? ;) why not just expand that?
how do we mark the seperation of "normal readme" and "meta data in readme". do we agree on a special marker or something like that? From my side I'm fine with adding it to the end of the README but I'm currently unshure how we make this reliable to seperate from normal readme content.
could be: "line starts with one of a list of predefined keywords, then :", or something like that?
ok I put everything in a gist to discuss further: https://gist.github.com/3197172 So everyone can see MD and raw data and can fork/edit easily. I had to put an empty line between every entry, that MD renders it right.
What I not included for now is @arturoc data, because I don't know how to include it. Especially needed packages on linux, because they are named differently on every platform. I think we need more discussion on this. Maybe @arturoc can add the info as he like and we can see if everyone is happy with it.
What I forgot: The syntax is now a valid YAML syntax, which a) renders good in MD and is b) easily parsable.
hm, that meta.md file looks like a good approach. I think I could use that also to store the meta information in https://github.com/bilderbuchi/ofStateManager, although I used json there. I'm not set on anything, though.
quickly, since this was discussed on the last OF dev chat, I thought I'd mention that I proposed we do the following WRT to the project generator and addons:
a) it searches for some sort of file (install.xml or other json/yaml/ etc file in the root directory). b) if it's doesn't find that, it parses the addon using the file structure system it uses now and generates that file c) that file contains all the info basically that ofxAddons parses now, but could contain more information such as, files for the bin folder (some addons want to put resources in bin), addition link info, frameworks, etc.
I think this is kind of the best of both worlds, allows us to keep the simplicity that exists with alot of addons now, but also helps us solve some of the trickier things the PG is hitting walls with such as link order, additional libraries, etc.
I know this doesn't address dependency or capability info but I think this could be part of the file as well.
what do you think?
we are working in refactoring the makefiles at the moment, (osx working already!) next step was look into this. my idea is more or less what @ofZach proposes but the file should be something easily parsable by the makefiles, so no json or xml. i guess some properties file with one property per line or something like that + sections for different platforms should do
I like what @ofZach proposes, with a slight alteration to b): if no such file exists, and the pg parses the structure, it should not create the file. why? because if it can successfully extract the info it needs from a (default-like) file structure, it should not put this structure into a file, too. otherwise the addon dev now has to manually handle that file if he changes stuff in the addon, we risk desyncing structure and structure-describing metadata if he does not, and all this is not necessary because the PG can understand the structure anyway. the file should only be created on demand, if the addon devs wants/needs non-standard changes (like linking order). for that we could supply a file creation command from within the PG. that way, simple and complex addons would also be neatly separated (by presence/absence of that file)
@arturoc: is there a (known) format which is easily parseable by make? the YAML file @benben posted above already looks pretty simple, would that be a workable approach?
btw, some food for thought: we'll probably need some way of dealing with platform-dependent source code, e.g. for ofxOsc, so that those adjustments become unnecessary: https://github.com/bilderbuchi/oscpack/commit/f9a2b5789cc00f87a633a21507a56bf5cfcb479f how do we best go about this? will the new makefile system be able to do that, @arturoc ?
i've just added support for this kind of files in the makefile refactoring branch, here's an example for ofxOpenCv:
https://github.com/arturoc/openFrameworks/blob/34e29c270430655c99c2c052d3bed7250813fc79/addons/ofxOpenCv/addon.make
the meta section contains meta tags for the addon, in common you can specify any flags that are common to all platforms and then there's a section per platform where you can add specifics needed only for that platform
everything is optional so if the file doesn't exist the makefile parses the file system and if any variable is empty the makefile parses the file system for that value. for example in the case of the openCv file the libraries are specified for some platforms because we need a specific order for those but everything else is empty so all the other settings are parsed from the file system
that looks great!
from the inclusion of tags, it seems you intend this to be indexable (e.g. by ofxaddons). in that case, may I propose adding ADDON_VERSION, and maybe ADDON_LICENSE tags?
also, I know that addon.make seems like the most straightforward name for that file, but I think there's very much confusion potential (on the user-side) between addon.make in the addons and addons.make in the addons' projects/examples. maybe a different name (metadata.make or addon_config.make) would be better?
On second thought, do we really want to mix metadata and build configuration in one file? Do you have a specification in mind in case we want to add more detailed (meta)data in the future (e.g. addon/OF dependencies including minimum/maximum versions, and places to get them from)?
i don't mind about the name, don't think it is that confusing since it's in different folders and people won't need to modify yhem most of the time but addon_config.make sounds good too.
about the metadata i think it is ok to have it in that file i would add version, probably license and url that way the ofxaddons the PG or any other tool can show that data pretty easily
sounds good.
url is a good idea. Also, might be good to add (optionally) compatible platforms (see @obviousjim's comment above) in case an addon doesn't run on all platforms?
could you mock-up a structure which would indicate the addon's dependencies, including location, and minimum (and maybe maximum) version, which would be parseable by our makefiles?
This would've been easier with json/yaml/xml and key-value pairs like benben showed, but maybe you already have a good way in mind which would be good for make? I think this is getting too much info for the ADDON_DEPENDENCY field in common, no? I was thinking of
ADDON_DEPENDENCY = ofxGUI>=1.5.2<3.0.0 ofxControlPanel>0.5, (we could split at the special chars) but this gets unreadable quickly...
Hi guys,
Sorry for not piping up early. I'm sitting here with James. We've been thinking about this problem for awhile from an ofxaddons perspective. We've been planning on adding an API to ofxaddons and we think it can help a lot with some of these problems.
We've come up with a way for addons to declare the other addons on which they depend, which will allow the Project Generator (or other command line tools) to communicate with ofxaddons.com to query for all of the information needed to resolve dependencies and clone them all from github.
This is focused on figuring out which addons an app or other addon depends on, not other build requirements. It's meant to replace things like this shell script: https://github.com/YCAMInterlab/ofxTimeline/blob/master/clone_addons.sh
We'd like addons and projects to declare their dependencies in a simple text file, something like "addons.txt" which would contain a series of lines like the following:
memo/ofxMSAPhysics sha=83eacde45 branch=experimental
where the format is:
(where everything inside brackets is optional.)
Our ofxaddons.com code, will look for these files in addons, parse them, and store a resolved dependency graph for each addon. When you query ofxaddons.com for information, it will return everything it knows about a particular addon including the full list of dependencies. That should be enough for the PG (or other cli tools) to successfully clone from github all the needed addons.
-- Greg and James
Hey Greg James
As i see it ofxaddons should return a list of available addons and the git urls for them but the dependencies and other info should be in the addons file so we can resolve those offline. not sure if what you propose will allow that?
take a look at this example
https://github.com/arturoc/openFrameworks/blob/f3c33d632c45930cd8ea1df2599556e78f54ffd1/addons/ofxOpenCv/addon.make
no dependencies there yet, my idea was to have just a list of addons the way the other rules are written. also having the possibility of specifying dependencies in the common area or per platform
also, i don't mind that much if the meta tags are in a separated file to the compiling rules
about the dependencies, i'm not sure that keeping version numbers or hashes is so necessary, i see how it can be useful but it'll also make maintaining this kind of files way harder. perhaps we can have the hash/branch as optional then if someone doesn't include them we default to latest in master and if the addon is downloaded in local whatever version is in the addons folder.
Hey Arturo,
Is the addons.make used for other platforms besides linux/android? I'm worried about forcing it on other platforms just for the feature of requiring dependencies.
We are thinking of this from the perspective of a user using the PG: We imagined that the project generator would be able to query and see addons from ofxaddons.com. If you select to install an addon to your project it'll also clone the the dependent addons, using ofxaddons to resolve the dependences from the addons.txt file and forcing them into the project projects.
This could be done offline if the addon has already been downloaded and the dependencies for it cached.
How much do you think the project generator should know about the addons.make?
Also the version numbers and hashes are optional. most files will look like this:
addon dependencies for ofxMSAPhysics
ofxMSAObjCPointer ofxMSACore
the repo name and sha/branch are not necessary.
addon.make will be used by the PG too and it's optional but can be used in any platform. for example in osx some addons need different search paths than the ones parsed from the file system so they can go in that file, in windows there's some addons with several libraries that need to be linked in a specific order
if someone just wants to put the meta tags in an addon they can omit everything else. we could put the meta tags in a separate file or even in the readme but at least the dependencies should go in addon.make so makefiles and the PG can parse them and include them in the project
thinking of this a little more, the way software repos work usually is by downloading a file (it could be an xml or json) with all the info about addons in the repo (name, url, author, description...), so in this case the PG could ask for such a file to ofxaddons then with that file it's able to show info for any addon, if it's already downloaded... you can update that file from time to time or it can be updated everytime you start the PG but it's faster than having to make a request to ofxaddons everytime you select an addon
~~james: yes, this addons.make is for the new makefile system which works on more platforms than linux/android (e.g. macos, not sure about windows, arturo will know more)~~ arturo ninja'd me :)
I'm not sure about making our build process reliant on ofxaddons.com. if the relevant info is already in an addon's addons.txt (which I think a more descriptive name would be dependencies.tx, btw), why do we need to have the extra step of contacting ofxaddons? We can just fetch dependent addons as per the info in addons.txt, fetch the addons named in those freshly fetched addons, etc, until we're done, right? also, what do we do if ofxaddons is offline/not up-to-date? I gather it hasn't been updated for quite some time, recently?
memo/ofxMSAPhysics sha=83eacde45 branch=experimental
I'm curious why you specify both a branch and an sha? those are both refs, one dynamic, one static, which is potentially ambiguous. what am I missing here?
thinking of this a little more, the way software repos work usually is by downloading a file (it could be an xml or json) with all the info about addons in the repo (name, url, author, description...), so in this case the PG could ask for such a file to ofxaddons then with that file it's able to show info for any addon, if it's already downloaded... you can update that file from time to time or it can be updated everytime you start the PG but it's faster than having to make a request to ofxaddons everytime you select an addon
at last, this discussion seems to have come full circle