BOINC
Improve support for universal apps
A BOINC "universal app" is a VM player. The VM images are part of the workunit. Umbrella projects will typically have a single universal app that is used for dozens or hundreds of science applications. (Call these "science apps", and assume that they have integer IDs and version numbers).
BOINC has a number of mechanism that, in essence, equate apps and science apps, and that don't work optimally in the presence of univeral apps.
-
The BOINC mechanism for estimating job runtime uses statistics that are maintained at the level of (host, app_version). What we'd like is (host, science_app_version).
-
Same for reliability (host_app_version.consecutive_valid).
-
Job submission permissions (user_submit_app table). These control what jobs a user can submit. Currently permissions are the level of app; they could be at the level of science app. (Not a high priority since this mechanism is not currently used AFAIK.)
-
The admin web interface has tools for looking at error rate and other stats on the level of app and (app, platform). We could increase the resolution to science app.
-
The user web interface shows what app a job is associated with. It should also show the science app.
It remains to be seen how severe each of these problems is.
How to address these potential problems? Some general approaches:
-
Add new tables for science app and science app version. This would greatly increase code size and complexity. and it would require DB access when adding new science apps. I'd prefer to avoid it.
-
Generalize existing structures: Add a "universal app" flag to app Add "science app ID" and "science app version" fields to workunit, populated for universal app jobs. Also possibly "science app name". For universal apps, host_app_version.app_version_id encodes science app ID and version. This would increase code complexity but not size.
Is this something that's already implemented? It seems to be demanding another level of abstraction without providing one. The proper place to encode which app is used is in the app and app_version. It's easier to me to include the universal app (the vm player) and the science app (the vm) as part of the app_version. If all apps for a project use the same vm player, it is still only downloaded once.
Sorry hit send too soon.
The problem of work estimation is still there, but it better done per app_version than per workunit.
On Mon, Aug 28, 2017 at 12:57 PM, Eric Korpela [email protected] wrote:
Is this something that's already implemented? It seems to be demanding another level of abstraction without providing one. The proper place to encode which app is used is in the app and app_version. It's easier to me to include the universal app (the vm player) and the science app (the vm) as part of the app_version. If all apps for a project use the same vm player, it is still only downloaded once.
-- Eric Korpela [email protected] AST:7731^29u18e3
"Universal app" is as I described - it allows job submitters to run new science apps with no changes to the server DB (i.e. no new app versions). This means that the science app executable is part of the workunit, typically a Docker layer. This approach is being used by several projects.
New workunits are a no less change to the DB than new app_versions are. To me it sounds like the app_version creation scripts were merely insufficient. Were my project running in this format, I would probably not use the scheme you describe.
On Mon, Aug 28, 2017 at 4:34 PM, David Anderson [email protected] wrote:
"Universal app" is as I described - it allows job submitters to run new science apps with no changes to the server DB (i.e. no new app versions). This means that the science app executable is part of the workunit, typically a Docker layer. This approach is being used by several projects.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/BOINC/boinc/issues/2078#issuecomment-325511961, or mute the thread https://github.com/notifications/unsubscribe-auth/AKXcsvHV1x7F-Buwj_mz5kkc3m_Dq4JEks5sc06YgaJpZM4PE71i .
-- Eric Korpela [email protected] AST:7731^29u18e3
Sorry, again, this editor sucks. Will change back to standard gmail. I meant to add "but since it's already implemeted its too late for that."
On Mon, Aug 28, 2017 at 4:58 PM, Eric Korpela [email protected] wrote:
New workunits are a no less change to the DB than new app_versions are. To me it sounds like the app_version creation scripts were merely insufficient. Were my project running in this format, I would probably not use the scheme you describe.
On Mon, Aug 28, 2017 at 4:34 PM, David Anderson [email protected] wrote:
"Universal app" is as I described - it allows job submitters to run new science apps with no changes to the server DB (i.e. no new app versions). This means that the science app executable is part of the workunit, typically a Docker layer. This approach is being used by several projects.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/BOINC/boinc/issues/2078#issuecomment-325511961, or mute the thread https://github.com/notifications/unsubscribe-auth/AKXcsvHV1x7F-Buwj_mz5kkc3m_Dq4JEks5sc06YgaJpZM4PE71i .
-- Eric Korpela [email protected] AST:7731^29u18e3
-- Eric Korpela [email protected] AST:7731^29u18e3
Remote job submission interfaces let scientists create jobs without server login or direct DB access: https://boinc.berkeley.edu/trac/wiki/RemoteJobs
I'm not sure what a "Universal app" is meant to be. It seems to be another abstraction above what we now know as "app". I'm not sure I understand what is meant by "VM player" also. Here are my thoughts on the points in the opening post:
-
runtime estimation is depending on the combination of science payload (input data) and a particular app_version. I don't see how a science_app_version relates to a current app_version so I don't agree that the runtime statistics should use (host, science_app_version).
-
see above
-
RNA World uses the job submission interface and we already introduced our notion of "universal app". In our case we have have one science application (cmsearch) that we had to split up in three BOINC applications (cmsearch S, cmsearch XL, cmsearch VM) which have different runtime characteristics and because the last also uses VBox different input files. The user who is able to submit user jobs only sees the science application and our work generator selects the correct BOINC app to be used based on some tests on the input data. Currently this uses hardcoded app id's.
Comments about how to address this:
-
I prefer to add a new table for such a "universal" science app which holds the associations to the BOINC apps and other meta data. This is based on my interpretation that a "universal" app is just a container to abstract a set of BOINC applications for the user. That's also why I don't see what a science_app_version can be. Maybe an example from an existing project may help.
-
Reusing existing DB fields for different purposes and giving it different meaning based on other context is bad in my opinion and should be avoided at all costs. It is ok to increase the size of the codebase if it helps to separate things from each other.
Just to be clear: I vote -1 to the current design because of the reasons given above mainly the proposed mangling of BOINC apps with this new science apps.
