getavalon
Ftrack integration
Goal
Implement a system which would enable connection between ftrack and avalon.
Motivation
Ftrack has been struggling for quite some time with writing good usable pipeline tools. Even though they've implemented pyblish in the latest publishing tools, the development is taking very long time, and it doesn't seem to be looking at the pipeline in all it's complexity as avalon is. However joining avalon's data management and ftrack production tracking capabilities would I believe create extremely strong tool. The same goes for shotgun where it would provide alternative to their very robust pipeline toolkit, but that's for another issue.
Implementation
To make this happen there are certainly many smaller and bigger tasks to be done, but we can start break them down here and slowly make issues for each of them as we go along. This is where I would start
- define what should be the relationship between avalon and ftrack database (what data needs to live in both, or in just one of them)
- map terms between the two systems, to make sure we're not mixing up vocabulary when talking about the two
- get launcher/environment setup API up an running to be able to use ftrack actions instead of the avalon default launcher
- create ftrack action that mirrors objects into avalon database. This can eventually be automated with ftrack event script that listens to new objects being created in ftrack and creates them in avalon too.
This would be sufficient for a start. It's fairly one way (ftrack to avalon), but that's more than enough to get the ball rolling.
The first major question is. Are there any relevant concepts in place already that need to be evaluated and considered, or are we looking the such setup from scratch?
Marking it epic as it's something we should split into smaller tasks and tackle each individually.
Nice issue :) Very interesting in this!
get launcher/environment setup API up an running to be able to use ftrack actions instead of the avalon default launcher
This is really comes down to how much we wanna rely on Ftrack for the end-user experience. In theory we could keep everything in Avalon as it is, and then just pull data base entities from Ftrack.
If we go with using Ftrack as the front-end, initially I could see if working by having a application launch event plugin to modify the default Ftrack actions with Avalon environment variables. BUT I think people will be confused by the addition of the other tools Ftrack provides, like the "Asset Importer" and "Asset Manager".
The coarse division between Avalon and ftrack could be: let ftrack handle anything related to schedule and Avalon everything related to files.
There is some overlap, but not much. There rarely if ever needs to be any direct interop between the two I think, apart from sending things during publishing and reading things during application start. This then applies Shotgun and friends too.
people will be confused by the addition of the other tools Ftrack provides,
I'd personally bypass them altogether. For the last 3 years we've been using ftrack-connect as a glorified launcher. Haven't really used a single feature inside as we just found them inadequate.
I could see if working by having a application launch event plugin to modify the default Ftrack actions with Avalon environment variables.
I'd use their current system of application store and adding environments upon launching, but would try to skip their plugins. Figuring out the way to use it while removing dependency on ftrack-connect would be lovely.
theory we could keep everything in Avalon as it is, and then just pull data base entities from Ftrack. That's the way I'd approach it at this point. As @mottosso said. Making quite strict line between the two.
Sequence of events as I see it for most if not all tasks.
- Launch maya (from task) from ftrack using action
- Action passes all the required information to avalon launcher API and uses that to launch the actual app (user, task context, app version, app name etc.)
- Avalon launches software as if it went from it's own launcher GUI
- Artist works. Everything is completely decoupled from ftrack at this point.
- Artist publishes avalon way to avalon database. If there are any reviewables (playblasts, images, etc) we push these to ftrack as well using pyblish-ftrack as we do now.
Eventually we could easily add integrators that push more data to ftrack upon publish. For example creating ftrack versions for actual published assets with thumbnails and components liking to data on disk (for reference) would be trivial at that point.
At the end I'd imagine extra menu item in avalon menu called 'ftrack' that would hold any extra functionality that need sync from ftrack. We could then over time take any useable pieces of ftrack code and just add them in. I can think of the crew module, task browser and such.
Figuring out the way to use it while removing dependency on ftrack-connect would be lovely.
Maybe we can use their events system to bypass ftrack-connect, but that might not be very applicable for other production trackers.
Another potential solution would be to host the ftrack web page inside of the launcher, and use the web addresses to get the entities and data.
Maybe the action in Ftrack should just "place" the launcher in the correct context? The action would have nothing to do with any application, kinda just a "Avalon Launcher", and from within the Avalon launcher you choose your application.
Maybe we can use their events system to bypass ftrack-connect, but that might not be very applicable for other production trackers.
this will have to be different to other production trackers anyway. We're trying to solve ftrack specifically and events and actions are exactly the way to go I think.
Maybe the action in Ftrack should just "place" the launcher in the correct context?
To be perfectly honest I really don't like that. Going from one app to another where you choose to launcher another would drive me crazy. Especially considering that actions can certainly do it directly.
Another potential solution would be to host the ftrack web page inside of the launcher, and use the web addresses to get the entities and data.
Killing a bird with a tank maybe? But probably possible. My thinking was that if we keep the avalon launcher separate (just in it's vanilla shape) it would act as a good backup for when ftrack server go down occasionally. Artist couldn't access the notes and such, but could still just open launcher get to his task (because the hierarchy would have to be mirrored in the avalon db), launch maya and still be able to work.
When this happen now, we can open maya with our tools via separate launcher, but can't really get tot the shot and task context. Avalon could be a remedy for that.
Launch maya (from task) from ftrack using action
But why from ftrack? What advantage does ftrack provide here over just using the Launcher for launching software? If it's a question of familiarity or eye candy, then that's something within our control to change and improve. In the not-so-distant future, I expect we'll look at Launcher and wonder how anyone could have launched software differently, given all of what it offers.
Another potential solution would be to host the ftrack web page inside of the launcher, and use the web addresses to get the entities and data.
I'd make it more explicit.
Whenever a new task or the date or assignment changes, simply push that change to Avalon and it's database via some callback.
I'd like to keep it so that there is no strict dependency on any third-party, but rather these loosely coupled integration. An immediate benefit of that is that you could cut ftrack at any point, and/or transition to something else or roll your own, and still do work in the meantime.
But why from ftrack?
That is a good question. Think the current advantages Ftrack has over the Launcher, might be the navigation and maybe its "My Tasks" page.
Whenever a new task or the date or assignment changes, simply push that change to Avalon and it's database via some callback.
Think this is what @mkolar was referring to with create ftrack action that mirrors objects into avalon database. This can eventually be automated with ftrack event script that listens to new objects being created in ftrack and creates them in avalon too.
create ftrack action that mirrors objects into avalon database. This can eventually be automated with ftrack event script that listens to new objects being created in ftrack and creates them in avalon too.
Yes, excellent. This has so much potential, there's no limit to the amount of data we can push from ftrack into Avalon, and do any required "massaging" of the data during the push. It'd keep ftrack from seeping into the pipeline development effort and become a requirement, and instead make it an addition where the advantages are there, but none of the disadvantages (e.g. performance, internet connectivity, vendor lock-in) stands in the way.
Once we get up to speed with this kind of information synchronisation, then pushing data from other sources should also seem more obvious; like from Slack, Gmail or any old Calender app. Even other pipeline frameworks and tools, like cgwire.
I'd like to keep it so that there is no strict dependency on any third-party, but rather these loosely coupled integration. An immediate benefit of that is that you could cut ftrack at any point, and/or transition to something else or roll your own, and still do work in the meantime.
Absolutely agreed
But why from ftrack? What advantage does ftrack provide here over just using the Launcher for launching software?
If production management is ftrack based, people are spending a lot of time there, looking for comments, surrounding shots etc. launching from ftrack is very quick, intuitive and convenient. But most importantly, they are already there in context. I want to avoid artists having to go through ftrack tasks, find the one they want to work one (they have to do that always to check on comments anyway), the go to launcher that is separate app, browse to the task again and then launch the software. You're doing the same thing twice for no reason and it will get very annoying.
'My Tasks' in ftrack. It's not just about launching the software, but also about triggering timers, and having a broad overview of their work, which task view in ftrack offers really well.
If it's a question of familiarity or eye candy, then that's something within our control to change and improve. In the not-so-distant future,
that will be great but still a second layer that I would use only if ftrack is down for some reason. We have people switching between tasks and launching many task in parallel very often so any extra steps are a hindrance.
A very reasonable requirement is having the launcher running in the background and doing the actual app launching, what we certainly need though is a way to trigger actions in launcher via simple API. A way to send a command to the launcher which then does it's thing.
Then all kinds of integrations are easy. Even using slack as 'shell' if someone wants that. You write 'launch maya' into slackbot, it triggers python script that tells launcher to launch maya. (extreme of course, but easily possible)
So the question now is... Is it currently possible to use separate python scripts to create assets and tasks in avalon and is it possible to communicate to launcher from outside? If not what can we do to start implementing such things.
Hi, If I may, I have something to add.
A very reasonable requirement is having the launcher running in the background and doing the actual app launching
Maybe this will be like avalon-tray or some kind of service?
I think the Avalon Launcher GUI is like a simple version of ftrack which also send the launch action but without other project tracking features.
Further more, If we are going to track the app's working status, I think that API will need to be able to receive task info's registration (maybe plus PID or else), not only for the apps launched by artist through start menu or some other way outside avalon or ftrack, and for the situation when the service went down, quit or restart.
That said, the app will become like clients ( if I think this right ), can receive task info when launch or send it to the service when the artist modified it or resync the info when service is back on.
I've got an idea.
How about if when booting Launcher, it looks for AVALON_PROJECT, AVALON_ASSET etc. in it's local environment. If found, it automatically traverses to the final destination upon launch. Leaving the user with the single choice of picking an application.
That way, from the perspective of ftrack, it's launching just a regular old application, Launcher. Setting a few environment variables it derives from its own environment, like project and asset. Whilst at the same time enabling standalone use of Launcher for studios/users that either don't have ftrack or for whatever reason would prefer launching things without it.
How about if when booting Launcher, it looks for AVALON_PROJECT, AVALON_ASSET etc. in it's local environment. If found, it automatically traverses to the final destination upon launch. Leaving the user with the single choice of picking an application.
I like it :) Simple and very easy to implement in Ftrack (and possible other management services).
What would happen if the avalon asset is not found? Prompt to create? This would happen if the syncing would not be implemented or working.
@mkolar What do you think? Its one extra step compared to launching the application directly from Ftrack. We would just have a "Avalon Launcher" action, that opens the launcher in the right context.
What would happen if the avalon asset is not found? Prompt to create?
I'd consider that a bug in the synchronisation. I'd throw an exception and rely on us the implementers to fix it rather than effectively passing on the "hot potato" to the artist.
Having said that, it's possible that could be the way synchronisation happens. Assets could be created in ftrack without ever interacting with Avalon, until it comes time to launch an application. Perhaps that way ftrack could contain lots more information that never necessarily enters Avalon.
There'll be pros/cons to each approach, but I can see either of them working enough to throw together an MVP of the idea.
There'll be pros/cons to each approach
Would you get best of both worlds, by utilizing both?
With "Auto Syncing" you get assets that haven't need launched, but there is change of the syncing not working. With "Launch Syncing" you get assets that have already been worked on, but offline you couldn't launch any other.
Also what happens if Ftrack is down, and a publish is made? Would you need to republish to propagate to Ftrack, or would we rely on auto syncing or manually Avalon push tools for pushing to Ftrack?
Also what happens if Ftrack is down, and a publish is made? Would you need to republish to propagate to Ftrack, or would we rely on auto syncing or manually Avalon push tools for pushing to Ftrack?
I'd separate the ftrack publishing step from overall publishing, something that is normally done automatically during publish, but could be done independently, like from a command line, given some data and a file path. That way you can publish regardless, and manually sync with ftrack at a later time. It'd enable us to eventually build support for an "ftrack/production-tracker sync GUI" or the like.
That way you can publish regardless, and manually sync with ftrack at a later time. It'd enable us to eventually build support for an "ftrack/production-tracker sync GUI" or the like.
So we would not have any integration with Ftrack in the publishing stage? User of Ftrack/Avalon would probably expect results of a publish to appear immediately after a successful publish.
So we would not have any integration with Ftrack in the publishing stage?
You would, I mean the integration is a separate process from overall publishing into Avalon, a separate process that normally happens automatically. But, if things go south, this separate process can be triggered by hand.
It's a different way of thinking about and implementing it. Basically this..
$ ftrack --submit some_data.json
Where this command then is triggered from a plug-in during publishing.
class IntegrateWithFtrack(api.ContextPlugin):
...
def process(self, context):
subprocess.call(["ftrack", "--submit", path_to_some_data])
Where some_data.json contains all of what ftrack is expecting from any given publish, including files. And the ftrack command is something written by us to take this file and do the necessary work to push it onto ftrack.
Basically separating Avalon and ftrack at the process level, enabling individual use of each independently.
The ftrack.exe could of course be a Python library too, but the above is a good example of how separate I'm thinking this could be. If you think of it as a binary, I think it will become easier to reason about. Even if a Python library would simplify the actual implementation.
I'd expect this to happen eventually.
$ python -m ftrack --submit some_data.json
Where ftrack.py is both callable and importable.
import ftrack
ftrack.submit(fname="some_data.json")
Where the name itself, ftrack.py will need to be something else, because I think it's currently reserved for ftrack's official Python API.
Makes sense?
How about if when booting Launcher, it looks for AVALON_PROJECT, AVALON_ASSET etc. in it's local environment. If found, it automatically traverses to the final destination upon launch. Leaving the user with the single choice of picking an application.
Yes. However once we are that far, what is the issue with user choosing the app as well in ftrack? it's just one level deeper in hierarchy of the launcher and trivial to implement in ftrack action if we already have the 'launcher' action that sends you to the launcher's final destination. That's exactly the way I was thinking. Leaving the launcher in place and just telling it what to do from ftrack.
You would, I mean the integration is a separate process from overall publishing into Avalon, a separate process that normally happens automatically. But, if things go south, this separate process can be triggered by hand.
Why not use already existing pyblish-ftrack? it will slot in easily as integration in to avalon's pyblish.
Having the separated publish for cases when it fails due to whatever, could then instead of making completely new binary or python library, be a simple case of using pyblish itself to republish these missed assets (would probably require some thin wrapper over pyblish and ftrack api that would simplify it)
I do have another proposal though that might be more elegant.
In case of failed ftrack integration during publishing we could simply tag these publishes as 'local only' (or better tag the successful ones with 'ftrack_published') and have them all republished or finish the publishing once ftrack is up globaly, rather than choosing explicitly what to publish. Keep in mind that if ftrack is down for half a day, a team of animators and modellers can actually producer a fair bit of reviewables in that time so doing all missed ones at once might be useful.
This could even be done as a project wide action in ftrack. That a projectmanager could run after any downtime happened.
sync_publishes_action.py
1. Look into avalon database
2. Find any publishes missing 'ftrack_published' tag
3. sync them to ftrack database
this could of course work as push or pull system. Pull being ftrack action, push being some sort of executable that marcus mentioned.
I have the first very simple version of ftrack to avalon action.
It doesn't really take into account any special scenarios, complicated hierarchies or anything like that, but when ran on a project, it gets created, while if ran on an assets or shots, they get created in the corresponding project (works on multiple selection) in avalon db.
From this point at least this part of the equation should only be the case of improving, but the concept works
For the initial project setup it can load a config file which has a matching name to the ftrack schema (right now only source from the same directory as the action for easier testing)
It only resides here for now, but with more work it'll get cleaned up and organised somewhere I expect :)
https://github.com/kredencstudio/avalon-config-kredenc/blob/ftrack_actions/colorbleed/ftrack/action_sync_db.py
I've also managed to launch avalon initialised app by simply adding all the needed env vars while launching ftrack_connect and tweaking the app environment during launching to copy task context into avalon variables. That is also a trivila solution that most likely will get replaced with something more robust and hopefully bypassing ftrack-connect
Looking over you action sync code creating a project is as simple as this:
def create_project(project_name, config):
os.environ["AVALON_PROJECT"] = project_name
io.install()
template = {"schema": "avalon-core:inventory-1.0"}
inventory.save(project_name, config, template)
io.uninstall()
What's the content of the schema you're loading? Like, what're you passing to config?
Once we have a simple entry point to create a project (even when outside of any) and then afterwards allowing to switch/move into the project context then the project manager should be able to get a "create project" button as well!
I'm using these pre-made .toml files. Ftrack works on the basis of schemas that dictate to ftrack server everything about the project configuration, kinda like .config.toml files do in avalon. So I query the name of the schema from ftrack (in our case vfx, commercial, '2d_show` etc.), find .toml file with the same name and load it.
So it's coming from here for testing https://github.com/kredencstudio/avalon-config-kredenc/blob/ftrack_actions/colorbleed/ftrack/vfx.toml
My though was to eventually just build it on the fly based on information pulled from ftrack schema, which also contains available tasks. path templates and softwares could be save on ftrack attributes. so producer could just fill out some details on ftrack project, and avalon would duplicate it into config.
I think that's a good idea. At the end of the day, TOML is just a file format for a plain old dictionary, which is all you need to sync with the Avalon database. I've had in mind for some time to support multiple file formats based on what's more comfortable to the user, like JSON and YAML. The reason for choosing TOML was because the files were hand-edited at first, and still are to some extent, and (1) JSON isn't friendly enough for that (those darn commas!) and (2) YAML didn't have a cross-Python compatible library (either Py 3 or 2) that I could distribute alongside Avalon, plus it's darn complex of a format, with 15-20 different ways to write even the simplest things, and includes a way to write source code too. With TOML there was less risk of user error.
Having said that, if you generate these procedurally, then you wouldn't need a file and could communicate with dictionaries natively.
@mkolar I remember you were prototyping with this and getting some decent progress. How are things now? Or what ended up holding you back?