EddieOffermann dot com

I've been doing a bit of coding specifically to make it easy for Big Blue Ceiling to be deployed into a facility that already has an established file structure.

By itself, this is a natural approach - and not that complicated. Because everyone with *any* sort of organization will have arranged their projects by job and by shots on that job, even though the exact heirarchy may be different it's easy to predict where the *gaps* in that heirarchy may exist and allow those to be configurable by the end user.

For instance, Company A, using predominantly linux machines may have a path to their publishing directory like this:

/FACILITY/cg/PROJECT/SUBPROJECT/shots/aa010/pub/

Company B may use a path on their Macs more like this:

/Volumes/Share/projects/PROJECT_SUBPROJECT/shot/aa010/publish/

It's not exactly brain surgery to identify that in the first case, the meaningful root directory is "/FACILITY/cg/" and in the second it's "/Volumes/Share/projects/" - then there's identifying the delimiter between "show" and "subshow" or "campaign" and "spot" or whatever your nomenclature is for project & subproject info. Maybe they're grouped as folder/subfolder, maybe they're companion folders.  Maybe there's no concept of "subshow" at the facility and we can treat both the delimiter and the subshow as empty.

But this got me to thinking: what I'm doing is creating a visual effects pipeline that is essentially filestructure agnostic.

A large part of a formalized visual effects pipeline is putting the filesystem behind an abstraction layer from the user.  Camera data moves along from artist to artist with each artist knowing nothing more than what shot they're working on. Someone will have specified, early on, what assets will be needed in the shot: ideally, when the shot comes to the animator, they pop open Maya and click a button that makes it populate their scene with the camera, models, set geo, rigged characters, etc.

They don't need (nor do they really want) to know that the model is called "candyBarDancer_v09.mb" and it's stored in "/Volumes/Share/projects/CANDY_DANCE/shot/cd090/publish/rig/".  They don't need to know it, they don't want to know it, and I believe they shouldn't even be *aware* of it because that information is nothing but a distraction from what they're trying to do.

When they're done, they can save a copy of their scene wherever they like for their own edification.  But the data needs to travel to the next artist - and they shouldn't have to tell the next artist where they saved it: again, they shouldn't even *know* this information because the data that gets written out isn't directly usable by them anyway. It's the formal vfx pipeline toolset that manages the handoff and makes sense of the data. Not the artist.

When the pipeline is properly structured, it's perfectly reasonable for multiple artists to work on the same shot at the same time: all the while getting regular updates of animation being done by other artists, receiving model changes relevant to their scene, etc. - all without having to be conscious of where this data is coming from and without having to do anything more than click an "update scene" button.

But here's the big deal!

By building the pipeline software itself around a filesystem agnostic abstraction layer, it allows us to link up established remote facilities. Company A in Los Angeles and Company B in England could be working on the same shots on the same shows, can still actively collaborating using something like Big Blue Ceiling coordinating between their facilities. The fact that their filesystems may have different naming schemes and different heirarchies, even calling the job and shots by different names, isn't going to affect their ability to actively communicate with one another.

Think about it: you have a small studio in Santa Monica and you're working on a medium budget feature with only a handful of effects shots. One of the sequences, however, needs a large set extension and a matte-painted cityscape in the background. You task your team to get to work on the foreground elements of the set extension and contact a small studio in Austin, passing along the requirements for the elements and having them install the toolset for using the shared effects pipeline. They can easily fit it into their filesystem and in no time, they're up and running.

As your artists work on the foreground elements, they register with the system to receive notifications as elements are updated. They click a shelf button to update their scene and as they build the foreground, they see the background elements and matte painting slowly taking shape: right there in their scene.

There's no messy FTP'ing of files back and forth, and no complicated backend server synchronization to set up. They don't even use the same file heirarchy: but that doesn't matter because structure and naming conventions are handled by the pipeline software, not the artists, so there's no confusion.

This is the promise of truly good pipeline design.

And Big Blue Ceiling is going to make that a reality.

As long as we're here, let's take a look at a commercial that I look on fondly, because a lot of what's described above I already wrote once when at Asylum and was used to create this spot.

I was talking to a friend today about what I'm doing with regards to managing data through an animation pipeline using XML. The more I work with it and the farther I get into the project, the more flexible and powerful the whole thing seems. Of course the goal to doing the implementation in Python is that virtually every software package in the vfx industry is python-friendly - so once the core routines are written, everything from Nuke and pyShake (the python plugin for Shake - if you haven't seen it yet, check it out here) to Maya, Houdini and RealFlow will be able to make use of them. I think most places are doing that these days, with a few nods to TCL/tk here and there - but broadly supported scripting languages are King and open description formats like XML are Queen.

My friend marveled at how nice it would be if one day, a couple years from now, everything was able to talk that smoothly: that a character animated in Maya could be pulled into Houdini, for instance, as something other than an OBJ sequence or a separately rigged character that you had to tediously (or with a lot of specific coding) link to exported channel data.

I wonder if that interoperability thing will ever extend beyond each individual studio's implementation. Everybody has a way of getting software to talk amongst themselves, some solutions being more elegant than others, but when you invest in creating something as elaborate as this it becomes your own proprietary tool. If you develop a tool that an animator can take an animated character with a complex rig on it, arbitrarily select additional elements that were never *really* meant to be animated and animate them anyway, and the modeling team can modify the model and issue a new version of it - and the animation gets seamlessly transferred over to the new model, even able to be read into RealFlow, substituting a different set of low poly independent objects that are driven by the data in that XML file: you don't put that pipeline tool on the internet for everyone to download for free.

That tool becomes your secret weapon. As a studio with an investment in a powerful and unique proprietary tool, even charging for it may not mean as much to you as the edge you gain during the heat of production.

Being XML based and implemented in Python does put my current project a wee bit closer to being an open standard, though. Even Shake will take Python scripts now - and they're really powerful in it and getting more so as development continues. The readability thing for XML is a gigantic plus, and the way it represents data is great. I can build a module that will write out the translation of a locator in both world and local space, as a baked set (every frame has a value) and as a set of keyframes (values only for those frames where the value was explicitly set by the artist), as well as screenspace UV values - so the same XML file could reconstruct a scene for a lighter to light and render from or another animator to tweak the animation curves, or for RealFlow to drive low-poly proxy objects with to disturb a drifting mist, or for a compositor in Toxic to link an effect to. And it's all one XML file - not a half dozen formats (often multiple versions of each) and a hundred-unit sequence of geometry exports.

I am officially in crunch...

It's mini-crunch, though, because we're just working to get things together for the trailer. Most of the staff isn't actually working overtime, but there's a few of us... the dynamics/fluid guys, an animator or two, the cg supe and me, piecing together the necessary bits to get the shots ready for the trailer.

I wish I could be a little more clear on what exactly it is I'm doing... When the trailer comes out in a couple weeks I can at least indicate then what segment of the film I'm addressing - I'd rather leave that part as a surprise for when the trailer hits (not to mention that talking about things like that would get me in hot water).

In a sufficiently vague sort of way, I can say that there's a segment I'm working on right now that's around 76 shots long, each ranging from 50-300 frames, that has a common element that's travelling between them. Our tracking team has been tracking that element, and its progress through each individual scene, for each of those 76 shots (and at least that many more as well that I'm not dealing with yet).

The particular task I've been working on is basically a continuity script. It's a MEL script that gathers all of that data: cameras, animated and matchmoved elements, and stitches them together end to end in a master shot. This way, the element that's moving through the scene will maintain an accurate location relative to the CG environment.

To coincide with what's happening in each shot, it's necessary on that big grandaddy master shot to make adjustments: turn something here, nudge something there... and that affects the path of that common element throughout all of the individual shots.

Do you see where I'm going with this?

Basically, if we imagine that I'm talking about a man driving a car across the salt flats in Utah (not at all what's happening in this film), there are features on the horizon that we want to be consistently and realistically moving in relation to the view of the car. If the driver swerves to miss a rock, we want to change the path he's taking across the salt flats (even though in the original plates, that driver would have been sitting in a stationary vehicle in front of a bluescreen, with some hydraulics underneath making the car tilt and sway to add realism). Our view of the car from the camera should stay the same... Car and camera must maintain their relationship... but the position, speed, orientation, etc of the car and camera combined will need to change. If there's a guy on one of those sail-carts out there, we need to make sure his path is realistic and that he's in the right place in the background of each shot...

Remember that the car, the driver, and the camera are often the only real things here... the sail-cart, other things on the horizon, in fact the entire salt flats... all are cg or augmented with cg.

So I've been developing tools for funneling data from all of these shots... everything from camera data to object matchmoves, world-space offsets for each shot, and animation data... stringing it together into a seamless master scene, manipulating the paths and locations of elements in that master scene and transferring that data back into the individual shots. If we need our hypothetical salt-flats driver to swerve this way and that across several separate shots, I'm making sure that it doesn't just "kinda look right" but that we can get that data integrated into the object and camera motion of several live camera plates.

If somebody didn't ask me if I could do it, I'd probably have never thought it could even be done.

If I'm kinda scarce around here in blogland, it's because I'm catching a much-needed nap in between long shifts.