EddieOffermann dot com

I've been working on the spec for Big Blue Ceiling and thought I'd share at least an introduction to what the project will deliver. I've changed jobs recently (still in the vfx software/pipeline/td world) and that, among other things, has interrupted the project a bit but I'm ramping back up on it now.

Below are some thoughts on what a pipeline is and how I envision Big Blue Ceiling.  There is so much more to say, but much of it will have to remain unsaid until the project is closer to launch. I believe it will change the world and transform how and where we do business.

What is a visual effects pipeline?

A vfx pipeline is a collection of tools, processes and standards that permit and ensure the following:

-    Efficiency of communication of assets

o      Models

o      Animation

o      Camera data

o      Textures

o      Lighting information

o      Simulation data

o      and anything else necessary to assemble a given scene

-    Standardization

o      Filenames

o      Storage hierarchy

o      Version management

o      Tools

-    Show, Sequence and Shot Management

o      Structure

o      Dependencies

o      Assignment and Evaluation

A formal visual effects pipeline in a traditional effects house serves to automate the transfer of data by abstracting it. Separating camera characteristics, lens and movement data, from its strict representation inside the software package. Treating models, rigs and animation and separate elements that can be updated individually: for instance, animation may begin before a character design is finalized and the models updated on the fly, per shot or across the entire show without losing the animation data or requiring a complicated manual process to transfer that animation onto the updated model or rig.

On the compositing side, it may involve providing an additional set of facility-wide tools that standardize certain effects or provide a particular look.  Tools for grading and color management may be developed for the facility or the particular show and distributed between the artists to ensure that different compositors are able to deliver visually similar shots with minimal headaches.

For the entire facility, a formalized pipeline will ensure that assets and media are located with predictable names and stored in standard locations, along with maintaining project history and asset version information. If the last animation, the last comp, or last week's version of a model are preferable, it's easy to return to that version - often this can be made to happen at the level of producers without having to send the shots back to an artist to make the necessary adjustments.

What then is a cloud-based visual effects pipeline?

With the emergence of cloud computing, many common applications are moving onto a new platform: instead of being contained by a conventional desktop operating system or local server infrastructure, they are being moved onto platform-agnostic, internet-based hosted application clusters. Popular examples that many users are familiar with are such services as Google Docs, the office suite provided online as part of Google’s service offerings, or Photoshop.com, Adobe Corporation’s online photo editing and gallery hosting service. Medical information, insurance software, procurement & fulfillment systems, and tons and tons of software development have all left localized corporate infrastructures and moved onto the Cloud.

For visual effects and other CG projects such as animated features or game development, the process is a little more complicated. It’s still impractical to move software like Maya, 3D Studio Max or Houdini off of workstations and onto a web-based platform. Pipeline management has been complicated by the sizes of files that are often involved, but more recent developments in existing internet infrastructure have brought fast broadband into the range of slower local area networks, opening up the option of intelligent management and delivery of data by software such as Big Blue Ceiling. While many facilities are hesitant even to share projects with other facilities because of the combination of file transfer times and manpower overhead involved in delivering hard drives or preparing data before and after manual transfers over FTP. A Software-as-a-service or cloud-based system such as Big Blue Ceiling handles the typical database and project management aspects of a traditional in-house pipeline, as well as managing data for effortless transfer between facilities or between facility and remote artist.

What is a “Software as a Service” solution?

Software as a service (SaaS, typically pronounced 'sass') is a model of software deployment whereby a provider licenses an application to customers for use as a service on demand. SaaS software vendors may host the application on their own web servers or upload the application to the consumer device, disabling it after use or after the on-demand contract expires. The on-demand function may be handled internally to share licenses within a firm or by a third-party application service provider (ASP) sharing licenses between firms.

What is Big Blue Ceiling?

Big Blue Ceiling is a special subcategory of SaaS solutions referred to as SaSS (Software as a Secure Service). Software as a secure service (SaSS) is a derivative of software as a service. SaSS denotes a class of software as a service which emphasises security, not only in the link to and from the service, and the storage of any content by the software providing the service, but also in the security of the user in terms of the ability to make consistent backups and restores of any data stored in the service, in a non-proprietary format. In other words, security in transmission, storage and control over the user's own data.

In the context of a Cloud-Based Visual Effects Pipeline Software-as-a-Secure-Service, Big Blue Ceiling is a method of providing the sort of sophisticated visual effects pipeline normally restricted to larger, established visual effects houses, not only enabling it to be used by smaller studios but at the same time decentralizing it in such a way that it is as efficient for a group of broadly spread artists working in different locations as it is for a tightly integrated team of artists in a centralized studio.

Why would I be interested in a cloud-based, software-as-a-service solution for my visual effects pipeline?

The traditional rationale for outsourcing of any IT system involves applying economies of scale to the operation of applications, such that a service provider can offer better, cheaper, more reliable applications than companies can themselves. Several important changes to the way people work have made the rapid acceptance of cloud-based solutions possible and these changes are even more notable when we consider the visual effects community.

I.               High performance computers are widespread: Most cg & visual effects artists not only have a home computer but have one capable of performing functions far in excess of the basic needs of checking email and consuming online media.

II.             Processing power is a commodity: In the vanishing past, innovations in hardware were considered strategic advantages. From optical printers in the analog days to the first digital to film transfer processes of the TRON era, for many years hardware was king. More recently, proprietary applications and software tools were viewed as strategic. Today, people know it’s the business processes and the data itself: customer records, artist techniques, pricing information, and good effects design. Computing and application licenses are cost centers, and as such, they’re suitable for cost reduction and outsourcing.

III.           “Insourcing” pipeline systems requires expensive overhead including salaries, health care, hardware, software and OS management, liability and physical building space: not to mention unproductively reinventing the wheel!

IV.          Applications have tended to standardize: with a few notable exceptions, most people spend most of their time using standardized applications. A handful of standard 3d software packages dominate the market and they’re capable of exchanging data in a small set of standardized exchange formats. This means that a comprehensive solution for managing digital assets will work for a wide range of projects and facilities. Purpose-built, facility-specific pipelines should be regarded as dinosaurs.

V.            Web systems are incredibly reliable: Despite sporadic outages and slow-downs in the past, most people today are more than willing to use the Internet as a critical component of their business. As of early 2010, the visual effects community is behind many other information-related industries in moving out of centralized work clusters and into the cloud, largely because there has been no sound approach to managing the complex assets and often considerable data transfers involved.

VI.          Security is sufficiently well trusted and transparent: Secure communication no longer requires a complicated VPN setup or, worse, leased lines. Confidential client data can be handled even remotely with little risk.

VII.        Bandwidth of wide-area networks has grown drastically following Moore's Law (more than 100% increase each 24 months) and is reaching the bandwidth of slow local networks. Added to network quality of service improvement this has driven people and companies to trustfully access remote locations and applications with low latencies and acceptable speeds. Additional layers of asset data management by Big Blue Ceiling additionally boosts apparent speed of access while boosting quality of the artist experience.

VIII.      Cloud-based “Software as a Service” solutions have the effect of democratizing software, allowing small and medium businesses to have access to functionality formerly the domain of large enterprises. Big Blue Ceiling provides pipeline services at a level that small and even midsize companies could never afford to develop on their own, with a continually evolving set of tools and features unmatched even by in-house proprietary systems.

Who can benefit from Big Blue Ceiling?

Simply put, nearly everyone!

An entirely centralized studio can use Big Blue Ceiling and continue working as a centralized facility, leaving ongoing pipeline development to the Big Blue development team, regularly rolling out new features, support for additional packages and powerful artist tools, while letting the studio focus on creativity and project execution. And that centralized studio can rest easy knowing that if they need to expand, open up other locations, or cooperate on projects with artists and other facilities around the world, a set of tools are already in place for them to make sharing data completely effortless!

Smaller, newer facilities, perhaps formed just to accomplish a single project such as an animated short or independent animated feature can benefit enormously from a convenient slip-on pipeline like Big Blue Ceiling. It’s low entry cost provides world-class effects facility capabilities at a budget that nearly any production can absorb. Your cost savings just in artist hours transferring files and maintaining versioning, will easily exceed the entry cost of the service for small projects.

Loose collectives of artists will benefit from the Big Blue toolset as well, whether they’re seasoned industry professionals working purely for the love of the art or students collaborating on a project, access to the class of data management tools provided by Big Blue Ceiling should easily catapult any group’s efficiency.

Mid-size and larger facilities will find the toolset handy not only for their in-house work but for the ability to quickly and easily add artists in remote locations, or to easily permit secure access to their assets and production database by producers and vfx supervisors who may be offsite.

Additionally, tools that monitor times to complete tasks are able to chart them against production cost estimates, easily indicating unexpected burdensome cost centers, suggesting areas that estimates might be adjusted or manpower requirements reconsidered to adjust schedules. These are benefits that very, very few pipelines can provide, even in large established facilities.

Read this question on cgtalk tonight and thought I'd kick the question and my response out here since I've repeated much this same advice many, many times.

I use Boujou for matchmoving but am tracking 640x480 camera resolution all of the time... I often have shots with handheld very shaky camera and motion blur. Boujou has trouble tracking it. Is it just that there's to much blur/shakiness or would a higher camera resolution help?

and my response:

Well, it looks like there are a number of issues to overcome:

First, the quality of the footage: handheld/shaky/moblurred footage is just awful to work with. If you're just the artist, ie, you get what's handed to you: you work with what you have. There's no telling Mann, Fincher or Stone they have to reshoot because your job is hard. Their job is harder and it's them the client paid for, not you. If it's for your own projects, ie, you're the director or at least the vfx supervisor on a smaller project: you have to take this into your hands (so to speak) and fix it before it gets past the camera! Since you're working with 640x480 footage, I have to assume you can address it at this stage: these are obviously personal projects, not features or broadcast. Fix the source first! Don't shoot/allow to be shot footage that's going to suck to do post on!

Second, the resolution of the footage: Until you've worked in HD or film res, it's hard to appreciate just how much easier it is to work with on a tracking level. A dot that is so small it'll disappear into noise at NTSC is an obvious crosshair at 2k. Smudges on the wall and skin blemishes become usable tracking points. If you have the option to work with higher resolution footage, insist on it!

Third: Lighting. Again, this is only something you can influence if you're shooting your own projects, but remember to light for contrast not final levels. More light means faster shutter speeds means less motion blur. Light for the type of shadows you want, not how dark you want the scene when you're shooting for visual effects. If your key and fills are set properly, you can always make the shot darker when grading. This isn't as much of an issue when shooting film or deep color digital like the Arri Digital cameras since their shadow detail is good enough to boost for tracking detail. But if you're shooting with consumer or prosumer cameras, you've got jack but noise in the shadows and the low light levels will mean excessive blur and bad noise overall.

Fourth: Manual vs Auto. Boujou has a reputation as being the fire-and-forget solution for people that don't understand tracking. The really hard shots are going to need manual solutions. I don't mean hand-keyed (necessarily) but we're talking careful selection of usable tracking detail and improved setting of constraints. Personally, I recommend SynthEyes - unbelievably good price point and an unbeaten feature set. I know there are shots that people say they can click a button and get a solve on in Boujou that they can't get in SynthEyes, but the reverse is true far more often than it's not. When the footage is hard to work with, throw away the automatic solutions. Give them one run at it if you've got the time and processor power to spare, then move to something where you have some control.

Fifth: Tracking techniques. Track the center, not the edges. When doing a supervised track of moblurred features, follow the center of those blurs.

Sixth: Rendering techniques. Moblurred footage is hard to render to match because what you have to match is a particular slice of time. Think of it this way: you're placing tracking points on each frame, generally spaced apart at the rate of 24, 25 or 30 frames per second. Between frame 1 and frame 2, there's a fraction of a second where anything could happen. The camera could jump up a couple centimeters and back down for instance, before hitting frame 2. This will show in the motion blur but not in the keyframes. You can't easily fix for this! (See tip 1). More often, though, the problem is the phase of the render. How to address this will depend on your renderer. You may be able to adjust this in the renderer, you may have to adjust it in the actual animation keyframes depending on what 3d software you're working from. Basically, you need control of the shutter timing and shutter offset: when in that frame it opens and how long it stays open.

For high end vs. low end facilities, #6 can often be the difference between them. For all the skill you can buy from freelancers in tracking, modeling, animation, textures, etc: having the resources to do test renders and tweak that setting (and having someone on staff who can make sure you're addressing this) makes all the difference.

A few weeks ago, I posted a video covering the process of relighting rendered CG elements in a compositing tool like Shake using a surface normals pass.

Since that covered normals-based relighting of rendered cg elements, I thought I'd write an article that takes the concept a little farther and should stretch the reader's mind on what's possible with this sort of approach.

The earlier tutorial covered something that naturally can be accomplished by going back to the original scene in your 3d application, adjusting lighting and re-rendering: but I think there are times when we want to adjust the lighting primarily in the compositing application but want more than some masked-off gamma correction, lifts and crushes. So it's handy, then, to be able to apply an effect more similar to real-world lighting and that's what a rendered surface normals pass will enable us to do.

But what about real world imagery? Can this same technique be applied to live action footage just as it can be on a CG element?

Absolutely!

You'll want to start with a few elements like the following:

A full-color image like the above picture of a farmhouse. Under ideal conditions, you'd have an image with very diffuse lighting: something with a lot of bounce light or an exterior on an overcast day. The idea here is that there are relatively few real shadows in the image. The reality is that you won't have that very often and you'll have to use what you've got. You may decide that the shadows that do exist need to be painted out before you continue. I haven't done that for this example, but you'd be hard pressed to find a high end effects project that relit a real world environment and didn't remove the existing shadows at some point in the pipeline.

The above image is a matte of the farmhouse pictured above. You'll probably need to generate something like this for your scene if you haven't already.

A surface normals pass of a polysphere with its shading set to be as unsmooth as possible. I prefer this because I have a better feeling of where these surfaces are facing than I do straight from a sphere. It makes it like a special kind of color palette so I can perform the next step:

Next, I roto out a super-simple normal map of the surface in the image. This can be more or less complicated, depending on how much detail you want in the relight. If the surfaces are softer, you may want to blur the resulting image some: this will soften the transitions. Pick colors for each of your surfaces from the faces of the polysphere normal pass, making sure to keep parallel faces the same color as you interactively create a normal pass for your real-world element.

Then, I set up the OpaqueWhite OneLight Plus node as described in the normals relighting video here. Shake users can download the tool for free here.

It'll look similar to this:

Over on the right, you see where I've created the interactive surface normals pass. Those are groups with a few quickshapes and Mult nodes layered to create the interactive normals. In this case, since I didn't build out the ground and the barn next to the farmhouse, I chose to use the alpha from the interactive normals as the matte for the OW_OneLight+ node instead of the matte extraction I showed above. A smoothly blended surface for the ground, and a couple blurred normal shapes for the trees could make this quite convincing.

On the left, the fade leading into the over and the mix node to mix the whole thing back together is just one way of tweaking the levels of relighting that you're doing.

I posted a really short render of the light source moving across so that you can get an idea of what can be done. I kept it fairly subtle, and keep in mind that a little more work to create normals data for the ground, barn and trees would be required for a real project.

Moving footage naturally requires animated roto work to generate the interactive surface normals pass (keep the colors the same unless the element itself is moving!), and more complex objects with broader camera moves can be relit if you:

1. Build 3d geometry to match the visible elements
2. matchmove that geometry to match the footage
3. render a surface normals pass of those elements
4. use that rendered surface normals pass to relight the live footage.

In a future installment, I'll describe a method to generate surface-normals data by painting a bump map onto an image. In the meantime, I'll leave it as an exercise to the curious reader: how could you convert a simple greyscale height map into surface normals data in Shake?