I needed a way to quickly create a usable "right eye" camera to correspond to a "left eye" camera. I needed to be able to specify interocular distance (the distance between the left and right eye), as well as the convergence distance (where the eyes were focusing). Our eyes naturally "converge" on objects that are in front of us - this is part of how we judge relative depth as this convergence affects the apparent relationship if the dual images our eyes perceive of *other* objects that are not at the point of convergence. But that's a subject for another entry.

Anyway, the tool needed to be able to take a tracked "left eye" camera and apply that tracking data to generate the "right eye" camera using convergence and interocular information that was either derived from a track, noted on set, or dynamically assigned in Maya. I wanted a right eye camera that could be created instantly and that would automatically follow any changes I made to the left eye camera. If I smoothed that camera's path, I needed the right eye to follow suit, if I performed some dramatic transformation to that camera's path, perhaps extending its animation or re-animating an object that was part of the track while conforming the camera to that object's new path: I needed the right eye camera to keep up. If a cg element was being added, moving toward the camera, I needed to be able to selectively lock the convergence to "look at" that object's depth without diverting the cameras to actually look directly at it - similar to changing the focus depth.

It's all about control and instantaneous response, with a camera whose path is known with some degree of certainty and a second camera that will follow that one in a way that will produce realistic cg and will enable artists to quickly and accurately duplicate the second camera position when the original interocular distance and convergence information is not known.

Matrix multiplication, as a mathematical operation, isn't complicated - it's simple multiplication and addition, just repeated a number of times to generate the required result matrix. Since there are already more-than-ample tutorials online about how to actually *perform* matrix math (if you were cursed to do it by hand or develop procedures to support it in programming environments that don't already have matrix math support) I won't be covering that in detail. If my simple explanation doesn't quite do it for you, put "matrix multiplication" into Google if you're stumped.

I'm also going to avoid covering application in a specific language. Most recently, I was doing this in MEL for a project I'll discuss in a moment, but there are Python libraries for doing matrix math, TCL/tk support for it, PHP and Perl support - you'll rarely find yourself with no built-in or easily-added matrix support, though you may want to write (as I did) a number of routines to make it a little more accessible.

To the left, you can see the standard format of a translation matrix. This matrix (when multiplied by a coordinate matrix as represented on the right-hand side of the equation) will translate that coordinate into a new space, offsetting it by (tx, ty, and tz). In the typical way of matrix multiplication, the element in each row of the coordinate matrix [x y z 1] is multiplied by a column of the translation matrix, as shown here:

The rows of this matrix are then added together (in typical matrix multiplication fashion) to give the resulting (x',y',z') location.

But there's more to matrix transformation than simple translation. If you wanted to find the new (x', y', z') for a translation like this one, simple addition would be enough. But manipulating points in 3d space is rarely that simple!

Fortunately, it's just as easy to scale a point using matrix math! Always away from the origin - we'll talk about how to scale a point away from somewhere other than the origin shortly. All matrix operations work with a similar setup, so you'll get used to seeing a similar notation here.

This matrix is multiplied in the same fashion as the one we see above, rows against columns, with row sums producing (x',y', z') for the new location.

I'll admit, though. I don't use matrix math for scaling things very often. You know what I do use it for, though?

Rotating a point in space with respect to the origin! Isn't that exciting? I LOVE rotating things! Well, ok, so it's not that big of a deal - but when we start combining some of these things, it can turn a complicated object tree in your scene into a relatively simple expression.

I have to warn you, though, rotation is a bitch. Ever notice how your favorite 3d software has this whole "rotation order" thing? That's because if you rotate something 10 degrees in X, then 15 degrees in Y, it's not the same as rotating it in Y first and then in X. There's also a different matrix for each axis of rotation, so let's take a look. Same matrix math process as translation and scaling, but from this one we get rotation around the Z axis.

And now here:

That one rotates in X! And lastly, as you'd expect, there's a matrix for rotating around the Y axis:

Now, combining these matrix transformations together is as simple as multiplying the matrices with each other. Now, you build the series of multiplications in order from right to left, but they're carried out from left to right. For instance, for "zxy" rotation order, you would create an expression similar to the following: (I'll use MEL for this example: $xRot, $yRot, and $zRot are each 4x4 matrices that already contain transformation data for x, y and z rotation):

matrix $r[4][4] = $yRot * $xRot * $zRot;

Provided proper matrix variables are supplied, MEL supports basic matrix operations with some limitations that you'll find as you stretch your legs.

These can be strung together into much longer expressions generating much more complex matrices. To rotate an object about a point other than the origin, for instance, subtract the values of that coordinate from the coordinates of the object (transform it in {-cx, -cy, -cz} where {cx,cy,cz} was the center to rotate it around), perform the rotation, the transform it back {+cx,+cy,+cz}.

It may take some time learning to visualize and plan out a complex matrix transform - but what makes it powerful is the ability to combine any number of transformations into a single operation. Some basic trigonometry and a well-applied matrix transform can accomplish all kinds of things!

That hasn't been my problem for about 25 years now. I get it. This is how you multiply a matrix with a vector. This is how you multiply two matrices together. Neither of those things actually *accomplishes* anything if you don't know what the data in those matrices represents or what adding, subtracting or multiplying them would do.

My problems always center around something more like this: I have a point at <10,20,30> with orientation <15,30,45>. Where is the point if I rotate it 15 degrees in y then translate it 100 units in the X axis?

What's screwed up is that I've so frequently solved this by doing trig that I know off the top of my head that pi/180 is estimated at 57.29577951. Some people pride themselves on the number of digits they know pi to: I find the radian conversion number to be far more handy. But for reference, pi to me is 3.14159265. Note the 8 significant digits behind the decimal number: can you tell what decimal significance I grew up programming in?

The thing is, I'm working on a number of tools for managing stereo cameras in Maya. I need to quickly determine from the transforms of one camera (derived from a track of live footage) the location and orientation of a second camera if we know (or can estimate or otherwise determine) the interocular distance (the distance between the eyes) and the convergence (how far in front of these cameras their line of sight intersects). Determining the angle of the second camera relative to the first is a straightforward bit of trig: think Pythagoras and arctangents and it'll come to you, or go find a good trig reference online - this part's easily found in a million places. Diagram that out for yourself, making an equilateral triangle with cameras at each base corner with convergence being the length of a line bisecting the triangle into two right triangles. If it's not immediately clear, work on it a bit: it might wake up parts of your brain that haven't seen the light of day since highschool.

The tricky bit is figuring out where a right-eye camera would be if it was offset along a line that isn't quite perpendicular to the left-eye camera.

I'm very nearly there - based on a given camera, I can find a point offset along that camera's X axis and apply a convergence factor to the resulting camera location by canting it back in along the Y. But rotating a known transform matrix by an additional value in the y axis *before* doing that x translation... still fussing with that bit. I can make that happen less elegantly - by actually transforming nulls through space in Maya - but I'm searching for the magical transformation matrix that'll make it happen in a simple series of math statements.

It's late. It'll probably be plenty clear to me in the morning.

UPDATE: Got it! Sleep did wonders - my tired brain just didn't quite get the matrix arrangement right before.  I'll write an entry this weekend covering concatenated matrix transforms.  Damn handy.  Damn powerful.

I use Boujou for matchmoving but am tracking 640x480 camera resolution all of the time... I often have shots with handheld very shaky camera and motion blur. Boujou has trouble tracking it. Is it just that there's to much blur/shakiness or would a higher camera resolution help?

and my response:

Well, it looks like there are a number of issues to overcome:

First, the quality of the footage: handheld/shaky/moblurred footage is just awful to work with. If you're just the artist, ie, you get what's handed to you: you work with what you have. There's no telling Mann, Fincher or Stone they have to reshoot because your job is hard. Their job is harder and it's them the client paid for, not you. If it's for your own projects, ie, you're the director or at least the vfx supervisor on a smaller project: you have to take this into your hands (so to speak) and fix it before it gets past the camera! Since you're working with 640x480 footage, I have to assume you can address it at this stage: these are obviously personal projects, not features or broadcast. Fix the source first! Don't shoot/allow to be shot footage that's going to suck to do post on!

Second, the resolution of the footage: Until you've worked in HD or film res, it's hard to appreciate just how much easier it is to work with on a tracking level. A dot that is so small it'll disappear into noise at NTSC is an obvious crosshair at 2k. Smudges on the wall and skin blemishes become usable tracking points. If you have the option to work with higher resolution footage, insist on it!

Third: Lighting. Again, this is only something you can influence if you're shooting your own projects, but remember to light for contrast not final levels. More light means faster shutter speeds means less motion blur. Light for the type of shadows you want, not how dark you want the scene when you're shooting for visual effects. If your key and fills are set properly, you can always make the shot darker when grading. This isn't as much of an issue when shooting film or deep color digital like the Arri Digital cameras since their shadow detail is good enough to boost for tracking detail. But if you're shooting with consumer or prosumer cameras, you've got jack but noise in the shadows and the low light levels will mean excessive blur and bad noise overall.

Fourth: Manual vs Auto. Boujou has a reputation as being the fire-and-forget solution for people that don't understand tracking. The really hard shots are going to need manual solutions. I don't mean hand-keyed (necessarily) but we're talking careful selection of usable tracking detail and improved setting of constraints. Personally, I recommend SynthEyes - unbelievably good price point and an unbeaten feature set. I know there are shots that people say they can click a button and get a solve on in Boujou that they can't get in SynthEyes, but the reverse is true far more often than it's not. When the footage is hard to work with, throw away the automatic solutions. Give them one run at it if you've got the time and processor power to spare, then move to something where you have some control.

Fifth: Tracking techniques. Track the center, not the edges. When doing a supervised track of moblurred features, follow the center of those blurs.

Sixth: Rendering techniques. Moblurred footage is hard to render to match because what you have to match is a particular slice of time. Think of it this way: you're placing tracking points on each frame, generally spaced apart at the rate of 24, 25 or 30 frames per second. Between frame 1 and frame 2, there's a fraction of a second where anything could happen. The camera could jump up a couple centimeters and back down for instance, before hitting frame 2. This will show in the motion blur but not in the keyframes. You can't easily fix for this! (See tip 1). More often, though, the problem is the phase of the render. How to address this will depend on your renderer. You may be able to adjust this in the renderer, you may have to adjust it in the actual animation keyframes depending on what 3d software you're working from. Basically, you need control of the shutter timing and shutter offset: when in that frame it opens and how long it stays open.

For high end vs. low end facilities, #6 can often be the difference between them. For all the skill you can buy from freelancers in tracking, modeling, animation, textures, etc: having the resources to do test renders and tweak that setting (and having someone on staff who can make sure you're addressing this) makes all the difference.

"Through these collective efforts, a harmonized master list of titles has been created that should be useful as a touchstone for all effects stakeholders," says VES Exec Director, Eric Roth. "We think it represents a major step forward for our craft in standardizing the use of titles in the credits process for effects artists."

Of course, I'll interject here that if the Visual Effects Society actually carried any weight, someone would care. I'll further interject that if they'd actually attached descriptions to these titles it would mean a bit more - but since a good chunk of these are too nonspecific to mean anything (or at least vague enough to get people to fight over them) they've basically left us where we were before. (Except that since my responsibilities shift and sway on a regular basis, I can dig through the list for suggestions of what I should be calling myself - like this weekend, I think I'm calling myself the Visual Effects Photographer)

Questions regarding these guidelines can be addressed by calling the VES at 818-981-7861. Credits/Titles to be submitted in accordance with VES Guidelines follow after the cut:

PRODUCTION

Senior Visual Effects Supervisor	Visual Effects Production Assistant
Visual Effects Supervisor	Visual Effects Accountant
Senior Animation Supervisor	Visual Effects Plate Supervisor
Additional Visual Effects Supervisor	Visual Effects Plate Producer
Visual Effects Co-Supervisor	Visual Effects Editor
Visual Effects Associate Supervisor	Visual Effects Assistant Editor
Animation Director	Visual Effects Previz Lead
Animation Supervisor	Visual Effects Previz Artist
Senior Visual Effects Producer	Visual Effects Previz Editor
Visual Effects Producer	Visual Effects Previz Sound Editor
Visual Effects Associate Producer	Visual Effects Data Coordinator
Visual Effects Production Supervisor	Visual Effects Data Assistant
Visual Effects Coordinator	Visual Effects Vendor Coordinator
Visual Effects Assistant Coordinator	Visual Effects Consultant
Visual Effects Production Associate

FACILITY [PRODUCTION]

Senior Visual Effects Supervisor	CG Supervisor
Visual Effects Supervisor	Compositing Supervisor
Additional Visual Effects Supervisor	Senior Visual Effects Art Director
Co-Visual Effects Supervisor	Visual Effects Art Director
Visual Effects Associate Supervisor	Concept Artist
Visual Effects Plate Supervisor	Storyboard Artist
Visual Effects Consultant	Visual Effects Editor
Senior Animation Supervisor	Assistant Visual Effects Editor
Animation Supervisor	Projectionist
Senior Visual Effects Producer	Color Grading Artist
Visual Effects Producer	Color Grading Supervisor
Visual Effects Executive Producer	Previz Supervisor
Visual Effects Associate Producer	Previz Artist
Visual Effects Coordinator	I/O Supervisor
Production Associate	I/O Operator
Production Supervisor	I/O Neg Cutter
Visual Effects Production Manager	I/O Coordinator
Visual Effects Plate Producer	Scan/Record Technician
Visual Effects Plate Coordinator	Still Photographer
3D Stereoscopic Supervisor

FACILITY [DIGITAL UNIT]

CG Sequence Supervisor	Lead Texture Artist
Digital Production Supervisor	Senior Texture Artist
Animation Supervisor	Junior Texture Artist
Animation Producer	Motion Capture Supervisor
Animation Production Manager	Motion Capture Producer
Animation Coordinator	Motion Capture Engineer
Technical Animation Supervisor	Motion Capture Technician
Technical Animation Lead	Dustbuster
Digital Effects Producer	Lead TD
Digital Effects Production Manager	TD
Digital Effects Coordinator	Assistant TD
Character Setup Supervisor	Lighting Supervisor
Character Setup Lead	Lighting Lead
Character Rigging Supervisor	Lighting Artist
Character Rigging Artist	FX Animator
Rigging Supervisor	Character Animation Lead
Rigging Artist	Character Animator
Look Development Supervisor	Animation Lead
Look Development Lead	Animator
Compositing Sequence Supervisor	Layout Supervisor
Compositing Lead	Layout Artist
Compositor	Modeling Supervisor
Matte Supervisor	Modeling Lead
Matte Painter	Modeler
Matte Artist	Simulation Supervisor
Matte Painting Supervisor	Simulation Lead
Lead Digital Matte Painter	Simulation Artist
Senior Digital Matte Painter	Matchmove Supervisor
Digital Matte Painter	Matchmove Lead
R&D Supervisor	Matchmove Artist
R&D Artist	Rotoscope Supervisor
Senior Technical Developer	Rotoscope Lead
Software Development Supervisor	Rotoscope Artist
Software Developer	Digital Artist
Systems Manager	CG Artist
Systems Coordinator	Paint Supervisor
Systems Engineer	Paint Lead
Integrators	Paint Artist
Pipeline Supervisor	Digital Paint Lead
Pipeline Engineer	Plate Restoration Supervisor
Shading Supervisor	Plate Restoration Lead
Shading Artist	Plate Restoration Artist
Texture Paint Supervisor	Technical Associate
Texture Lead	Technical Assistant
Texture Painter	Render Coordinator
Texture Paint Artist

MOTION/PERFORMANCE CAPTURE UNIT

Motion Capture Line Producer	Motion Capture Camera Operator
Motion Capture Supervisor	Motion Capture Pipeline TD
Motion Capture Producer	Motion Capture Pipeline Engineer
Motion Capture Systems Technician	Motion Capture Stage Manager
Motion Capture Systems Engineer	Motion Capture Coordinator
Motion Capture Systems Operator	Motion Capture Assistant
Motion Capture Real Time Operator	Motion Capture Production Assistant

VFX PHYSICAL PRODUCTION UNIT

Visual Effects Director of Photography	Visual Effect Data Capture
Visual Effects Photographer	Visual Effects 2nd Assistant Camera
Visual Effects Camera Operator	Visual Effects Camera Loader
Visual Effects 1st Assistant Camera	Digital Camera Engineer
Visual Effects Survey Supervisor	Digital Camera Assistant
Visual Effects Surveyor

SPECIAL EFFECTS UNIT

Senior Special Effects Supervisor	Special Effects Best Boy
Special Effects Supervisor	Senior Special Effects Stage Technician
Senior Special Effects Coordinator	Special Effects Stage Technician
Special Effects Coordinator	Senior Special Effects Rigger
Special Effects Technician	Special Effects Rigger
Pyrotechnic Supervisor	Senior Special Effects Workshop Technician
Pyrotechnic Technician	Special Effects Workshop Technician
Head Special Effects Electronic Engineer	Head Special Effects Maintenance Engineer
Special Effects Electronic Engineer	Special Effects Maintenance Engineer
Special Effects Electronic Technician	Head Special Effects Machinist
Head Special Effects Stage Technician	Special Effects Machinist

MOTION CONTROL UNIT

Motion Control Supervisor	Motion Control Camera Assistant
Motion Control Producer	Motion Control Previz
Motion Control Cameraman	Motion Control Technician
Motion Control Operator	Motion Control Crane Operator

MODELS / MINIATURES UNIT

Models/Miniatures Director of Photography	Models/Miniatures Supervisor
Models/Miniatures Producer	Senior Modelmaker
Models/Miniatures Gaffer	Chief Modelmaker
Models/Miniatures Coordinator	Modelmaker
Models/Miniatures Sr. Camera Operator	Models/Miniatures Sculptor
Models/Miniatures Camera Operator	Senior Plasterer
Models/Miniatures 1st Assistant Camera	Senior Painter
Models/Miniatures 2nd Assistant Camera	Graphic Artist
Models/Miniatures Camera Loader	Creature Supervisor
Models/Miniatures Technical Apprentice	Senior Creature Tech
Digital Camera Engineer	Creature Tech
Digital Camera Assistant	Supervising Mechanical Engineer
Models/Miniatures Stagehand	Mechanical Engineer
Models/Miniatures Production Manager	Chief Scenic Artist
Modelshop Supervisor	Scenic Artist
Models/Miniatures Effects Supervisor	Assistant Scenic Artist

STOP MOTION UNIT

Head Stop Motion Animator	Stop Motion Lead Sculptor
Stop Motion Animator	Stop Motion Sculptor
Stop Motion Assistant Animator	Stop Motion Lead Model Rigger
Stop Motion Puppet Coordinator	Stop Motion Model Rigger
Stop Motion Puppet Wrangler	Stop Motion Lead Mold/Mould Maker
Stop Motion Junior Puppet Wrangler	Stop Motion Mold/Mould Maker
Stop Motion Lead Armature Engineer	Stop Motion Camera Manager
Stop Motion Armature Engineer	Stop Motion Camera Operator

ANIMATRONICS UNIT

Animatronics Supervisor	Animatronics Foam Latex Technician
Animatronics Coordinator	Animatronics Mold/Mould Makers
Animatronics Designer	Animatronics Sculptor
Animatronics Mechanical Supervisor	Animatronics Seamstress
Animatronics Technician	Animatronics Hair/Fur Technician
Animatronics Control Systems Designer

]]> http://eddieoffermann.com/blog/2008/05/24/visual-effects-society-call/feed/ 0 http://eddieoffermann.com/blog/2008/05/16/xml-python-visual-effects-pipeline/ http://eddieoffermann.com/blog/2008/05/16/xml-python-visual-effects-pipeline/#comments Fri, 16 May 2008 00:49:21 +0000 Eddie http://eddieoffermann.com/blog/2008/05/16/xml-python-visual-effects-pipeline/ I was talking to a friend today about what I'm doing with regards to managing data through an animation pipeline using XML. The more I work with it and the farther I get into the project, the more flexible and powerful the whole thing seems. Of course the goal to doing the implementation in Python is that virtually every software package in the vfx industry is python-friendly - so once the core routines are written, everything from Nuke and pyShake (the python plugin for Shake - if you haven't seen it yet, check it out here) to Maya, Houdini and RealFlow will be able to make use of them. I think most places are doing that these days, with a few nods to TCL/tk here and there - but broadly supported scripting languages are King and open description formats like XML are Queen.

My friend marveled at how nice it would be if one day, a couple years from now, everything was able to talk that smoothly: that a character animated in Maya could be pulled into Houdini, for instance, as something other than an OBJ sequence or a separately rigged character that you had to tediously (or with a lot of specific coding) link to exported channel data.

I wonder if that interoperability thing will ever extend beyond each individual studio's implementation. Everybody has a way of getting software to talk amongst themselves, some solutions being more elegant than others, but when you invest in creating something as elaborate as this it becomes your own proprietary tool. If you develop a tool that an animator can take an animated character with a complex rig on it, arbitrarily select additional elements that were never *really* meant to be animated and animate them anyway, and the modeling team can modify the model and issue a new version of it - and the animation gets seamlessly transferred over to the new model, even able to be read into RealFlow, substituting a different set of low poly independent objects that are driven by the data in that XML file: you don't put that pipeline tool on the internet for everyone to download for free.

That tool becomes your secret weapon. As a studio with an investment in a powerful and unique proprietary tool, even charging for it may not mean as much to you as the edge you gain during the heat of production.

Being XML based and implemented in Python does put my current project a wee bit closer to being an open standard, though. Even Shake will take Python scripts now - and they're really powerful in it and getting more so as development continues. The readability thing for XML is a gigantic plus, and the way it represents data is great. I can build a module that will write out the translation of a locator in both world and local space, as a baked set (every frame has a value) and as a set of keyframes (values only for those frames where the value was explicitly set by the artist), as well as screenspace UV values - so the same XML file could reconstruct a scene for a lighter to light and render from or another animator to tweak the animation curves, or for RealFlow to drive low-poly proxy objects with to disturb a drifting mist, or for a compositor in Toxic to link an effect to. And it's all one XML file - not a half dozen formats (often multiple versions of each) and a hundred-unit sequence of geometry exports.

Hi!  I haven't been updating the vfx blog because I've been crazy busy.  Work has me deep in MEL and Python code most of the day, writing software to manage animation data, shuttle it this way and that and process it in interesting ways.  There's also the secret project that everyone isn't talking about but is quietly and excitedly waiting for (assuming they have any idea what it is).   We're on super-special-secret-lockdown at work, I'm guessing for the rest of the month.  I can scarcely go to the bathroom or grab a soda without passing through at least two security doors now.

Ok, I admit it, we've been contracted to create the lost tapes of the lunar landing.  Seems NASA has gotten enough flack about the original landing with people claiming they can prove it wasn't real that they've decided to create more proof to put the concerns to rest.

Or at least that's my amusing cover story until I'm allowed to talk about the other thing.

Between all the other little tasks, I'm in the process of working up a new tool for moving complex animation data around the pipeline.  The idea is to be able to read and write an XML file that contains as much data as anyone would care to put onto a model.  The model can be updated, have new elements added to or removed from it, all sorts of parts of that model can be animated at a whim by the artist and passed along to subsequent versions of the model, low res or high res versions, keeping all of the data intact, allowing for scaling of the animation data if it's transferred to a different model entirely, even passing along worldspace location and orientation of parts of the model (and maybe even eventually the full animation) to other platforms like Houdini and Realflow.

It's a little important for what we're doing on The Curious Case of Benjamin Button, but the existing tools I developed are working ok there (even if they're still a little limited and occasionally awkward - they're getting better on a weekly basis).  There's already a small test of what I'm doing in a broader sense: we have a particular element with a moderately complex rig on it that I'm publishing and subscribing the animation data on.

The Big Goal for this tool will be on T4 which we should be working on soon.  That'll be a huge show and we've got some really epic-scale work on it: most likely we'll be needing the more sophisticated animation tools for managing our data throughout the pipeline - so that's what I constantly keep in the back of my mind as I script the current version of the tools.  They need to serve the current project in the most complete way possible, and they need to be able to expand to do more than I'd currently dream of asking them to do.

It's a lesson that's come up recently because of a little tool I wrote that I've mentioned here before "CurveBaby" - CurveBaby allows me to take a scene with an animated camera and one or more animated objects, preserve the camera's relationship to one of those objects but change that object's animation - or, conversely, reanimate the camera and have the animation of the object automatically adjust to still look the same from the camera.  It's a powerful tool, and one that I can barely imagine how I'd get through half a day without using it at least once.

Big tools make big projects go a lot more smoothly.

Since that covered normals-based relighting of rendered cg elements, I thought I'd write an article that takes the concept a little farther and should stretch the reader's mind on what's possible with this sort of approach.

The earlier tutorial covered something that naturally can be accomplished by going back to the original scene in your 3d application, adjusting lighting and re-rendering: but I think there are times when we want to adjust the lighting primarily in the compositing application but want more than some masked-off gamma correction, lifts and crushes. So it's handy, then, to be able to apply an effect more similar to real-world lighting and that's what a rendered surface normals pass will enable us to do.

But what about real world imagery? Can this same technique be applied to live action footage just as it can be on a CG element?

Absolutely!

You'll want to start with a few elements like the following:

A full-color image like the above picture of a farmhouse. Under ideal conditions, you'd have an image with very diffuse lighting: something with a lot of bounce light or an exterior on an overcast day. The idea here is that there are relatively few real shadows in the image. The reality is that you won't have that very often and you'll have to use what you've got. You may decide that the shadows that do exist need to be painted out before you continue. I haven't done that for this example, but you'd be hard pressed to find a high end effects project that relit a real world environment and didn't remove the existing shadows at some point in the pipeline.

The above image is a matte of the farmhouse pictured above. You'll probably need to generate something like this for your scene if you haven't already.

A surface normals pass of a polysphere with its shading set to be as unsmooth as possible. I prefer this because I have a better feeling of where these surfaces are facing than I do straight from a sphere. It makes it like a special kind of color palette so I can perform the next step:

Next, I roto out a super-simple normal map of the surface in the image. This can be more or less complicated, depending on how much detail you want in the relight. If the surfaces are softer, you may want to blur the resulting image some: this will soften the transitions. Pick colors for each of your surfaces from the faces of the polysphere normal pass, making sure to keep parallel faces the same color as you interactively create a normal pass for your real-world element.

Then, I set up the OpaqueWhite OneLight Plus node as described in the normals relighting video here. Shake users can download the tool for free here.

It'll look similar to this:

Over on the right, you see where I've created the interactive surface normals pass. Those are groups with a few quickshapes and Mult nodes layered to create the interactive normals. In this case, since I didn't build out the ground and the barn next to the farmhouse, I chose to use the alpha from the interactive normals as the matte for the OW_OneLight+ node instead of the matte extraction I showed above. A smoothly blended surface for the ground, and a couple blurred normal shapes for the trees could make this quite convincing.

On the left, the fade leading into the over and the mix node to mix the whole thing back together is just one way of tweaking the levels of relighting that you're doing.

I posted a really short render of the light source moving across so that you can get an idea of what can be done. I kept it fairly subtle, and keep in mind that a little more work to create normals data for the ground, barn and trees would be required for a real project.

1. Build 3d geometry to match the visible elements
2. matchmove that geometry to match the footage
3. render a surface normals pass of those elements
4. use that rendered surface normals pass to relight the live footage.

In a future installment, I'll describe a method to generate surface-normals data by painting a bump map onto an image. In the meantime, I'll leave it as an exercise to the curious reader: how could you convert a simple greyscale height map into surface normals data in Shake?

In fact, if you google "Crowd Generation" there are two hits for my own websites before theirs comes up.

While DynaCyc is nowhere near as sophisticated for generating realistic behavior (it's more of a dynamic matte painting tool than a crowd generator - I created it for doing plants and trees to algorithmically make forests and such), it's pretty cool that I'm placing so well.

I really need to get the tutorial for my uv remapping tool recorded so that people can start using it. At the moment, though, I'm working 12-14 hrs a day, 6 days a week scripting some fancy animation pipeline tools.

The funny bit is something I've come to realize of the last couple days: while I fully grasp the minutiae of what I do, some of what I'm doing just baffles me in the big picture. I wrote a tool for "recasting" an animation from one space to another. For instance, if there's an object in a shot done during previs, and that original object has a particular animation on it, and there's a camera with a camera angle that we like... but the object itself is given a different animation in our scene because it has to fit in the "world" relative to other objects and be moving at a certain speed, reach a certain place at a certain time... I can take the camera motion from the original animation and transform it to the new shot maintaining the relationship between the camera and the element... while replacing that element's animation.

That's tricky... but I know how to make it happen and I've embodied it in a deceptively simple tool.

Separately, I worked out a method for stitching multiple animated shots containing a common object and many separate cameras together. The object's motion remains constant, and the cameras follow it along the path. Additionally, I developed a method for steering that element so that I can change its motion, adjusting the cameras to follow it.

That last one works using the first tool... so they end up all being connected to each other.

Once that happens, it feels like magic. I no longer understand the big picture: how we now have nearly a hundred separate shots stitched seamlessly together, but with different animation... and still working with the cameras and tracked objects matching to the footage? That starts to get away from me... I can conceptualize it as steps, as stages to reach a goal... I can break it down into its component parts: but the whole big process I understand only in that way: as big steps. I don't hold the whole thing in my head at once, though I use the tools that make it happen several times a day.

What's also funny is how much I'm even animating at this point... and I'm no animator, believe me. Let me write you a few expressions, though, and I can sometimes fake it.

It's mini-crunch, though, because we're just working to get things together for the trailer. Most of the staff isn't actually working overtime, but there's a few of us... the dynamics/fluid guys, an animator or two, the cg supe and me, piecing together the necessary bits to get the shots ready for the trailer.

I wish I could be a little more clear on what exactly it is I'm doing... When the trailer comes out in a couple weeks I can at least indicate then what segment of the film I'm addressing - I'd rather leave that part as a surprise for when the trailer hits (not to mention that talking about things like that would get me in hot water).

In a sufficiently vague sort of way, I can say that there's a segment I'm working on right now that's around 76 shots long, each ranging from 50-300 frames, that has a common element that's travelling between them. Our tracking team has been tracking that element, and its progress through each individual scene, for each of those 76 shots (and at least that many more as well that I'm not dealing with yet).

The particular task I've been working on is basically a continuity script. It's a MEL script that gathers all of that data: cameras, animated and matchmoved elements, and stitches them together end to end in a master shot. This way, the element that's moving through the scene will maintain an accurate location relative to the CG environment.

To coincide with what's happening in each shot, it's necessary on that big grandaddy master shot to make adjustments: turn something here, nudge something there... and that affects the path of that common element throughout all of the individual shots.

Do you see where I'm going with this?

Basically, if we imagine that I'm talking about a man driving a car across the salt flats in Utah (not at all what's happening in this film), there are features on the horizon that we want to be consistently and realistically moving in relation to the view of the car. If the driver swerves to miss a rock, we want to change the path he's taking across the salt flats (even though in the original plates, that driver would have been sitting in a stationary vehicle in front of a bluescreen, with some hydraulics underneath making the car tilt and sway to add realism). Our view of the car from the camera should stay the same... Car and camera must maintain their relationship... but the position, speed, orientation, etc of the car and camera combined will need to change. If there's a guy on one of those sail-carts out there, we need to make sure his path is realistic and that he's in the right place in the background of each shot...

Remember that the car, the driver, and the camera are often the only real things here... the sail-cart, other things on the horizon, in fact the entire salt flats... all are cg or augmented with cg.

So I've been developing tools for funneling data from all of these shots... everything from camera data to object matchmoves, world-space offsets for each shot, and animation data... stringing it together into a seamless master scene, manipulating the paths and locations of elements in that master scene and transferring that data back into the individual shots. If we need our hypothetical salt-flats driver to swerve this way and that across several separate shots, I'm making sure that it doesn't just "kinda look right" but that we can get that data integrated into the object and camera motion of several live camera plates.

If somebody didn't ask me if I could do it, I'd probably have never thought it could even be done.

If I'm kinda scarce around here in blogland, it's because I'm catching a much-needed nap in between long shifts.

I had a long time to wait on a couple renders that I'm using in the tutorial, so I kept tweaking the interface. The original version of the tool was created about 3 years ago - and the underlying code hasn't changed. I have gotten a little pickier about how my tools should work though. If there's a process that I'm going to want to do most of the time I use the tool (whether it's a Shake Tool, a MEL or Python tool in Maya, a Sizzle Script, etc, that process should be built into the tool itself. It saves time and annoyance and makes the tool complete.

As a result, I ended up with a tool I like even more. Unfortunately... I won't be posting the real update tonight. It's late, I haven't recorded the final version of the tutorial yet, and I still need to package the latest set of tools for download.