EddieOffermann dot com

A few weeks ago, I posted a video covering the process of relighting rendered CG elements in a compositing tool like Shake using a surface normals pass.

Since that covered normals-based relighting of rendered cg elements, I thought I'd write an article that takes the concept a little farther and should stretch the reader's mind on what's possible with this sort of approach.

The earlier tutorial covered something that naturally can be accomplished by going back to the original scene in your 3d application, adjusting lighting and re-rendering: but I think there are times when we want to adjust the lighting primarily in the compositing application but want more than some masked-off gamma correction, lifts and crushes. So it's handy, then, to be able to apply an effect more similar to real-world lighting and that's what a rendered surface normals pass will enable us to do.

But what about real world imagery? Can this same technique be applied to live action footage just as it can be on a CG element?

Absolutely!

You'll want to start with a few elements like the following:

A full-color image like the above picture of a farmhouse. Under ideal conditions, you'd have an image with very diffuse lighting: something with a lot of bounce light or an exterior on an overcast day. The idea here is that there are relatively few real shadows in the image. The reality is that you won't have that very often and you'll have to use what you've got. You may decide that the shadows that do exist need to be painted out before you continue. I haven't done that for this example, but you'd be hard pressed to find a high end effects project that relit a real world environment and didn't remove the existing shadows at some point in the pipeline.

The above image is a matte of the farmhouse pictured above. You'll probably need to generate something like this for your scene if you haven't already.

A surface normals pass of a polysphere with its shading set to be as unsmooth as possible. I prefer this because I have a better feeling of where these surfaces are facing than I do straight from a sphere. It makes it like a special kind of color palette so I can perform the next step:

Next, I roto out a super-simple normal map of the surface in the image. This can be more or less complicated, depending on how much detail you want in the relight. If the surfaces are softer, you may want to blur the resulting image some: this will soften the transitions. Pick colors for each of your surfaces from the faces of the polysphere normal pass, making sure to keep parallel faces the same color as you interactively create a normal pass for your real-world element.

Then, I set up the OpaqueWhite OneLight Plus node as described in the normals relighting video here. Shake users can download the tool for free here.

It'll look similar to this:

Over on the right, you see where I've created the interactive surface normals pass. Those are groups with a few quickshapes and Mult nodes layered to create the interactive normals. In this case, since I didn't build out the ground and the barn next to the farmhouse, I chose to use the alpha from the interactive normals as the matte for the OW_OneLight+ node instead of the matte extraction I showed above. A smoothly blended surface for the ground, and a couple blurred normal shapes for the trees could make this quite convincing.

On the left, the fade leading into the over and the mix node to mix the whole thing back together is just one way of tweaking the levels of relighting that you're doing.

I posted a really short render of the light source moving across so that you can get an idea of what can be done. I kept it fairly subtle, and keep in mind that a little more work to create normals data for the ground, barn and trees would be required for a real project.

Moving footage naturally requires animated roto work to generate the interactive surface normals pass (keep the colors the same unless the element itself is moving!), and more complex objects with broader camera moves can be relit if you:

1. Build 3d geometry to match the visible elements
2. matchmove that geometry to match the footage
3. render a surface normals pass of those elements
4. use that rendered surface normals pass to relight the live footage.

In a future installment, I'll describe a method to generate surface-normals data by painting a bump map onto an image. In the meantime, I'll leave it as an exercise to the curious reader: how could you convert a simple greyscale height map into surface normals data in Shake?

1. 
   SofusGraae said...
   12:07 am - May 11th, 2008

   Brilliant – excellent information you have here (and the rest of the site as well) – A sure bookmark

Name (required)

Mail (will not be published) (required)

Website