EddieOffermann dot com

Read this question on cgtalk tonight and thought I'd kick the question and my response out here since I've repeated much this same advice many, many times.

I use Boujou for matchmoving but am tracking 640x480 camera resolution all of the time... I often have shots with handheld very shaky camera and motion blur. Boujou has trouble tracking it. Is it just that there's to much blur/shakiness or would a higher camera resolution help?

and my response:

Well, it looks like there are a number of issues to overcome:

First, the quality of the footage: handheld/shaky/moblurred footage is just awful to work with. If you're just the artist, ie, you get what's handed to you: you work with what you have. There's no telling Mann, Fincher or Stone they have to reshoot because your job is hard. Their job is harder and it's them the client paid for, not you. If it's for your own projects, ie, you're the director or at least the vfx supervisor on a smaller project: you have to take this into your hands (so to speak) and fix it before it gets past the camera! Since you're working with 640x480 footage, I have to assume you can address it at this stage: these are obviously personal projects, not features or broadcast. Fix the source first! Don't shoot/allow to be shot footage that's going to suck to do post on!

Second, the resolution of the footage: Until you've worked in HD or film res, it's hard to appreciate just how much easier it is to work with on a tracking level. A dot that is so small it'll disappear into noise at NTSC is an obvious crosshair at 2k. Smudges on the wall and skin blemishes become usable tracking points. If you have the option to work with higher resolution footage, insist on it!

Third: Lighting. Again, this is only something you can influence if you're shooting your own projects, but remember to light for contrast not final levels. More light means faster shutter speeds means less motion blur. Light for the type of shadows you want, not how dark you want the scene when you're shooting for visual effects. If your key and fills are set properly, you can always make the shot darker when grading. This isn't as much of an issue when shooting film or deep color digital like the Arri Digital cameras since their shadow detail is good enough to boost for tracking detail. But if you're shooting with consumer or prosumer cameras, you've got jack but noise in the shadows and the low light levels will mean excessive blur and bad noise overall.

Fourth: Manual vs Auto. Boujou has a reputation as being the fire-and-forget solution for people that don't understand tracking. The really hard shots are going to need manual solutions. I don't mean hand-keyed (necessarily) but we're talking careful selection of usable tracking detail and improved setting of constraints. Personally, I recommend SynthEyes - unbelievably good price point and an unbeaten feature set. I know there are shots that people say they can click a button and get a solve on in Boujou that they can't get in SynthEyes, but the reverse is true far more often than it's not. When the footage is hard to work with, throw away the automatic solutions. Give them one run at it if you've got the time and processor power to spare, then move to something where you have some control.

Fifth: Tracking techniques. Track the center, not the edges. When doing a supervised track of moblurred features, follow the center of those blurs.

Sixth: Rendering techniques. Moblurred footage is hard to render to match because what you have to match is a particular slice of time. Think of it this way: you're placing tracking points on each frame, generally spaced apart at the rate of 24, 25 or 30 frames per second. Between frame 1 and frame 2, there's a fraction of a second where anything could happen. The camera could jump up a couple centimeters and back down for instance, before hitting frame 2. This will show in the motion blur but not in the keyframes. You can't easily fix for this! (See tip 1). More often, though, the problem is the phase of the render. How to address this will depend on your renderer. You may be able to adjust this in the renderer, you may have to adjust it in the actual animation keyframes depending on what 3d software you're working from. Basically, you need control of the shutter timing and shutter offset: when in that frame it opens and how long it stays open.

For high end vs. low end facilities, #6 can often be the difference between them. For all the skill you can buy from freelancers in tracking, modeling, animation, textures, etc: having the resources to do test renders and tweak that setting (and having someone on staff who can make sure you're addressing this) makes all the difference.

photo credit: Capture Queen ™I have a side project for iPhone that I'm working on. There's a patent filing in process, incorporation paperwork to do, all that good stuff - so I won't be doing a lot of specifics about the project until it's ready to go. It's basically a location-based-services thing but when we've looked over what everyone seems to be doing with LBS and social networking and the like, it all seems to be thinking very inside the box. Just like a few years ago there was a rash of "... on the internet!" patents where things we'd done forever in the wetworld were being done online and called "visionary", now there's a rash of similar "... on a mobile phone!" patents and websites that are no more groundbreaking than when we did them online or on our feet. It's like patenting driving in nails with the side of a hammer. It works, but it's not especially groundbreaking nor is it even ideal.

We really feel that this project is going to move mountains though, and change the way people view the real world around them, not just when they're in front of a computer.

The tendency towards developing for the iPhone is that the iPhone has arguably the best SDK there is: and a substantial and rapidly growing marketshare. Still, though, that limits our deployment: it's the largest selling smartphone-style device, but it doesn't represent the majority of the market. No-one does.

Today I've been evaluating Mojax, a cross-platform mobile development environment. It looks promising. It promises cross-platform compatibility, including access to a number of necessary core services like GPS, and currently runs on all phones supporting J2ME (Java on Mobile), all color Blackberry phones, and shortly Windows Mobile devices from WinMo 2003 onward, any mobile running Brew V2 or later, and Helio devices. Just the J2ME support gets me onto most smartphones - so this effectively would get the product onto every mobile device there is with gps capability.

This excites me because I like the prospect of running the application on every feature-capable mobile phone without having to separately develop for Blackberry, HTC, Samsung, and Nokia. Getting the product out simultaneously for iPhone, Windows Mobile, PalmOS and Blackberry would be fantastic. There are some annoyances with using J2ME midlets, but I'll take the slight user experience tradeoff for being able to deliver an application at all without breaking the bank

So I'm working on a side project that's mainly just for me, but it's so damn handy I might just make it publicly available when it's done (probably for a small monthly fee). It's basically a super quick way to run certain types of scan for stocks during the day, picking out data that's usually hard to get until the end of the day. I'm tying it into the OptionsXpress API so that when I identify a good entry or exit point on a stock, I can change my position with the click of a button.

I've been designing the site to automatically rescale for my mobile phone, but I need a way to determine if I'm looking at it on the phone or on a desktop - and since my mobile browser is generally configured to identify as a desktop browser (so that I don't get sent to the "mobile version" of the sites I visit) it's harder to do. My browser does a great job of rendering full-screen pages and scaling them - if you've seen the iPhone browser, both Opera Mini and Pixsel do something similar though without 'multi-touch'.

That said, there's something to be said for designing a website that will gracefully scale when it's on a phone.

But when the browser identifies itself as a desktop browser, what's left? Javascript can pick up the screen size, but what about PHP? All my pages are created in PHP...

(more...)

I was talking to a friend today about what I'm doing with regards to managing data through an animation pipeline using XML. The more I work with it and the farther I get into the project, the more flexible and powerful the whole thing seems. Of course the goal to doing the implementation in Python is that virtually every software package in the vfx industry is python-friendly - so once the core routines are written, everything from Nuke and pyShake (the python plugin for Shake - if you haven't seen it yet, check it out here) to Maya, Houdini and RealFlow will be able to make use of them. I think most places are doing that these days, with a few nods to TCL/tk here and there - but broadly supported scripting languages are King and open description formats like XML are Queen.

My friend marveled at how nice it would be if one day, a couple years from now, everything was able to talk that smoothly: that a character animated in Maya could be pulled into Houdini, for instance, as something other than an OBJ sequence or a separately rigged character that you had to tediously (or with a lot of specific coding) link to exported channel data.

I wonder if that interoperability thing will ever extend beyond each individual studio's implementation. Everybody has a way of getting software to talk amongst themselves, some solutions being more elegant than others, but when you invest in creating something as elaborate as this it becomes your own proprietary tool. If you develop a tool that an animator can take an animated character with a complex rig on it, arbitrarily select additional elements that were never *really* meant to be animated and animate them anyway, and the modeling team can modify the model and issue a new version of it - and the animation gets seamlessly transferred over to the new model, even able to be read into RealFlow, substituting a different set of low poly independent objects that are driven by the data in that XML file: you don't put that pipeline tool on the internet for everyone to download for free.

That tool becomes your secret weapon. As a studio with an investment in a powerful and unique proprietary tool, even charging for it may not mean as much to you as the edge you gain during the heat of production.

Being XML based and implemented in Python does put my current project a wee bit closer to being an open standard, though. Even Shake will take Python scripts now - and they're really powerful in it and getting more so as development continues. The readability thing for XML is a gigantic plus, and the way it represents data is great. I can build a module that will write out the translation of a locator in both world and local space, as a baked set (every frame has a value) and as a set of keyframes (values only for those frames where the value was explicitly set by the artist), as well as screenspace UV values - so the same XML file could reconstruct a scene for a lighter to light and render from or another animator to tweak the animation curves, or for RealFlow to drive low-poly proxy objects with to disturb a drifting mist, or for a compositor in Toxic to link an effect to. And it's all one XML file - not a half dozen formats (often multiple versions of each) and a hundred-unit sequence of geometry exports.

I don't know where I read that, but it amused me and stuck in my head.

Hi!  I haven't been updating the vfx blog because I've been crazy busy.  Work has me deep in MEL and Python code most of the day, writing software to manage animation data, shuttle it this way and that and process it in interesting ways.  There's also the secret project that everyone isn't talking about but is quietly and excitedly waiting for (assuming they have any idea what it is).   We're on super-special-secret-lockdown at work, I'm guessing for the rest of the month.  I can scarcely go to the bathroom or grab a soda without passing through at least two security doors now.

Ok, I admit it, we've been contracted to create the lost tapes of the lunar landing.  Seems NASA has gotten enough flack about the original landing with people claiming they can prove it wasn't real that they've decided to create more proof to put the concerns to rest.

Or at least that's my amusing cover story until I'm allowed to talk about the other thing.

Between all the other little tasks, I'm in the process of working up a new tool for moving complex animation data around the pipeline.  The idea is to be able to read and write an XML file that contains as much data as anyone would care to put onto a model.  The model can be updated, have new elements added to or removed from it, all sorts of parts of that model can be animated at a whim by the artist and passed along to subsequent versions of the model, low res or high res versions, keeping all of the data intact, allowing for scaling of the animation data if it's transferred to a different model entirely, even passing along worldspace location and orientation of parts of the model (and maybe even eventually the full animation) to other platforms like Houdini and Realflow.

It's a little important for what we're doing on The Curious Case of Benjamin Button, but the existing tools I developed are working ok there (even if they're still a little limited and occasionally awkward - they're getting better on a weekly basis).  There's already a small test of what I'm doing in a broader sense: we have a particular element with a moderately complex rig on it that I'm publishing and subscribing the animation data on.

The Big Goal for this tool will be on T4 which we should be working on soon.  That'll be a huge show and we've got some really epic-scale work on it: most likely we'll be needing the more sophisticated animation tools for managing our data throughout the pipeline - so that's what I constantly keep in the back of my mind as I script the current version of the tools.  They need to serve the current project in the most complete way possible, and they need to be able to expand to do more than I'd currently dream of asking them to do.

It's a lesson that's come up recently because of a little tool I wrote that I've mentioned here before "CurveBaby" - CurveBaby allows me to take a scene with an animated camera and one or more animated objects, preserve the camera's relationship to one of those objects but change that object's animation - or, conversely, reanimate the camera and have the animation of the object automatically adjust to still look the same from the camera.  It's a powerful tool, and one that I can barely imagine how I'd get through half a day without using it at least once.

Big tools make big projects go a lot more smoothly.

A few weeks ago, I posted a video covering the process of relighting rendered CG elements in a compositing tool like Shake using a surface normals pass.

Since that covered normals-based relighting of rendered cg elements, I thought I'd write an article that takes the concept a little farther and should stretch the reader's mind on what's possible with this sort of approach.

The earlier tutorial covered something that naturally can be accomplished by going back to the original scene in your 3d application, adjusting lighting and re-rendering: but I think there are times when we want to adjust the lighting primarily in the compositing application but want more than some masked-off gamma correction, lifts and crushes. So it's handy, then, to be able to apply an effect more similar to real-world lighting and that's what a rendered surface normals pass will enable us to do.

But what about real world imagery? Can this same technique be applied to live action footage just as it can be on a CG element?

Absolutely!

You'll want to start with a few elements like the following:

A full-color image like the above picture of a farmhouse. Under ideal conditions, you'd have an image with very diffuse lighting: something with a lot of bounce light or an exterior on an overcast day. The idea here is that there are relatively few real shadows in the image. The reality is that you won't have that very often and you'll have to use what you've got. You may decide that the shadows that do exist need to be painted out before you continue. I haven't done that for this example, but you'd be hard pressed to find a high end effects project that relit a real world environment and didn't remove the existing shadows at some point in the pipeline.

The above image is a matte of the farmhouse pictured above. You'll probably need to generate something like this for your scene if you haven't already.

A surface normals pass of a polysphere with its shading set to be as unsmooth as possible. I prefer this because I have a better feeling of where these surfaces are facing than I do straight from a sphere. It makes it like a special kind of color palette so I can perform the next step:

Next, I roto out a super-simple normal map of the surface in the image. This can be more or less complicated, depending on how much detail you want in the relight. If the surfaces are softer, you may want to blur the resulting image some: this will soften the transitions. Pick colors for each of your surfaces from the faces of the polysphere normal pass, making sure to keep parallel faces the same color as you interactively create a normal pass for your real-world element.

Then, I set up the OpaqueWhite OneLight Plus node as described in the normals relighting video here. Shake users can download the tool for free here.

It'll look similar to this:

Over on the right, you see where I've created the interactive surface normals pass. Those are groups with a few quickshapes and Mult nodes layered to create the interactive normals. In this case, since I didn't build out the ground and the barn next to the farmhouse, I chose to use the alpha from the interactive normals as the matte for the OW_OneLight+ node instead of the matte extraction I showed above. A smoothly blended surface for the ground, and a couple blurred normal shapes for the trees could make this quite convincing.

On the left, the fade leading into the over and the mix node to mix the whole thing back together is just one way of tweaking the levels of relighting that you're doing.

I posted a really short render of the light source moving across so that you can get an idea of what can be done. I kept it fairly subtle, and keep in mind that a little more work to create normals data for the ground, barn and trees would be required for a real project.

Moving footage naturally requires animated roto work to generate the interactive surface normals pass (keep the colors the same unless the element itself is moving!), and more complex objects with broader camera moves can be relit if you:

1. Build 3d geometry to match the visible elements
2. matchmove that geometry to match the footage
3. render a surface normals pass of those elements
4. use that rendered surface normals pass to relight the live footage.

In a future installment, I'll describe a method to generate surface-normals data by painting a bump map onto an image. In the meantime, I'll leave it as an exercise to the curious reader: how could you convert a simple greyscale height map into surface normals data in Shake?

I noticed the other day that my own 2d crowd generation software tool (DynaCyc) is ranking higher on google than Massive Software's listing for their incredible 3d crowd sim software.

In fact, if you google "Crowd Generation" there are two hits for my own websites before theirs comes up.

While DynaCyc is nowhere near as sophisticated for generating realistic behavior (it's more of a dynamic matte painting tool than a crowd generator - I created it for doing plants and trees to algorithmically make forests and such), it's pretty cool that I'm placing so well.

I really need to get the tutorial for my uv remapping tool recorded so that people can start using it. At the moment, though, I'm working 12-14 hrs a day, 6 days a week scripting some fancy animation pipeline tools.

The funny bit is something I've come to realize of the last couple days: while I fully grasp the minutiae of what I do, some of what I'm doing just baffles me in the big picture. I wrote a tool for "recasting" an animation from one space to another. For instance, if there's an object in a shot done during previs, and that original object has a particular animation on it, and there's a camera with a camera angle that we like... but the object itself is given a different animation in our scene because it has to fit in the "world" relative to other objects and be moving at a certain speed, reach a certain place at a certain time... I can take the camera motion from the original animation and transform it to the new shot maintaining the relationship between the camera and the element... while replacing that element's animation.

That's tricky... but I know how to make it happen and I've embodied it in a deceptively simple tool.

Separately, I worked out a method for stitching multiple animated shots containing a common object and many separate cameras together. The object's motion remains constant, and the cameras follow it along the path. Additionally, I developed a method for steering that element so that I can change its motion, adjusting the cameras to follow it.

That last one works using the first tool... so they end up all being connected to each other.

Once that happens, it feels like magic. I no longer understand the big picture: how we now have nearly a hundred separate shots stitched seamlessly together, but with different animation... and still working with the cameras and tracked objects matching to the footage? That starts to get away from me... I can conceptualize it as steps, as stages to reach a goal... I can break it down into its component parts: but the whole big process I understand only in that way: as big steps. I don't hold the whole thing in my head at once, though I use the tools that make it happen several times a day.

What's also funny is how much I'm even animating at this point... and I'm no animator, believe me. Let me write you a few expressions, though, and I can sometimes fake it.

I am officially in crunch...

It's mini-crunch, though, because we're just working to get things together for the trailer. Most of the staff isn't actually working overtime, but there's a few of us... the dynamics/fluid guys, an animator or two, the cg supe and me, piecing together the necessary bits to get the shots ready for the trailer.

I wish I could be a little more clear on what exactly it is I'm doing... When the trailer comes out in a couple weeks I can at least indicate then what segment of the film I'm addressing - I'd rather leave that part as a surprise for when the trailer hits (not to mention that talking about things like that would get me in hot water).

In a sufficiently vague sort of way, I can say that there's a segment I'm working on right now that's around 76 shots long, each ranging from 50-300 frames, that has a common element that's travelling between them. Our tracking team has been tracking that element, and its progress through each individual scene, for each of those 76 shots (and at least that many more as well that I'm not dealing with yet).

The particular task I've been working on is basically a continuity script. It's a MEL script that gathers all of that data: cameras, animated and matchmoved elements, and stitches them together end to end in a master shot. This way, the element that's moving through the scene will maintain an accurate location relative to the CG environment.

To coincide with what's happening in each shot, it's necessary on that big grandaddy master shot to make adjustments: turn something here, nudge something there... and that affects the path of that common element throughout all of the individual shots.

Do you see where I'm going with this?

Basically, if we imagine that I'm talking about a man driving a car across the salt flats in Utah (not at all what's happening in this film), there are features on the horizon that we want to be consistently and realistically moving in relation to the view of the car. If the driver swerves to miss a rock, we want to change the path he's taking across the salt flats (even though in the original plates, that driver would have been sitting in a stationary vehicle in front of a bluescreen, with some hydraulics underneath making the car tilt and sway to add realism). Our view of the car from the camera should stay the same... Car and camera must maintain their relationship... but the position, speed, orientation, etc of the car and camera combined will need to change. If there's a guy on one of those sail-carts out there, we need to make sure his path is realistic and that he's in the right place in the background of each shot...

Remember that the car, the driver, and the camera are often the only real things here... the sail-cart, other things on the horizon, in fact the entire salt flats... all are cg or augmented with cg.

So I've been developing tools for funneling data from all of these shots... everything from camera data to object matchmoves, world-space offsets for each shot, and animation data... stringing it together into a seamless master scene, manipulating the paths and locations of elements in that master scene and transferring that data back into the individual shots. If we need our hypothetical salt-flats driver to swerve this way and that across several separate shots, I'm making sure that it doesn't just "kinda look right" but that we can get that data integrated into the object and camera motion of several live camera plates.

If somebody didn't ask me if I could do it, I'd probably have never thought it could even be done.

If I'm kinda scarce around here in blogland, it's because I'm catching a much-needed nap in between long shifts.

Spent most of the day today working on my website, building another tutorial, and refining the uv-pass retexturing tool.

I had a long time to wait on a couple renders that I'm using in the tutorial, so I kept tweaking the interface. The original version of the tool was created about 3 years ago - and the underlying code hasn't changed. I have gotten a little pickier about how my tools should work though. If there's a process that I'm going to want to do most of the time I use the tool (whether it's a Shake Tool, a MEL or Python tool in Maya, a Sizzle Script, etc, that process should be built into the tool itself. It saves time and annoyance and makes the tool complete.

As a result, I ended up with a tool I like even more. Unfortunately... I won't be posting the real update tonight. It's late, I haven't recorded the final version of the tutorial yet, and I still need to package the latest set of tools for download.

As I'd mentioned before, this blog is going to get more technical and I'll be spawning off a personal blog shortly. For now, bear with me non-technical readers: I'm about to talk geek for a bit. When I was teaching at FullSail, a lot of my "curriculum development" time, as well as a substantial portion of my free time, was spent doing R&D on advanced topics - things that are done at a handful of visual effects studios around the world, or even things that nobody was doing yet except in university settings.

Things like relighting using what's called a bent normals pass (a special render pass that contains information about the direction a point on the face of an object is pointing) - a process that enables you to gain a fair bit of control over how the rendered object appears to be lit, long after the frames have left the render farm. Other experimental tools like an algorithmic forest and crowd generator, or today's topic: retexturing in the comp, instead of at rendertime.

There are a couple of tools for doing this already: basically, you render out a diffuse color pass, full intensity colors, no shading, but instead of finished shaders, the elements are textured with these especially bright and colorful red/green gradients. The color of each pixel corresponds to its relative location on the UV map, so even after the texture is wrapped around an object and that object viewed from an arbitrary camera position, each pixel still holds information about its original position in UV texture space.

Now, most of the tools I've seen for this sort of thing will apply one texture to one UV pass. Certainly an exciting demonstration, but hardly practical: a typical scene will often have dozens of texture maps at the very least, frequently hundreds, even thousands. Naturally, each of these may be rendered as a separate UV pass, and each separate element brought into the comp: but that can make for some pretty unwieldy scenes and a headache to maintain all those passes in the pipeline. Re-render an animation, and the number of potential UV-passes can quickly make each iteration into a big undertaking.

Of course, if you're dealing with 8-bits per channel UV-coordinate textures, like a typical JPEG, then your possible texture size to project onto that is fairly small. 256x256 to be exact, and for practical reasons, maybe only about half of that. Hardly anyone even uses textures that small in games these days, let alone feature films. If you crank those UV-coordinate textures up to 16-bpc, now you wind up with some pretty impressive texture space: each of the red and green channels can now represent up to 65,536 coordinates: the equivalent of 16 4096-pixel-wide textures. Even if we want to be similarly conservative like in our above estimate, you can see where I'm going with this...

So my tool, which I call "OpaqueWhite ReMap Plus" uses nice, big UV coordinate textures, all generated by your rendering application in 16 or 32 bits per channel, rendered as described above as an unshaded UV-coordinate pass and is designed to accept up to 16 separate textures at a time, anywhere from 256x256 to 8192x8192. I provide a collection of high-res, high-bit-depth UV-coordinate textures for users to set up their render with, and a tool that is able to interpret the data from that render pass and automatically re-plot the texture in the render-specific UV space.

There's a lot more stuff going up on the site shortly. It's ALMOST THERE, I swear. To those that might be interested, you can learn a little more about and (soon) download my Shake Tools here.