EddieOffermann dot com

As I'd promised earlier, I'll be doing a quick run-through of how matrix multiplication can be used to rotate, scale and translate coordinates through 3d space. First, though, I'm going to write a little bit about a tool I've been working on and just launched. Sorry: the actual code for it is proprietary, so I can't share it, but a brief discussion of the tool and what it needed to be able to do will help illustrate why matrix transformation is so important, even when you're not creating a 3d package from scratch or developing a game engine.

I needed a way to quickly create a usable "right eye" camera to correspond to a "left eye" camera. I needed to be able to specify interocular distance (the distance between the left and right eye), as well as the convergence distance (where the eyes were focusing). Our eyes naturally "converge" on objects that are in front of us - this is part of how we judge relative depth as this convergence affects the apparent relationship if the dual images our eyes perceive of *other* objects that are not at the point of convergence. But that's a subject for another entry.

Anyway, the tool needed to be able to take a tracked "left eye" camera and apply that tracking data to generate the "right eye" camera using convergence and interocular information that was either derived from a track, noted on set, or dynamically assigned in Maya. I wanted a right eye camera that could be created instantly and that would automatically follow any changes I made to the left eye camera. If I smoothed that camera's path, I needed the right eye to follow suit, if I performed some dramatic transformation to that camera's path, perhaps extending its animation or re-animating an object that was part of the track while conforming the camera to that object's new path: I needed the right eye camera to keep up. If a cg element was being added, moving toward the camera, I needed to be able to selectively lock the convergence to "look at" that object's depth without diverting the cameras to actually look directly at it - similar to changing the focus depth.

It's all about control and instantaneous response, with a camera whose path is known with some degree of certainty and a second camera that will follow that one in a way that will produce realistic cg and will enable artists to quickly and accurately duplicate the second camera position when the original interocular distance and convergence information is not known.

Matrix multiplication, as a mathematical operation, isn't complicated - it's simple multiplication and addition, just repeated a number of times to generate the required result matrix. Since there are already more-than-ample tutorials online about how to actually *perform* matrix math (if you were cursed to do it by hand or develop procedures to support it in programming environments that don't already have matrix math support) I won't be covering that in detail. If my simple explanation doesn't quite do it for you, put "matrix multiplication" into Google if you're stumped.

I'm also going to avoid covering application in a specific language. Most recently, I was doing this in MEL for a project I'll discuss in a moment, but there are Python libraries for doing matrix math, TCL/tk support for it, PHP and Perl support - you'll rarely find yourself with no built-in or easily-added matrix support, though you may want to write (as I did) a number of routines to make it a little more accessible.

To the left, you can see the standard format of a translation matrix. This matrix (when multiplied by a coordinate matrix as represented on the right-hand side of the equation) will translate that coordinate into a new space, offsetting it by (tx, ty, and tz). In the typical way of matrix multiplication, the element in each row of the coordinate matrix [x y z 1] is multiplied by a column of the translation matrix, as shown here:

The rows of this matrix are then added together (in typical matrix multiplication fashion) to give the resulting (x',y',z') location.

But there's more to matrix transformation than simple translation. If you wanted to find the new (x', y', z') for a translation like this one, simple addition would be enough. But manipulating points in 3d space is rarely that simple!

Fortunately, it's just as easy to scale a point using matrix math! Always away from the origin - we'll talk about how to scale a point away from somewhere other than the origin shortly. All matrix operations work with a similar setup, so you'll get used to seeing a similar notation here.

This matrix is multiplied in the same fashion as the one we see above, rows against columns, with row sums producing (x',y', z') for the new location.

I'll admit, though. I don't use matrix math for scaling things very often. You know what I do use it for, though?

Rotating a point in space with respect to the origin! Isn't that exciting? I LOVE rotating things! Well, ok, so it's not that big of a deal - but when we start combining some of these things, it can turn a complicated object tree in your scene into a relatively simple expression.

I have to warn you, though, rotation is a bitch. Ever notice how your favorite 3d software has this whole "rotation order" thing? That's because if you rotate something 10 degrees in X, then 15 degrees in Y, it's not the same as rotating it in Y first and then in X. There's also a different matrix for each axis of rotation, so let's take a look. Same matrix math process as translation and scaling, but from this one we get rotation around the Z axis.

And now here:

That one rotates in X! And lastly, as you'd expect, there's a matrix for rotating around the Y axis:

Now, combining these matrix transformations together is as simple as multiplying the matrices with each other. Now, you build the series of multiplications in order from right to left, but they're carried out from left to right. For instance, for "zxy" rotation order, you would create an expression similar to the following: (I'll use MEL for this example: $xRot, $yRot, and $zRot are each 4x4 matrices that already contain transformation data for x, y and z rotation):

matrix $r[4][4] = $yRot * $xRot * $zRot;

Provided proper matrix variables are supplied, MEL supports basic matrix operations with some limitations that you'll find as you stretch your legs.

These can be strung together into much longer expressions generating much more complex matrices. To rotate an object about a point other than the origin, for instance, subtract the values of that coordinate from the coordinates of the object (transform it in {-cx, -cy, -cz} where {cx,cy,cz} was the center to rotate it around), perform the rotation, the transform it back {+cx,+cy,+cz}.

It may take some time learning to visualize and plan out a complex matrix transform - but what makes it powerful is the ability to combine any number of transformations into a single operation. Some basic trigonometry and a well-applied matrix transform can accomplish all kinds of things!

Matrix math is a bitch. And it's nigh impossible to find any good resources online. Every "transform matrix tutorial" or "matrix multiplication lesson" online explains in detail how to multiply matrices together.

That hasn't been my problem for about 25 years now. I get it. This is how you multiply a matrix with a vector. This is how you multiply two matrices together. Neither of those things actually *accomplishes* anything if you don't know what the data in those matrices represents or what adding, subtracting or multiplying them would do.

My problems always center around something more like this: I have a point at <10,20,30> with orientation <15,30,45>. Where is the point if I rotate it 15 degrees in y then translate it 100 units in the X axis?

What's screwed up is that I've so frequently solved this by doing trig that I know off the top of my head that pi/180 is estimated at 57.29577951. Some people pride themselves on the number of digits they know pi to: I find the radian conversion number to be far more handy. But for reference, pi to me is 3.14159265. Note the 8 significant digits behind the decimal number: can you tell what decimal significance I grew up programming in?

The thing is, I'm working on a number of tools for managing stereo cameras in Maya. I need to quickly determine from the transforms of one camera (derived from a track of live footage) the location and orientation of a second camera if we know (or can estimate or otherwise determine) the interocular distance (the distance between the eyes) and the convergence (how far in front of these cameras their line of sight intersects). Determining the angle of the second camera relative to the first is a straightforward bit of trig: think Pythagoras and arctangents and it'll come to you, or go find a good trig reference online - this part's easily found in a million places. Diagram that out for yourself, making an equilateral triangle with cameras at each base corner with convergence being the length of a line bisecting the triangle into two right triangles. If it's not immediately clear, work on it a bit: it might wake up parts of your brain that haven't seen the light of day since highschool.

The tricky bit is figuring out where a right-eye camera would be if it was offset along a line that isn't quite perpendicular to the left-eye camera.

I'm very nearly there - based on a given camera, I can find a point offset along that camera's X axis and apply a convergence factor to the resulting camera location by canting it back in along the Y. But rotating a known transform matrix by an additional value in the y axis *before* doing that x translation... still fussing with that bit. I can make that happen less elegantly - by actually transforming nulls through space in Maya - but I'm searching for the magical transformation matrix that'll make it happen in a simple series of math statements.

It's late. It'll probably be plenty clear to me in the morning.

UPDATE: Got it! Sleep did wonders - my tired brain just didn't quite get the matrix arrangement right before.  I'll write an entry this weekend covering concatenated matrix transforms.  Damn handy.  Damn powerful.

Installing pyShake on the laptop today.

It's every bit as painful as it was when I installed it on my old desktop machine.  Maybe a liiiiiittle bit better since at there's a configure for it now, but installing prebuilt binaries really spoils you.

If you've never installed a traditional unix/linux/macosx application,  you're in for a treat. It's one thing if it's your own application, as well, since you'll know what's going on: but when it's someone else's...

Well, you never *really* know what any application installer is doing.  Data gets socked away hither and thither, but mostly you don't know about it.

Doing a configure/make/make install sequence usually displays TONS of data.  There will be a thousand lines of successful completions, another thousand of warnings of varying severity (with usually alarming-sounding messages that may or may not be important - and very frequently aren't), and maybe even a few outright failures that *also* may or may not be important.

A truly great build will usually ease your mind a little.  Several of the warnings you get for things like boost::python will reassure you that the error you're seeing happens to nearly everyone and shouldn't alarm you.

Somehow, that doesn't make me feel entirely at ease. It's like your girlfriend telling you, "It's ok, honey. It could happen to anyone."

Then there are errors like this:
Please ensure that LDFLAGS is configured properly, or specify --with-boost-root

Thanks.  I'll do that.

I just read that Ubisoft bought Hybride Technologies - a large video game company buying a medium-size visual effects house.  The article had a lot of talk about the future of video games and movies and seemed to miss the point when it quoted recent reviews of "Beowulf" that said it was like a "long video game scene".

The point was that wasn't a positive statement...

But it got me thinking: there's been so much talk about how video games will become as good as movies - and it's a pet peeve of mine that writers can be so ignorant.

What??? You say?

Sure.  Video games will eventually look as good as what you see when you go to see Hellboy this weekend.

But when that happens, movies won't look like that any more.

Now eventually, and we're not talking in the next year or two, we're talking a decade down the road perhaps, video games will start to look a lot like what you see when you walk outside (except of course that there will be zombies or terminators in the video game and may or may not be killer cyborgs and diseased undead prowling your real-life neighborhood).  By then, feature films may well be predominantly 3d - but your video games probably won't be.  Another five or ten years and then maybe the playing field will be levelled.  Except that the movie theaters will have vibrating chairs, a better sound system than you're likely to have at home, misters and blowers and scent generators, actuators to lift and tilt the chairs.  Oh - and even better 3d and higher res displays than consumer televisions can deliver (what, you thought HD was the best it got?  Please.  Not even today.  But don't count on a 4k home television standard any time soon.)

Oh, and their screens will always be bigger than yours.

The constant talk about video games 'replacing' movies on some level just sounds like people talking 100 years ago about radio replacing books.  The internet hasn't even replaced books - though it has put printed encyclopedias out of business, but then how many people ever kept a consistently updated set of encyclopedias in their homes?  If you ever even owned a set, chances are you owned one that still talked about the USSR and the apartheid government of South Africa.  If the internet replaced *that* bookshelf dominating monstrosity, I'd say that's not even surprising.  Public libraries had been replacing private encyclopedias for decades already.

Ubisoft's purchase of Hybride is motivated by the same thing that's making companies like Digital Domain say they want to get into video game production: fear. The fear that they won't be prepared if things go that way.  The fear they're going to be missing out on something.  They see something like the release of the new GTA and it looks like video games are the road to making billions.  It's just not true, any more than a small multimedia house in New Orleans should look to Wall•E as the future of their company.  Ubisoft doesn't need Hybride to understand how to make better CG. And the guys at Hybride probably can't help much with getting their ideas into the game engine: because that's an entirely different discipline.

My point is, video games will change.  But I don't think they'll "converge" with feature films any more than television "converged" with feature films.  Television is still a poor substitute for going to the movies and as much as we make the decision to sometimes catch something when it comes out on television/pay-per-view/dvd/bluray/what-have-you, as a society the rise of television has done nothing to diminish movie attendance.  Now we just expect more entertainment, and predominantly entertainment that we can enjoy just by sitting on our ass with a bucket of popcorn in our lap or a plate of nachos on the coffee table and no video game controller anywhere in site.  A theater full of people aren't going to be strumming their Guitar Hero guitars or thrashing around with their Wii controllers, and next year's Batman installment isn't going to require you to learn to drive the batmobile.

Unless of course you want to.

Read this question on cgtalk tonight and thought I'd kick the question and my response out here since I've repeated much this same advice many, many times.

I use Boujou for matchmoving but am tracking 640x480 camera resolution all of the time... I often have shots with handheld very shaky camera and motion blur. Boujou has trouble tracking it. Is it just that there's to much blur/shakiness or would a higher camera resolution help?

and my response:

Well, it looks like there are a number of issues to overcome:

First, the quality of the footage: handheld/shaky/moblurred footage is just awful to work with. If you're just the artist, ie, you get what's handed to you: you work with what you have. There's no telling Mann, Fincher or Stone they have to reshoot because your job is hard. Their job is harder and it's them the client paid for, not you. If it's for your own projects, ie, you're the director or at least the vfx supervisor on a smaller project: you have to take this into your hands (so to speak) and fix it before it gets past the camera! Since you're working with 640x480 footage, I have to assume you can address it at this stage: these are obviously personal projects, not features or broadcast. Fix the source first! Don't shoot/allow to be shot footage that's going to suck to do post on!

Second, the resolution of the footage: Until you've worked in HD or film res, it's hard to appreciate just how much easier it is to work with on a tracking level. A dot that is so small it'll disappear into noise at NTSC is an obvious crosshair at 2k. Smudges on the wall and skin blemishes become usable tracking points. If you have the option to work with higher resolution footage, insist on it!

Third: Lighting. Again, this is only something you can influence if you're shooting your own projects, but remember to light for contrast not final levels. More light means faster shutter speeds means less motion blur. Light for the type of shadows you want, not how dark you want the scene when you're shooting for visual effects. If your key and fills are set properly, you can always make the shot darker when grading. This isn't as much of an issue when shooting film or deep color digital like the Arri Digital cameras since their shadow detail is good enough to boost for tracking detail. But if you're shooting with consumer or prosumer cameras, you've got jack but noise in the shadows and the low light levels will mean excessive blur and bad noise overall.

Fourth: Manual vs Auto. Boujou has a reputation as being the fire-and-forget solution for people that don't understand tracking. The really hard shots are going to need manual solutions. I don't mean hand-keyed (necessarily) but we're talking careful selection of usable tracking detail and improved setting of constraints. Personally, I recommend SynthEyes - unbelievably good price point and an unbeaten feature set. I know there are shots that people say they can click a button and get a solve on in Boujou that they can't get in SynthEyes, but the reverse is true far more often than it's not. When the footage is hard to work with, throw away the automatic solutions. Give them one run at it if you've got the time and processor power to spare, then move to something where you have some control.

Fifth: Tracking techniques. Track the center, not the edges. When doing a supervised track of moblurred features, follow the center of those blurs.

Sixth: Rendering techniques. Moblurred footage is hard to render to match because what you have to match is a particular slice of time. Think of it this way: you're placing tracking points on each frame, generally spaced apart at the rate of 24, 25 or 30 frames per second. Between frame 1 and frame 2, there's a fraction of a second where anything could happen. The camera could jump up a couple centimeters and back down for instance, before hitting frame 2. This will show in the motion blur but not in the keyframes. You can't easily fix for this! (See tip 1). More often, though, the problem is the phase of the render. How to address this will depend on your renderer. You may be able to adjust this in the renderer, you may have to adjust it in the actual animation keyframes depending on what 3d software you're working from. Basically, you need control of the shutter timing and shutter offset: when in that frame it opens and how long it stays open.

For high end vs. low end facilities, #6 can often be the difference between them. For all the skill you can buy from freelancers in tracking, modeling, animation, textures, etc: having the resources to do test renders and tweak that setting (and having someone on staff who can make sure you're addressing this) makes all the difference.

photo credit: Capture Queen ™I have a side project for iPhone that I'm working on. There's a patent filing in process, incorporation paperwork to do, all that good stuff - so I won't be doing a lot of specifics about the project until it's ready to go. It's basically a location-based-services thing but when we've looked over what everyone seems to be doing with LBS and social networking and the like, it all seems to be thinking very inside the box. Just like a few years ago there was a rash of "... on the internet!" patents where things we'd done forever in the wetworld were being done online and called "visionary", now there's a rash of similar "... on a mobile phone!" patents and websites that are no more groundbreaking than when we did them online or on our feet. It's like patenting driving in nails with the side of a hammer. It works, but it's not especially groundbreaking nor is it even ideal.

We really feel that this project is going to move mountains though, and change the way people view the real world around them, not just when they're in front of a computer.

The tendency towards developing for the iPhone is that the iPhone has arguably the best SDK there is: and a substantial and rapidly growing marketshare. Still, though, that limits our deployment: it's the largest selling smartphone-style device, but it doesn't represent the majority of the market. No-one does.

Today I've been evaluating Mojax, a cross-platform mobile development environment. It looks promising. It promises cross-platform compatibility, including access to a number of necessary core services like GPS, and currently runs on all phones supporting J2ME (Java on Mobile), all color Blackberry phones, and shortly Windows Mobile devices from WinMo 2003 onward, any mobile running Brew V2 or later, and Helio devices. Just the J2ME support gets me onto most smartphones - so this effectively would get the product onto every mobile device there is with gps capability.

This excites me because I like the prospect of running the application on every feature-capable mobile phone without having to separately develop for Blackberry, HTC, Samsung, and Nokia. Getting the product out simultaneously for iPhone, Windows Mobile, PalmOS and Blackberry would be fantastic. There are some annoyances with using J2ME midlets, but I'll take the slight user experience tradeoff for being able to deliver an application at all without breaking the bank

So I'm working on a side project that's mainly just for me, but it's so damn handy I might just make it publicly available when it's done (probably for a small monthly fee). It's basically a super quick way to run certain types of scan for stocks during the day, picking out data that's usually hard to get until the end of the day. I'm tying it into the OptionsXpress API so that when I identify a good entry or exit point on a stock, I can change my position with the click of a button.

I've been designing the site to automatically rescale for my mobile phone, but I need a way to determine if I'm looking at it on the phone or on a desktop - and since my mobile browser is generally configured to identify as a desktop browser (so that I don't get sent to the "mobile version" of the sites I visit) it's harder to do. My browser does a great job of rendering full-screen pages and scaling them - if you've seen the iPhone browser, both Opera Mini and Pixsel do something similar though without 'multi-touch'.

That said, there's something to be said for designing a website that will gracefully scale when it's on a phone.

But when the browser identifies itself as a desktop browser, what's left? Javascript can pick up the screen size, but what about PHP? All my pages are created in PHP...

More »

and it's not even in the theater yet - but the trailer is.

If you've seen Indiana Jones and the Kingdom of the Crystal Skull in theaters this weekend, then I'm certain you've seen the first trailer for David Fincher's The Curious Case of Benjamin Button. You might remember seeing a rather weird trailer with Brad Pitt and Cate Blanchett and no dialogue but with beautifully melancholic music - that was the Benjamin Button trailer. Remarkably it's not online yet, so I can't exactly show it to everyone, but I can say that is absolutely phenomenal. I was not expecting it when I went in to midnight showing of Crystal Skull a few nights ago, but I was astounded at what I saw. I've got to start talking about this now, I can't constrain myself, because this trailer looked downright amazing, so much so, that I'm already claiming this is next year's Best Picture.
- FirstShowing.net

It's not often that a movie that's neither a remake nor a sequel creates such a staggering worldwide buzz within days of the appearance of its trailer (and only in theaters!  Except for a Spanish bootleg, there's still nothing online).  But just as I'd said weeks ago when I first saw the edit (when there was still bluescreen and placeholders for some scenes and we were watching versions of it with various pieces of temp music): the freakin' *trailer* gave me a lump in my throat.

Apparently it's not just because I was emotionally connected to it.  I've seen several bloggers say that they've gone back to their local theater 4-5 times SINCE THURSDAY NIGHT just to see this trailer again.  With words like "phenomenal", "masterpiece", "extraordinary" and people already insisting this will be the next year's shoo-in for the "Best Picture" Oscar, and finding that others have commented months back that just reading the script made them cry, I think I'm working on a great, great film.

DD's work looks great (I've seen a good bit more than is visible on the trailer, now, and I'll leave it to the future to reveal exactly what they're doing on it since I have yet to see any mention of it in the press) and I'm proud as hell of what we've been doing at Asylum.

It was good to come home today after a really long day  (I was on set for something else today from 7am to 10pm) and check the internet for the trailer and comments which I'd heard were appearing.  When you're doing something you love, have the respect of your peers, and get to work around people you actually look up to: it's a pretty nice way to live your life.  If I ever start aging backwards, like our dear friend Mr. Button, remind me to start doing this sooner.

So this week, the Visual Effects Society (VES) published the first ever set of guidelines for titles in the visual effects industry. They assure us that Many Very Important People have looked over the list and given it the thumbs up.

"Through these collective efforts, a harmonized master list of titles has been created that should be useful as a touchstone for all effects stakeholders," says VES Exec Director, Eric Roth. "We think it represents a major step forward for our craft in standardizing the use of titles in the credits process for effects artists."

Of course, I'll interject here that if the Visual Effects Society actually carried any weight, someone would care. I'll further interject that if they'd actually attached descriptions to these titles it would mean a bit more - but since a good chunk of these are too nonspecific to mean anything (or at least vague enough to get people to fight over them) they've basically left us where we were before. (Except that since my responsibilities shift and sway on a regular basis, I can dig through the list for suggestions of what I should be calling myself - like this weekend, I think I'm calling myself the Visual Effects Photographer)

Questions regarding these guidelines can be addressed by calling the VES at 818-981-7861. Credits/Titles to be submitted in accordance with VES Guidelines follow after the cut:
More »

I know I sound like a broken record to some, but I had this footage sitting on my workstation and didn't want it to go to waste.

Since my earlier blog about normals-based relighting of rendered cg elements (something that can obviously be done in the original CG software if time allowed), I'd written a short piece on relighting real-world footage. Since then, I expanded on it slightly and now I've put it up on YouTube as a quick technique video.

A detailed video covering the normals-based relighting process is located here.