The Cinematography of "The Lego Movie"
"Emmet's Apartment"
Challenges
The brief from the directors was clear: PHOTOREALISM. A word that sends a shiver down the spine of even the most seasoned CG veteran (perhaps especially so the more seasoned).
"We want it to look as though it's all been made in someone's basement - someone with a lot of time on their hands - and then lit and photographed by an absolutely top-notch miniatures photographer."
I think we were all initially thinking "It's plastic - how hard can it be?" Well.....hard.....very hard.
Let's leave aside the technical considerations (the subject of a future article) and consider the cinematographic challenges.
Pretty much everyone is familiar with Lego. We "know" what it looks like - how it reacts to different lighting conditions, the patina of its surface, the way light transmits through it. So to represent Lego photorealistically via computer graphics is most unforgiving - the slightest mistake will just look wrong, and almost everyone will react to the mistake, even if they could not articulate what was wrong about the image they were viewing.
A few challenges we needed to rise to:
The brief from the directors was clear: PHOTOREALISM. A word that sends a shiver down the spine of even the most seasoned CG veteran (perhaps especially so the more seasoned).
"We want it to look as though it's all been made in someone's basement - someone with a lot of time on their hands - and then lit and photographed by an absolutely top-notch miniatures photographer."
I think we were all initially thinking "It's plastic - how hard can it be?" Well.....hard.....very hard.
Let's leave aside the technical considerations (the subject of a future article) and consider the cinematographic challenges.
Pretty much everyone is familiar with Lego. We "know" what it looks like - how it reacts to different lighting conditions, the patina of its surface, the way light transmits through it. So to represent Lego photorealistically via computer graphics is most unforgiving - the slightest mistake will just look wrong, and almost everyone will react to the mistake, even if they could not articulate what was wrong about the image they were viewing.
A few challenges we needed to rise to:
- Scale - how miniature should the world feel? Can we retain a sense of the epic while evoking the tiny?
- Emotion - how can we enhance the drama of the film, given that our characters have cylindrical heads that are simpler in their response to light than human heads, and lack the emotional focal point of the human face, the eyes (minifig dots scarcely count as eyes in the photographic sense)
- Light in a Lego World - How can we make people believe that this Lego world was actually constructed from real plastic bricks and photographed? How does light of various intensities and hardness react with Lego? What about coloured light?
- Photographic considerations - Depth of Field, lens breathing, chromatic aberration, anamorphic distortion, caustics, subsurface scatter, atmospherics, dust motes
Inspirations
Before I came on board the show, Production Designer Grant Freckelton had been doing extensive research and creating exploratory artwork to begin to get the "feel" of the visuals. The many "brickfilms" found on the internet served as a point of departure, but the photographic work that really resonated was that of Vesa Lehtimäki a.k.a. Avanaut whose miniature photography using Lego is both evocative dramatically and completely charming, like the example below:
"Bad to the Bone" Copyright AVANAUT
Grant spoke to Lehtimäki, who revealed that one of his influences was the work of Douglas Trumbull, the effects legend behind films such as Blade Runner and Close Encounters of the Third Kind, amongst others. One of Trumbull's signature approaches to miniature cinematography was the use of "smoke tanks", using a very high density of smoke in the room in which the miniatures were filmed to help separate elements and enhance the sense of scale. (There is an interesting video on his website in which Trumbull explains the miniatures work done to create the "Blade Runner Hades Landscape").
Blade Runner "Hades" miniature set showing smoke density
Avanaut says he calls his own technique "Forced Atmospheric Perspective Photography".
"Last Ship to Rendezvous Point" Copyright AVANAUT
Grant and I were not exactly averse to the heavy use of atmospheric smoke and particulates, as evidenced by the work we did on "Legend of the Guardians", so it felt natural to embrace an atmos-heavy approach.
Heavy smoketank effect in "Destruction of Cloud Cuckoo Land" sequence of "The Lego Movie"
We also discussed the cinematography of live action films. For example, Mitchell Amundsen's work on Transformers was particularly informative when it came to balancing the contribution of lightsources in our freeway chase sequences. We talked about how the HMI lights dominated the scene and provided dramatic emphasis, and how practical lightsources such as streetlights and headlights contributed little to the actual lighting of the scene, but were nonetheless hot at their source. Practical lightsources also play strongly in reflective surfaces, a phenomenon we made much use of in our lighting work.
Photographic Analysis
We also embarked on a series of photographic investigations of real Lego under various lighting conditions. The Art Department had been making Lego models of vehicles and buildings, and we took these out of the studio early one sunny morning and arranged them into streetscapes, photographing them at various exposures and making High Dynamic Range (HDR) probes of the surrounding environment for future CG look development explorations. There was also a lot of photographic reference ranging from individual bricks to characters to sets photographed under various conditions of artificial and natural light, including studies of backlit elements and visible LED lightsources. We analyzed these studies closely to determine what "worked" visually, and this not only helped us formulate an approach to the lighting styles of the movie but also was informative for the Look Development team, led by Jean-Pascal le Blanc, as they worked on the Lego shader and surfacing.
We also embarked on a series of photographic investigations of real Lego under various lighting conditions. The Art Department had been making Lego models of vehicles and buildings, and we took these out of the studio early one sunny morning and arranged them into streetscapes, photographing them at various exposures and making High Dynamic Range (HDR) probes of the surrounding environment for future CG look development explorations. There was also a lot of photographic reference ranging from individual bricks to characters to sets photographed under various conditions of artificial and natural light, including studies of backlit elements and visible LED lightsources. We analyzed these studies closely to determine what "worked" visually, and this not only helped us formulate an approach to the lighting styles of the movie but also was informative for the Look Development team, led by Jean-Pascal le Blanc, as they worked on the Lego shader and surfacing.
Scale
As mentioned above, we had to balance the simulation of a miniature world with a sense of the epic, while also allowing for intimacy at appropriate moments. Partly this was achieved through a generally shallow depth of field, coupled with camera work led by cinematographer and Layout Supervisor Pablo Plaisted which strove to emulate a camera operated at a Lego scale. Pablo articulates the process pretty thoroughly in this interview on fxguidetv, the salient points being:
Depth of field was set in Layout, and we inherited those DOF settings via the pipeline in Lighting, but DOF was ultimately implemented in our compositing stage. The DOF that Layout set was treated as "intent", but occasionally we would alter it as we observed how it worked with the lighting. Part of finding a sense of the "epic" sometimes involved deepening the DOF (bearing in mind that Layout was setting the DOF without the benefit of fully shaded and lit renders). Also, the directors would sometimes ask for background action or scenery to be more sharply in focus to help sell a gag or to highlight some of the funny secondary animation that our superb team of animators had created.
Floating motes of dust were also used as a nod to the real world but also to help the sense of scale. Occasionally a large hair will drift by, or motes that are relatively much larger than what you would see in a human scale movie would be caught in a beam of light. As a general rule, motes would be composited so they were only visible in light beams so they weren't distracting (we had several motes that seemed to be going for an Oscar that needed to be expunged from various shots).
As mentioned above, we had to balance the simulation of a miniature world with a sense of the epic, while also allowing for intimacy at appropriate moments. Partly this was achieved through a generally shallow depth of field, coupled with camera work led by cinematographer and Layout Supervisor Pablo Plaisted which strove to emulate a camera operated at a Lego scale. Pablo articulates the process pretty thoroughly in this interview on fxguidetv, the salient points being:
- lens as though at Lego scale using virtual lenses that are proxies to real-world equivalents
- DOF is accurate to the physics of those virtual lenses
- as a general principle the point of focus aligned with the stereoscopic convergence point, which resolved the tension between the audience desire to rove their eyes around the frame in a stereo image and the need to guide the audience to action or story points
- rather than allow the CG camera to pivot on a nodal point, rigs were built to emulate real-world camera rigs such as Steadicam rigs, albeit at a Lego minifig scale. This allowed for more "human" camera movement without the artificial perfection of normal CG camera work.
- a sense of weight and inertia was keyframed into the camera movement, again to seat the camerawork in the "real world" of the Lego sets, giving the audience the sense that the virtual camera had actual mass.
- imperfections were explored, such as deliberate overshoot of focus pulls and camera moves. With such a generally shallow depth of field it was found that overshooting DOF led to popping and was distracting, so the focus pulls were carefully and accurately done (sometimes frame by frame for very close camera work), but overshoots on camera moves were retained.
Depth of field was set in Layout, and we inherited those DOF settings via the pipeline in Lighting, but DOF was ultimately implemented in our compositing stage. The DOF that Layout set was treated as "intent", but occasionally we would alter it as we observed how it worked with the lighting. Part of finding a sense of the "epic" sometimes involved deepening the DOF (bearing in mind that Layout was setting the DOF without the benefit of fully shaded and lit renders). Also, the directors would sometimes ask for background action or scenery to be more sharply in focus to help sell a gag or to highlight some of the funny secondary animation that our superb team of animators had created.
Floating motes of dust were also used as a nod to the real world but also to help the sense of scale. Occasionally a large hair will drift by, or motes that are relatively much larger than what you would see in a human scale movie would be caught in a beam of light. As a general rule, motes would be composited so they were only visible in light beams so they weren't distracting (we had several motes that seemed to be going for an Oscar that needed to be expunged from various shots).
Characters and objects were covered in fingerprints, and exhibited wear and tear such as nicks and divots, all of which help to convince the viewer they are looking at real Lego pieces.
The screengrab at the left shows how the placement of a fingerprint (on the side of Batman's head) subtly reinforces the scale of the world we are viewing (click to view larger).
The screengrab at the left shows how the placement of a fingerprint (on the side of Batman's head) subtly reinforces the scale of the world we are viewing (click to view larger).
There are many shots in the film in the larger environments where our default lighting schemes did not do enough to give a sense of the epic to the shots. In these cases we artificially controlled the lighting falloff from the top of the environment to ground level, either using depth data to reduce exposure according to the height of objects in compositing or emulating this effect in mattepainting. This also helped action to play better in shots looking down - brightly lit foreground characters or props playing against a darker background. At ground level in these sets we used the same technique, often holding the sunlight off the ground and fading it off the buildings or canyons so the foreground action could play against that reduced exposure.
Lighting Principles
- Thou shalt not suffer a character to be frontally lit without a very good reason - this rule is designed to maximize the shaping of the characters via rimlighting and to utilize the subsurface scattering properties of the Lego plastic shader i.e. to allow light to scatter through the plastic for aesthetic reasons. Of course, there are inevitably times when characters will end up frontally lit, especially in very dynamic shots or where lighting continuity is critical, but in general we would choose beautiful lighting over strict continuity - it's more important to create and maintain a mood than to be too literal.
- Maintain Separation and ensure a good, clear read of the most important characters in a shot. This can be achieved via silhouette (for example staging against atmospherics, or by use of contrast), by colour or by judicious use of Depth of Field. Occasionally we would resort to a technique we call the "Shatner light", named for the cheesy technique of placing a bar of light over the eyes or face of Captain Kirk in the old Star Trek episodes - we always implemented this with a sense of fun on Lego, (though the technique was used with more serious intent on Legend of the Guardians).
- Clarity of action - with a lot of chaotic action scenes in the film we strove to allow a clear read of the important action, often utilizing the same techniques as for maintaining separation of the characters as described above, but also by directing the eye of the viewer via lighting gags, such as the spotlights from the helicopters in the freeway chase sequence.
- Shape and modelling, even of rectilinear or trapezoidal objects or characters. These could pose a real challenge (e.g. the Unikitty character who is pretty much all cubes). Sometimes we had to artificially enhance the light coming from one direction to get variation, or control the separate components of one or more lights in compositing, to reduce reflections off a planar surface at certain angles, for example. In general, the dominant lightsource in a shot would be placed to provide interesting shape and shadow, being mindful of but not shackled by continuity.
- Allow overexposure - we used overexposure to sell the notion that we were actually exposing film photographically. Sometimes the overexposure is only evident in hot specular "pings" of light, but still helps convey that bright light is reflecting realistically.
- Keep an eye on density - make sure nothing in the frame drops off to complete black and have enough range for grading work in the digital intermediate stage.
- Enhanced photorealism - we wanted a convincing simulation of photorealism, but we used every trick in the book to treat that as a point of departure rather than a limitation. We departed from strict physically correct DOF when required, used special lights that only affected certain objects or characters in a way that could not be done with real world lighting, utilized AOVs (Arbitrary Output Variables, output as part of our rendering process) that allowed us to manipulate lights in our compositing software (or even subcomponents of individual lights, such as the diffuse, specular, scatter or indirect contribution) or to work with other data such as normals or depth, for example to reduce the exposure of distracting areas of a set. Standard outputs for a shot included "grademattes" that allowed targeted grading work in the DI (digital intermediate) process - for example we could enhance a particular character in DI, or increase contrast of the face decals of a character for better clarity.
- Fix it in the mix - the Digital Intermediate was treated not as a balancing and correcting process, but as an integral part of the lighting process. This enabled us to work with fewer iterations in Lighting and Compositing (critical as we had a small crew), pushing some of the finessing work into the DI suite where iterations were faster. The provision of grademattes, as mentioned above, meant that granular control could be exerted over important parts of shots in DI
Emotion and Enhancement of Drama
It's one thing to light for action in a Lego movie, but quite another to light for drama and emotion. Our characters have trapezoid bodies and cylindrical heads - the flat planes of the bodies reflect light at the camera inconveniently, creating distracting flashes, while the heads model light without subtlety or nuance.
Part of the solution is to apply the principles listed above - a dominant rimlight, soft lighting on the face. Shallow depth of field helps create a sense of intimacy. The colour temperature of the light plays a role, with cooler tones indicating a more sombre emotional state. It can work well to contrast background with foreground colour temperature, for instance in the scene where Lucy farewells Emmet in the tunnel before the risky phase of their mission, it is implied that they may not see each other again. Lighting Lucy with warm light enhances her "humanity", while the steel blue of the tunnel reinforces that the environment is hostile:
It's one thing to light for action in a Lego movie, but quite another to light for drama and emotion. Our characters have trapezoid bodies and cylindrical heads - the flat planes of the bodies reflect light at the camera inconveniently, creating distracting flashes, while the heads model light without subtlety or nuance.
Part of the solution is to apply the principles listed above - a dominant rimlight, soft lighting on the face. Shallow depth of field helps create a sense of intimacy. The colour temperature of the light plays a role, with cooler tones indicating a more sombre emotional state. It can work well to contrast background with foreground colour temperature, for instance in the scene where Lucy farewells Emmet in the tunnel before the risky phase of their mission, it is implied that they may not see each other again. Lighting Lucy with warm light enhances her "humanity", while the steel blue of the tunnel reinforces that the environment is hostile:
Angled light or shadow can also serve to emphasize drama, for example in this shot where the slanted rays, counterintuitively not coming from the direction of the window, guide the eye to Emmet in the frame and reinforce that dramatically, this is Emmet's moment:
The ability to isolate components of light striking the characters allowed us to remove light that interfered with performance or was generally annoying. For instance, it was a general rule to remove bright specular highlights from the characters' faces if it played over the animating face decals. We could also choose to remove or reduce a light reflecting off a plane at certain angles to avoid distracting flashing.
Emulating Lens Effects
Real-world lenses exhibit various photographic idiosyncracies, and we wanted to emulate some of these (while retaining creative control over them) in order to avoid a clean look that would indicate that a CG camera had been used. After the trailer for the film was released there were a few comments we noticed where people were authoritatively holding forth about how they "knew" exactly what camera had been used to "film" the "stop motion". Mission accomplished :)
Colour Pipeline Lead and compositor Alex Fry created a LegoLens kit for use in our compositing software, Nuke, where lens effects could be applied and controlled.
At its most basic, the LegoLens was designed to apply the Depth of Field accurately to the image, based on the focal distance and fstop settings that had flowed through from the Layout Department. Additionally, the LegoLens included:
Flares were also added to images, sometimes using plugins in Nuke or elements created in other software but more often added in DI. The flares were either added for reasons of photographic fidelity e.g. to emulate what happens when a bright LED light source is visible in the framing of a shot with an anamorphic lens (such as the cliched anamorphic flares that J.J. Abrams actually apologized for over-using all over the place in his movies), or to contribute to dramatic reveals in a thematic sense, for example in the reveals of Wildstyle/Lucy where a playful visual sense of the ridiculous coupled with a musical motif underscored a recurring gag. Or, of course, when they looked cool.
Real-world lenses exhibit various photographic idiosyncracies, and we wanted to emulate some of these (while retaining creative control over them) in order to avoid a clean look that would indicate that a CG camera had been used. After the trailer for the film was released there were a few comments we noticed where people were authoritatively holding forth about how they "knew" exactly what camera had been used to "film" the "stop motion". Mission accomplished :)
Colour Pipeline Lead and compositor Alex Fry created a LegoLens kit for use in our compositing software, Nuke, where lens effects could be applied and controlled.
At its most basic, the LegoLens was designed to apply the Depth of Field accurately to the image, based on the focal distance and fstop settings that had flowed through from the Layout Department. Additionally, the LegoLens included:
- Focal Distance Offset - enabled trimming of predefined focal distance
- Fstop Exposure Value Offset - allowed us to change the fstop as an offset to the inherited setting
- Anamorphic Focal Length Factor - choose between spherical lens mathematics or compensatory math to allow for wider field of view of the anamorphic lens
- Chromatic Aberration - control over the spread of chromatic aberration smear
- Convolution - control the colour bias and saturation of the DOF kernel
- Flare - control the size of the blur that comprises the flare and the ratio between this blur and the original image
- Lens Breathing - a way to simulate the distortions over time introduced during focus pulls. Looked cool at times, though we frequently disabled it when it altered our framing away from the Layout intent.
- Bokeh - incorporated into the LegoLens tool was a node that set the base look of the bokeh, that is, the aesthetic quality of the blur of the out-of-focus pixels of the image
Flares were also added to images, sometimes using plugins in Nuke or elements created in other software but more often added in DI. The flares were either added for reasons of photographic fidelity e.g. to emulate what happens when a bright LED light source is visible in the framing of a shot with an anamorphic lens (such as the cliched anamorphic flares that J.J. Abrams actually apologized for over-using all over the place in his movies), or to contribute to dramatic reveals in a thematic sense, for example in the reveals of Wildstyle/Lucy where a playful visual sense of the ridiculous coupled with a musical motif underscored a recurring gag. Or, of course, when they looked cool.
Lighting for Action Sequences
Action sequences in The Lego Movie tend to have a lot of chaos from which it is essential to isolate and highlight the critical action so the audience doesn't lose the flow of the drama.
Our approach to solving this relies on application of our Lighting Principles, the main point being to separate the hero action from the visual and kinetic chaos of the background.
In this complex shot from the final battle there are several story beats where different characters deliver lines while performing different actions, with the common thread of Emmet steering his giant mech through the madness. All these have to play clearly against a busy background, where it would be easy to lose our action in the visual clutter and swirl of movement:
Action sequences in The Lego Movie tend to have a lot of chaos from which it is essential to isolate and highlight the critical action so the audience doesn't lose the flow of the drama.
Our approach to solving this relies on application of our Lighting Principles, the main point being to separate the hero action from the visual and kinetic chaos of the background.
In this complex shot from the final battle there are several story beats where different characters deliver lines while performing different actions, with the common thread of Emmet steering his giant mech through the madness. All these have to play clearly against a busy background, where it would be easy to lose our action in the visual clutter and swirl of movement:
In the following shot, we need to register the predicament of our hero group and then to comprehend that Emmet, in his mech, is coming to the rescue:
In the lengthy chase sequence set in the Wild West the guiding dramatic theme is that our heroes are trying to get to the sun, which serves as a focus for the entire action sequence. This affords us plenty of opportunity for raking light across the environment, rimlighting characters and using the shadow side of objects such as mesas to stage lit action against:
Cloud Cuckoo Land is a purposely chaotic environment with "no rules" so we had to proceed with that as a principle in the lighting design, but also find ways to keep the audience looking at what is important. The following shot is a long, complex reveal of the environment and the dance party craziness:
Practical Lights
The Lego Movie had sets that required the implementation of thousands of "practical" lights, emulating the placement by an extremely patient stop-motion animator of many individual LED lights in a real Lego set. Contractually I can't say much about how we went about this, except that we decided early on that it would be a bad idea to place each of these lights individually (because it would take a very long time). Suffice to say we developed a system that automated the placement of lights in nominated parts of sets, in vehicles and other props, and also a system that populated FX brick clouds with "LEDs" inside individual bricks or clumps of bricks, to internally light explosions, for example.
This, coupled with the development of new render technology, made complex lighting schemes not only feasible but efficient.
The Lego Movie had sets that required the implementation of thousands of "practical" lights, emulating the placement by an extremely patient stop-motion animator of many individual LED lights in a real Lego set. Contractually I can't say much about how we went about this, except that we decided early on that it would be a bad idea to place each of these lights individually (because it would take a very long time). Suffice to say we developed a system that automated the placement of lights in nominated parts of sets, in vehicles and other props, and also a system that populated FX brick clouds with "LEDs" inside individual bricks or clumps of bricks, to internally light explosions, for example.
This, coupled with the development of new render technology, made complex lighting schemes not only feasible but efficient.
Practical lights could also be used to help draw attention to important action. In the freeway chase we used the spotlights of the "Copper Chopper" to highlight Emmet and Lucy on the Supercycle, and also to silhouette a couple of crucial action moments:
Mattepainting
Many of our shots rely heavily on Mattepainting for their effectiveness, and I have to acknowledge the outstanding contribution of Dudley Birch and his crew of talented mattepainters to the visual success of The Lego Movie.
Mattepainting on The Lego Movie was no simple task involving "painted" backgrounds, but often involved intricate sets made from tens of thousands of Lego bricks with a whole suite of techniques applied to them, often rendered with as much or more difficulty than the larger scale lighting scenes.
Kudos also to Grant Freckelton, whose laser eye and visual direction really sealed the success of these mattepainting elements and who often contributed sections of mattepainting himself.
One of the great challenges in constructing these shots was integrating mattepainted elements with those rendered in Lighting, a task that our highly proficient compositors and lighters rose to superbly.
Unfortunately I can't show mattepainting elements or work-in-progress of any kind due to legal restrictions, but here are a few frames where almost everything in the image is mattepainting:
Many of our shots rely heavily on Mattepainting for their effectiveness, and I have to acknowledge the outstanding contribution of Dudley Birch and his crew of talented mattepainters to the visual success of The Lego Movie.
Mattepainting on The Lego Movie was no simple task involving "painted" backgrounds, but often involved intricate sets made from tens of thousands of Lego bricks with a whole suite of techniques applied to them, often rendered with as much or more difficulty than the larger scale lighting scenes.
Kudos also to Grant Freckelton, whose laser eye and visual direction really sealed the success of these mattepainting elements and who often contributed sections of mattepainting himself.
One of the great challenges in constructing these shots was integrating mattepainted elements with those rendered in Lighting, a task that our highly proficient compositors and lighters rose to superbly.
Unfortunately I can't show mattepainting elements or work-in-progress of any kind due to legal restrictions, but here are a few frames where almost everything in the image is mattepainting:
The End