Working with visual effects and compositing: A guide for directors and cinematographers.
(Excerpt taken from the visual effects chapter in my new film book, "Making Your First Blockbuster", due for release early 2019 by Michael Wiese Productions.)
“All art is technology. That’s the very nature of it. The artist is always bumping against that technology.” — George Lucas, visual-effects pioneer / director
The 14-year-old me had my first attempt at special effects when I put some firecrackers inside my Millennium Falcon Star Wars toy (please don’t tell my parents). It blew up but didn’t look particularly spectacular. The build-up was more exciting than the actual effect to be honest. I’m sure the special effects company, Industrial Light and Magic (ILM), could have done it a lot better...
I’ve always been fascinated by effects movies. From my Star Wars (1977) and Back to the Future (1985) viewings as a kid through the subtle use of effects in Forrest Gump (1994) and Cast Away (2000), I’ve been trying to work out how they did this magic. What sleight of hand did the magician use? Computer Generated Imagery (CGI) gets a bad rap when used in films that are all spectacle and no heart; but it is just one of many tools the filmmaker uses, and the CGI isn’t necessarily the reason a film didn’t work.
Some visual effects are easy and some are incredible hard. My artists and I have completed gorgeous shots in minutes; others take days or even months of working, like slowly building up a matte painting for it to be on screen for just five seconds.
The world of visual effects has moved on as the technology has improved. The artist’s creative visions have forced the technology to keep up. The work involved in generating a visual effects shot requires an enormous amount of talent and skill on the part of the artist and people involved. Pioneers like directors Robert Zemeckis, George Lucas, and James Cameron, use these tools to create the images they want to tell the story with, without drawing the audience’s eyes to the fact that they’re watching visual effects. Audiences might also fail to notice the heavy visual effects and CGI work in dramas like Pride and Prejudice (2005) and Atonement (2007). That gorgeous midnight moon that Lizzy Bennet looks up at? CGI. That setting sun and painterly sky above the fields in Atonement? CGI. Those crowd scenes in the marketplace? The list goes on . . .
Let’s take a look at some of the common terminology and elements found in the area of visual effects. It took me a while to get my head around these various elements and to understand how they work with each other. Once you’ve seen the process in action, from location or studio shoot to the visual effects suite, you’ll have a better grasp of the process. Any directors, producers, DPs, or editors out there should familiarize themselves with these terms so they are able to speak the language of their visual effects supervisor or artist when on set.
Composite — The final image containing all elements in the shot. The background plate with the actors, the CG dinosaur (or whatever CG model you may have), and the CG smoke effects all added together, ready to go back to the editor and be placed in the film. All of the individual layers are squashed down to create a single file clip as if it was shot as one on set. A compositing program such as Adobe After Effects or Nuke can be used to do this. Anything can be placed on top of your video layer, but to have things behind it would involve cutting your subject out either with green screen or rotoscoping work.
Green / Blue screen — Objects or actors can be filmed in front of a green or blue screen as foreground elements, and then composited into a background plate. The green / blue part of the image is removed in the computer by the “keying” process to leave an empty space behind to be filled with other images. The green screen process allows a “matte” to be created.
Matte — In the older days of film processing, a “matte” was created by having part of the image blocked off, usually by a black card or paint, to prevent exposure in that part of the frame. It could then be optically composited with another image, allowing two images to be combined. Optical compositing was the way two film images were combined before it was done digitally. A matte is an opaque image (not transparent) in which we control what areas of the frame we discard and what areas we keep. The green screen process and rotoscoping are two ways of achieving a matte that can then be used for compositing.
Keying — The process of removing a particular color from a filmed image. Usually blue or green screen, but can also be black if filming pyrotechnics (white smoke and spark effects). The end result is a matte that can be composited with other elements.
Background plate — The background image that is composited behind the CG elements or green / blue screen keyed images. This background plate can be filmed on location, or be a CG environment or still image. Most of the time, if the camera is moving, the background plate will also have to move to match the foreground action. Therefore, background plates are normally of a higher resolution or size in order to be able to move around the composite. Background plates are usually shot first so that the lighting on any foreground elements such as your green screen actors can match it.
Elements — Elements are smaller parts of a shot that could either be produced using CG from a 3D animation program or shot as live that are placed on top of or under the live-action layers. The elements might be particles of dust, smoke, debris, water, rain, snow, or fire. They could be moving or still-based, and are added to the composition to help tell the story and enhance the reality of the shot. As an example, below are fire and smoke elements on a transparent layer, ready to be placed into a composition. A “particle system” is section of a 3D animation package or standalone program that can generate and animate hundreds or thousands of particles or objects. It then allows you to apply any natural forces to it within the computer such as wind or gravity to help create a natural or magical phenomenon. Particles systems can be used to create falling leaves, snow, rain, smoke, dust, clouds, water, tornadoes, fireworks, or even magic dust.
Render — When the compositing or 3D program outputs the final image containing all the elements in their finished and finessed state, it is called a “render.” So if you hear of “render times” being high, you know that the shot is complicated and is using a lot of processing power to finish it. It is then saved in a format and codec that your NLE system can recognize, and is placed back into the edit timeline. A render from a 3D program can be saved as a whole complete file or can be done as various lighting “passes” that are then placed together within the compositing program. An artist producing a 3D object could render a “diffusion pass,” a “specular pass,” a “reflection pass,” and finally an “ambient occlusion pass” of that object. All of these lighting variables can be tweaked and adjusted within the final composite, which is then itself rendered out. The big visual effects houses have “render farms,” which are rooms full of computers all linked together to utilize all of their processing power.
Let’s look at a breakdown of a composited visual effects shot to help understand the process:
Rotoscoping — The process of manually cutting out your subject from the background image frame by frame to end up with the actor/object isolated as a single element, as if they have been keyed out from a green or blue screen. Rotoscoping is an alternative to creating a matte with a green screen. This is a time-consuming process since the actor or object has to be removed or “cut out” from the frame by hand over the course of the shot, one frame at a time, at 24 (25 in the PAL countries) frames per second. The process of rotoscoping uses a mask to draw around the area in question. Rotoscoping is sometimes favored over keying (if your background plate is also the actual background shot on location) since the lighting of your actor and background will match exactly, being the same shot; conversely, green screen necessitates you trying to match the background lighting in a studio weeks later. Objects or elements such as atmospheres, tornadoes, monsters, or explosions can then be placed behind the actor and in front of the background. You may even rotoscope your actor out of the shot and place them in an entirely different environment. The choices are endless. Rotoscoping is also sometimes used as a last resort with poorly filmed green screen to remove the subject from the shot. One of the hardest things to rotoscope is hair or anything flimsy and moving like leaves or plants as they don’t have a consistent shape or movement. Rotoscoping is also your get-out-of-jail card if there isn’t time or the resources on the shooting day for correctly set up a green screen.
Spill — When the subject stands too close to the green / blue screen, light can reflect off the screen and “spill” onto the actors or objects, which can cause keying problems. Parts of the subject will disappear, along with the green screen. Check out the green spill on our actor’s shoulder in the picture below.
Tracking markers — Tracking markers are colored balls, colored tape, or stickers placed on a green / blue screen when the camera is moving to allow the camera movement to be “tracked.” That movement information (stored as X, Y, and Z coordinates in 3D space) can then be applied to the imported background plate, so it can move with the foreground camera shot to create the illusion that everything was filmed at the same time.
If I’m shooting something as a live-action plate, and there are no green screen or tracking markers, the compositing software can use markers in the scene itself to help track the shot. Corners of buildings or lampposts can be tracked to get an idea of what the camera is doing. So if you find yourself shooting a moving background plate, make sure you have something in shot that can be tracked that stays in the frame for the entire length of the shot and doesn’t get obscured by anything in the foreground.
Matte painting — A matte painting in the old days of visual effects were images painted on sheets of glass to help extend a set or create some fantastical part of that world. They were painted on glass so they didn’t move or sway when filmed and ruin the illusion. Parts of the painting were left clear so that other elements or shots of actors could then be inserted into the space. Today we have digital matte paintings, stitched together from a series of high-resolution stills or CG-rendered images and are then sometimes used as background plates.
Plug-ins — A plug-in is a smaller computer program used in combination with a compositing package or 3D program. Plug-ins cannot run on their own, but must be used within another effects package. Plug-in programs can range from producing specialized areas such as creating particles like smoke and sparks, emulating film emulsions, creating lens flares, to a combination of neat little effects that can be added to your shots. Each layer or element within your composited shot might contain many plug-in effects from external programs or from effects within the main program itself.
Pre-vis — A moving storyboard. Pre-vis shots can be produced within the same 3D animation package that you’re using for the main animation work. It’s a crude, blocky version of the shots to show other crew members and actors what the shot currently being filmed will look like (and its timings). They can then be handed over to the editor to roughly cut together the film and get a sense of the timings required.
When you’re working with new technology or a new process, no doubt you’ll find yourself a little out of your comfort zone. This is the best way to learn. You might not pull it off, it might look fake, or you might end up scrapping it altogether, but at least you’ll learn. Visual effects work is very time consuming, and your patience will be tested. But when it works, it’s a great feeling and could give your blockbuster an epic sense of scale.
"Making Your First Blockbuster" is due to be released early 2019 from Michael Wiese Productions.