Adverti horiz upsell
A Designers Tutorial in using Image Based Illumination and High Dynamic Range Imagery
A Designers Tutorial in using Image Based Illumination and High Dynamic Range Imagery
Theremina, updated 2006-12-05 17:20:22 UTC 74,960 views  Rating:
(4 ratings)
Page 1 of 2

A Designer's Tutorial in using Image Based Illumination and High Dynamic Range Imagery


Brock J. Stearn

Department of Industrial, Interior and Visual Communication Design
The Ohio State University


Introduction

This tutorial follows a pipeline using Image Based Illumination and High Dynamic Range Imagery to more effectively composite a CG mock-up into a live action photograph. For example the mock-up camera seen here is composited into a lobby. On the left is the typical way a designer will composite a mock-up into a photograph by lighting the model with basic lighting and then using Photoshop to place it into the scene. On the right is the same camera composited into the same photograph using this pipeline.
Figure 1

Figure 1

click for larger version
Comparison of a result from the exisiting pipeline (left) to the result of this proposed pipeline (right)

This pipeline is specific to Alias Maya 5.0 or higher, but could be adapted with some effort into other software packages. Also, there will be references to my MFA thesis paper on this pipeline and related work throughout this tutorial. The page references listed in this tutorial are to that document.


This pipeline is specific to Alias Maya 5.0 or higher, but could be adapted with some effort into other software packages. Also, there will be references to my MFA thesis paper on this pipeline and related work throughout this tutorial. The page references listed in this tutorial are to that document.



Software/Downloads
For this tutorial there are certain available pieces of software, plug-ins, and scripts that I will be referencing. Below are links to download them and directions as to where to install them.


* HDRShop - Install to any directory you like.
* Lightgen - Install to your HDRShop/plugins directory
* Brock's HDRSetup Control - Extract to your current version of Maya's script directory
* Brock's Lightgen Control - Extract to your current version of Maya's script directory


Note: This tutorial includes instructions for Adobe Photoshop 7.01 and Alias Maya 5.0 (which includes Mental Ray) which are not available for free.



"On Set" Capturing
The first step in this process is to find a location to take photographs. This location should ideally be where you could see your CG mock-up existing realistically. Once this location has been found you will need to take some photographic reference of this area. The best way to do this is with a digital camera and a reflective sphere, such as a gazing ball.


Camera Setup
Place the reflective sphere where the computer generated mock-up will later be composited in. Find the desired angle of the camera and mark this spot (pp 36). Now rotate from that spot in 45-50� each direction and these two spots will be where you will take the photographs of the reflective sphere (pp 37-39). The reason we are shooting two different angles is so that they can be composited together to make one whole image without any loss of environmental information, or any objects in the images, such as the camera and camera person (pp 42).

Framing the Sphere
It is important to get as much information from the reflective sphere as possible without allowing the camera, and you, to take up a large part of the sphere. The best way to do this is to fully zoom in on the reflective sphere and then manually walk the camera towards the sphere so that the sphere takes up 30-50% of the image (pp 37-38).



Figure 2



Photograph showing Sphere taking up between 30-50% of the image



Figure 3





Range of Images

In order to use High Dynamic Range Imagery (HDRI) we want to take a full exposure range of images of the sphere. So adjust the camera's shutter speed setting and take one photo at each setting from the fastest shutter speed, such as 1/2000, to the slowest, such as 4. All of these exposure images will be later compiled into one image that more accurately captures all of the lighting information of the environment (pp 20). Below is a cropped example of photographs of the reflective sphere taken through a range of exposures.


Figure 4

click for larger version

The full exposure range of photographs of the reflective sphere in the lobby



Matching the Angles
As both angles' range of images are going to be compiled together it is very important that each angle's images were taken from the same distance from the sphere with the same settings. A very simple way to do this it to tie a string to the base of the sphere. Pull this string back to where your first angle is and when it reaches the base of the tripod either mark or cut the string there. When you are ready to shoot your second angle use the string to pivot around the base and set the tripod at the same point on the string at the other angle. Make sure you take a few photos at the center line which you can use later for compositing (pp 36).




Figure 5



Illustration of Shooting two angles while pivoting from a center line with a string tied to the base of the Gazing Ball


Assembling the images
Now that all the photographs are taken we need to assemble them so that we can use them in our 3D and Rendering software. To do this we need two pieces of software; Adobe Photoshop (or something similar such as Gimp) and HDRShop available free for noncommercial use.


Cropping and Masking the images
We only need the sphere for our information so we need to crop the rest out and mask everything but the sphere. To do this drag a circle marquee around the sphere in the photograph and select the inverse and fill that in with total black. All of the photographs should have the sphere in an identical position in the image. Given this, it is possible to simply copy the black filled area to each image file. Once the sphere is cropped out in every image, save the images in a name you will recognize, i.e. angle1_1_2000.jpg, in JPEG format.


Figure 6


Image after sphere isolated from background. Blue lines indicate area to be cropped




Converting from Angular to Latitude-Longitude
A perfect sphere's image can be transformed into several other formats since we know the equations of a perfect sphere. The most convenient way to do this is in HDRShop. We are going to convert angle one's images to Latitude-Longitude, or planar, format directly. The other set we are going to do the same process for angle 2 but offset by 90� so that we can composite the two together (pp 41).


For Angle 2:

* Import your Angle 2 images in HDRShop
* For each Angle 2 image click Image - Panorama - Panoramic Transformations


Figure 7


Panoramic Transformation in HDRShop


o Leave format as mirrored ball, change destination format to latitude/longitude
o In the 3d rotation section, click Arbitrary Rotation and then click Settings
o Set Y axis to 90, leave the others at 0
o Press OK on both windows

Your settings should look like those below:

Figure 8


Panoramic Transformation Settings for Angle 2

Figure 9


Arbitrary Rotation Settings for Angle 2


For Angle 1:

* Repeat the same steps as above but set your Arbitrary Rotation to 0 for all axises



See a Video of this Step




Compositing the two sets of images together.

This next step is perhaps the longest process but it is necessary to create a very nice lightprobe. I used Adobe Photoshop for this process but any similar program that supports multiple layers and layer locking should work.

Bring each exposure of angle 1 into Photoshop and copy each image into a new Photoshop file, labeling the layers accordingly. Repeat the same process for Angle 2 but places those images in a seperate Photoshop file. Bring in your complete image range for Angle 1 into a Photoshop file with a layer for each step in the range. Repeat the same for Angle 2.

Now look at the areas of Angle 1 that need to be removed. This is usually the camera, cameraman, whatever is holding up the reflective sphere, and any black areas (which is where you didn't capture any information in the photographs) You need to find those areas in Angle 2 and copy them to Angle 1 and transform them into the right location and rotation (Figure 10). The reason for doing this in Photoshop with multiple layers is that you can lock your layers together after you paste in each part. This allows everything you do to one layer to affect all the other linked layers. Below are the angles shot from the steps above and the final image composited together. Since this image will be just used for illumination purposes the images do not have to look precisely accurate as long as the local area of colors is still intact. For these images below you will notice the indoor window's edge in the middle of the image doesn't match properly but it will be enough for a IBI file. Once all the angles have been combined to create a better image, save out each exposure into its own JPEG file and once again name it accordingly (pp 40).
Figure 10



Angle 1 (top) and Angle 2 (bottom) after being converted to Lat/Long. Boxes indicate areas used from Angle 2 to fix and complete Angle 1.


Figure 11
click for larger versionLat/Long image after compositing. There are still some minor differences in the corrected areas but since this will get blurred at rendering it isn't critical.




Creating a High Dynamic Range Image (HDRI)
The best application I have found for creating HDRI is HDRShop, but there are other utilities available such as MkHDR a command line utility, and HDRIE and Photogenics HDR which are both commercial applications with some minor image editing capabilities. For our purposes we will be continuing to use HDRShop (pp 41-42).

Creating an HDRI

* Launch HDRShop
* From the Create menu choose Assemble HDR from Image Sequence
* Click Load Images and load in your newly created range of fixed JPEG (.jpg) images.
* Your images should automatically be sorted correctly, if not then manually adjust their order so that the exposures are in the correct order.
* Click the Calculate button so it automatically adjusts the exposure difference
* Once that is complete click Generate Image and you will have a HDRI
* If your new HDR image seems dark then use the + or - to find what should be a new middle ground for your exposure range. Once this is found go to Image - Pixels - Scale to current Exposure and the HDR image will be adjusted.
* If needed you can now convert your image to the type of map you need by going to Image - Panorama - Panoramic Transformations and select the appropriate pull down menus. It is possible to create a Latitude/Longitude, Cubic, and Angular maps (Figure 12). For this tutorial we will be using a Latitude/Longitude map so create one of those.
* Save a Latitude/Longitude HDRI in HDR format and a compressed HDR Iin JPEG format.

Figure 12


Latitude/Longitude Map of Composited image

Cubic Map of Composited image

Angular Map of Composited image


How Image Based Illumination works


Principles
The basic principle behind Image Based Illumination is that a computer generated model is surrounded by another piece of geometry (i.e. sphere, cube), with an image mapped onto that geometry. To calculate the lighting of a particular point of the computer genereated model, IBI takes extends the rays, usually based on the vector of the normal (N), until it intersects with the surrounding surface. It then samples the color at that particular point on that surrounding surface's mapped image and uses that for the lighting color of the point where the vector started. Below is an example of this occuring with several rays.

Figure 13


Illustration of the principle behind Image Based Illumination



In Maya and Mental Ray

For Mental Ray, which comes bundled in with Maya 5.0, IBI is implemented through creating some surrounding geometry and using the Mental Ray Final Gathering technique. This is the most straight forward method and what will be used in this tutorial. There are some scripts I have written or modified for this tutorial which have been written for Maya 5.0 but should be useable in any later versions as long as Mental Ray is installed and the plug-in is loaded as shown below. The plug-in can be loaded by going through Windows -Settings - Plug-in Manager and checking load box for the Mayatomr.mll plugin.

Figure 14

Mental Ray Plug-In


Setting Up for IBI

More work in HDRShop

There are two more steps in HDRShop that we must complete before moving onto Maya. The first is to open your compiled HDR image and set the exposure to 0 by pressing 0. If the image appears dark this is probably due to a Gamma difference from the camera to the application. We will solve this through a script in Maya but it is important to note whether the image is bright or dark at 0 exposure. The second step is to use the Lightgen plug-in for HDRShop on our HDR image. Make sure the plug-in is in the HDRShop plug-in directory (pp 44-45).


Lightgen

* Launch HDRShop
* Open your composited HDR image
* Blur the image by going to Image - Filter - Guassian Blur
* The image will need to be shrinked for Lightgen so from the Image menu choose Arbitrary Resize


Figure 15

Arbitrary Resize in HDRShop

Enter 128 and 64 for the new size
Figure 16

New Image Resize numbers for Lightgen

* Once the image is resized go to the Plugins menu choose Lightgen

Figure 17

Launching the Lightgen plugin



* Enter an Output filename (Browse to your desired location) and set the rest of the settings to the image below
Figure 18

Lightgen GUI plugin in HDRShop


* Click Execute and Lightgen will think for a short bit as it is creating the output
* If the plug-in was successful you should receive the following message

Figure 19


Successful output in Lightgen


* The final result should look like a bunch of colored dots in a black field

Figure 20

Visual output from Lightgen

* If you receive the following error it is most likely you forgot to resize your image before launching Lightgen

Figure 21

Lightgen plugin error in HDRshop

* Lightgen is also useable through the command line. Type Lightgen.exe to list the different flags

Making a MAP image

Mental Ray's native image file format is MAP. We are going to convert our HDR image to MAP to speed up render times in Mental Ray and to eliminate the Range Clamping that Maya does with HDR images.

hdr2map

* Open a command shell
* Navigate to where your HDR file is located
* Use my hdr2map batch file to convert your map
* Useage: hdr2map filename.hdr filename.map
* A MAP file will be created in that directory for you to use

Working in Maya

Basic Workflow
There are four basic steps in this pipeline for using Image Based Illumination and High Dynamic Range Imagery. The first is using the HDR image to provide the basic low level lighting, such as ambience and incandescence. If the object is reflective it can also be mapped into the specular channels. The second step is to use the HDR image to derive actual lights in Maya and to set them up so that they are convincing. The third step is specific to any special effects or elements the model may have. This may include Caustics, Fog, Sub-Surface Scattering and so on. The final step is to "dirty up" the image so it doesn't look so real and can blend back in with the live action image. The two scripts available in this tutorial greatly speed up this process. It should be noted that throughout this process it will be neccessary to move back and forth between the different stages to produce a great result.


Installing Scripts

Both scripts should be downloaded to your Maya's script directory. I recommend making shelf buttons for both scripts so that they can be easily accessed. In Maya open the Script editor and inside the script editor Open the brock_lightgen.mel script. The contents of the script will be displayed in the Script Editor window. Highlight the all of the text and drag, using the middle mouse button, to the Shelf and release. A new icon will be created that will launch Brock's Lightgen Control script. For Brock's HDRSetup Control script type brock_hdrsetup.mel into the command line and drag that text to the Shelf to create a new icon.


Figure 22
click for larger version
Placing brock_lightgenconrol.mel into the Shelf

Figure 23

Command Line entering of brock_hdrsetup.mel

Step 1: Using just HDR through IBI for general lighting

* Launch Maya
* Open a blank scene with just your model (or import your model into a new scene) and place the model as close to 0,0,0 as possible
* Load Brock's HDRSetup script
* A Create HDRSetup window will appear. Size the Sphere so that it will completely surround your object but not so that it is so far away. The closer it is the less the render time can be with Final Gathering. We also want a Full Sphere for our setup.
Figure 24

IBI Sphere setup


* After the Sphere is created you will be prompted with a new HDRI Setup menu
* Click the Browse button and locate your MAP file. I recommend placing this MAP file in your sourceimages directory of the active project in case you are using a distributed renderer.

Figure 25

Loading a MAP image

* After the file has been loaded you can now start affecting the MAP image through the HDRI Setup Controls panel
Figure 26

Brock's HDRI Setup Script

* Remember earlier when I mentioned to jot down what the gamma is of your image/camera. Enter that number in the Gamma Correct boxes. If needed you can skew the gamma to one channel or another since the correction is broken down to Red, Green, and Blue.
* The Expose Increaser and Expose Decreaser sections allow you to fake the exposure difference in the HDR image. This normally doesn't need to be touched unless you are trying to give a brighter or darker look then the original image.
* Reflection Energy controls how much the surrounding IBI geometry will continue to reflect back the image, the default value is usually fine.
* Before rendering the image should be blurred slightly since we are just using this image to generate the fill light for the object. In the Blur slider, a setting of1 results in no effect.
* If you want to resize the IBI sphere, use the Radius slider
* If you render now, you should receive a blank scene. This is because we need to setup Mental Ray accordingly.



* Click the Initialize Mental Ray Setup button
* Open up the Render Globals and click on the Mental Ray tab
* The Final Gather tab should already be opened

Figure 27

Final Gather settings in Mental Ray

* These values have just been initialized through the HDR Setup script. If you render now you will actually see your model being lit. However, the Max Radius needs to be tweaked to be just a bit larger then the radius of your surrounding sphere. The Final Gather Rays can also be increased to produce a better result, but that will also increase the render time.
* Render the image and continue to adjust these various settings until you have a good amount of fill lighting such as the image below
* In order to provide a visual example of where the placement of the image on the surrounding image is the MAP file must be replaced with a JPEG file. Launch the HDRSetup Control script again and open the JPEG file. Before rendering, the MAP file should be loaded back in.

Figure 28

First step in Maya (left) and the resulting rendered image (right)