Tutorial: iso-photographic rendering
This tutorial will guide you through the process of iso-photographic rendering.
What is Iso-Photography ?
An iso-photograph ("iso" formerly "equal to") is a computer simulated image that is virtually equal to a photograph. A photograph and iso-photograph, viewed side by side, cannot be differentiated by an human eye.
An iso-photograph is an objective prediction of the appearance of objects and scenes as they would be captured by a digital camera.
Beyond the magic, taking iso-photographs allows you to make the right decision about the choice of materials for your prototypes. Iso-photographic simulation is also a key step in the validation process of the accuracy of your rendering software. Last but not least, video compositing with iso-photograph renderings is more natural and does not require any adjustment effort.
iso-photograph
photograph
Taking reference pictures
Iso-photography process requires different kinds of reference pictures.
The first kinds are reference pictures used to determine the intrinsic parameters of the camera : the focal length, field of view and distortion :
It is recommended to take between 3 to 20 pictures of a flat chessboard where it appears in whole with as many different sizes, positions and orientations as possible.
The exposure of the chessboard should be correctly set and it is better to avoid any lighting variation such as the projection of shadows by other elements across its surface in order to facilitate its detection.
These reference pictures do not need to be taken with the same lighting condition as the scene.
These reference pictures must be taken with the same camera and lens settings (focal length, zoom, focus distance, aperture) that will be used.
The second kinds are reference pictures used to determine the camera position :
The same chessboard and camera (with its lens) used for the intrinsic calibration must be used and the camera position must be exactly the same as in the final image for which one wants to perform the iso-photography.
The chessboard must appear free of occlusions by any other element in the scene.
If multiple camera positions are needed, this reference picture of the chessboard must be taken each time the camera is moved.
The exposure of the chessboard should be correctly set.
Beware that the chessboard is better detected facing the camera and may probably not be detected at a shallow angle.
Calibrate the camera with Predictive Cam (standalone)
Calibrate the camera and export the .intrinsic, .extrinsic and .chessboard files from Predict Cam Standalone
Intrinsic camera properties
In order to find and export the camera intrinsic parameters from a series of photographs with Predictive cam (standalone):
Run ∽camera.exe
Choose Track Object -> Chessboard.
Set the settings of the chessboard (width, height, edge length, margin, depth).
Choose Resize -> No in order to use the full image resolution.
Open a set of images. A number of images between 3 to 12 is needed to get a fine calibration: The more the better. The pictures should be focused on the chessboard with no motion blur so that the edges appear sharp and the borders are fully visible. The viewpoints of the chessboard should cover the whole field of view with multiple inclinations and rotations.
Press Camera View in order to see the superposition of the virtual chessboard over the picture.
Before calibration, the alignment is not right.
Go through each image using the Next Frame button (or the right arrow) and press Push to Buffer. If the chessboard is correctly detected, the Buffer Size is incremented and the alignment between the virtual chessboard and the picture is updated.
Save the calibration data to an .intrinsic file using Save Intrinsic in the File section.
Extrinsic camera properties
In order to find and export the camera extrinsic parameters from a photograph with Predictive cam (standalone):
Run ∽camera.exe
Load an intrinsic calibration file using Load Intrinsic (see the previous section on how to compute the intrinsic parameters if needed).
Choose Track Object -> Camera so that the position of the chessboard is fixed and the camera is moving.
Set the following settings of the chessboard: width, height, edge length, margin, depth.
Open the image Camera -> Stream -> Single Image. Once loaded, the virtual chessboard should coincide with the real chessboard. You may have to make the plane of the image closer or further by changing the value of Image Plane Distance. If the chessboard is not detected, try changing the resolution of the input image in Tracking-> Resize or toggle the Tracking-> Fast checkbox.
Load a mesh representing the scene using File -> Import Geometry and place the virtual chessboard at the correct position in the virtual scene. Note that only OBJ format is supported. From the camera viewpoint, the geometry of the scene should coincide with the picture.
Save the extrinsic parameters using Save Extrinsic as well as the settings of the chessboard using Save Chessboard.
Export the camera to the UVR format using Export Camera.
Import the Predictive Cam Standalone camera in Unity
Once you have exported the camera from Predictive Cam (standalone):
Open your Unity project,
Create a new scene for your iso-photography setup,
Import the intrinsic camera calibration : add the UVR Intrinsic Calibration component to the camera, and give it a reference to the .intrinsic file that was generated by Predict Cam Standalone,
Import the extrinsic camera calibration : right click on the camera GameObject, menu PredictSuite/Transform/Import Extrinsic Matrix, give a reference to the .extrinsic file and import the Transform. NB : you must select the option "Convert to Y up",
To make sure the alignment is correct, you can add a reference plane to the camera, see section "Place the geometries precisely in the scene" bellow.
Import the geometries in Unity
To create an iso-photography, you must have all the objects and lights modelized in 3D in your scene.
Import all the corresponding geometries in Unity and place them roughly in the scene you created. Make sure you don't move the camera in the process.
The chessboard used for the camera calibration can be imported differently, especially if you have access to the .chessboard file from Predict Cam Standalone :
Instantiate the chessboard GameObject by right clicking anywhere in your Unity scene and selecting the menu "PredictiveSuite/Camera/Instantiate Chessboard",
The opened Chessboard Editor enables you to create a geometry with a custom chessboard. The chessboard should match the one you used to take the reference pictures with your camera. Import the chessboard settings using the button on the top right corner, or fill in the required fields :
the Square Count field is the number of squares (width*height) in the chessboard,
the Square Dimension field is the dimension of one square (width*height) in Unity units,
the Margin Dimension field is the dimension of the white margin around the chessboard (X is left, Y is right, Z is bottom, W is top) in Unity units,
the Origin Color field is the color of the square in the bottom left corner,
the Origin outside the margin field defines whether the origin (red dot on the preview) is at the very outside of the mesh, or if it is at the corner of the first square,
the Depth field is the actual depth of the geometry that will be created, in Unity units.
Click the "Create Chessboard" button to create and instantiate the geometry in your scene.
Place the chessboard in the 3D scene : if you have the .chessboard file for your current setup, you can use the same method as to place the camera in the scene (right click on the GameObject, menu PredictSuite/Transform/Import Extrinsic Matrix, give a reference to the .chessboard file)
You can also create a chessboard manually with your own geometry and texture, but you must make sure that the origin of your GameObject is placed on a white corner of the chessboard
Place the geometries precisely in the scene
Add the reference image as a backplane
To be able to place the geometries precisely in the scene, import the reference picture and place it as a transparent overlay in front of your camera :
Import the reference picture in your Unity project.
Right-click on your camera in the "Hierarchy" tab, menu "PredictiveSuite/Camera/Add reference picture plane".
In the selection window, select your reference picture (the reference picture must be in the Assets folder).
Position the geometries
Once the overlay is ready, move your geometries in the scene to fit precisely their position in the reference picture.
You can change the transparency of the plane by changing the "Color" field on the "Image" component.
When all the geometries are placed precisely, make sure to disable the Canvas GameObject. Otherwise it will be visible when you start the Predictive Engine render.
Set the right materials
Measurement
All materials and objects in the scene shall to be measured to output iso-photographic image of high quality and accuracy:
Glass and metal-like materials are usually measured using an ellipsometer which outputs spectral complex index of refraction.
Measures of optical indexes for standard glass and metallic materials like BK7 glass can be obtained from databases but may not correspond to actual materials which can sometimes include additional coatings (for protection for instance).For opaque specular and diffuse materials, it is the reflective properties of the material (the BRDF) that shall be measured. This property is usually measured with a specific device like a goniometer.
It is recommended to use measurements that are as close as possible to the true materials used in the scene.
It is strongly recommended to use spectral measurement instead of RGB to gain in accuracy.
See the Physics section for more details on spectral rendering.
Association
In Predictive Engine, a material defines the separation between two mediums on each side of the mesh the material is applied on.
When an object is made of multiple parts with different materials, its mesh must be separated into separate sub-meshes such that each sub-mesh is associated with a single material.
In the specific case of a Dielectric of Conductor partially contained within another Dielectric, the surfaces of the meshes should be cut along the change of mediums. For instance, a partially filled glass of water needs 3 materials and sub-meshes for each combination of mediums between air, water and glass: air-water, air-glass, water-glass.
It is the direction of the normal of the surface that indicates the external medium. See the detailed documentation for more details on normals.
Some material require properly defined texture coordinates on the mesh to function as expected.
Set the right lights
Lights are usually built from AreaLight with a specific mesh as a support for the shape of the light source. They need to be positioned and oriented in the same way as their real counterpart and be associated with the correct material.
The measurement of light properties encompasses their spectral and angular distributions. It is of extreme importance that it is done accurately since it has a large impact on the colors and brightness of the rendered scene.
Rendering the iso-photography
The last step before starting the rendering is to define the camera as a Photographic Camera or Bayer Camera and use the measured response curves of the camera sensor for the channel R, G and B. Once your scene is ready, you can start the application to get the Predictive Engine render. Don't forget to set the correct resolution and aspect ratio in the Unity Game tab.