Tutorial: iso-photographic rendering

This tutorial will guide you through the process of iso-photographic rendering.

What is Iso-Photography ?

An iso-photograph ("iso" formerly "equal to") is a computer simulated image that is virtually equal to a photograph. A photograph and iso-photograph, viewed side by side, cannot be differentiated by an human eye.


An iso-photograph is an objective prediction of the appearance of objects and scenes as they would be captured by a digital camera.

Beyond the magic, taking iso-photographs allows you to make the right decision about the choice of materials for your prototypes. Iso-photographic simulation is also a key step in the validation process of the accuracy of your rendering software. Last but not least, video compositing with iso-photograph renderings is more natural and does not require any adjustment effort.

iso-photograph

photograph

Taking reference pictures

Iso-photography process requires different kinds of reference pictures.

The first kinds are reference pictures used to determine the intrinsic parameters of the camera : the focal length, field of view and distortion :


  • It is recommended to take between 3 to 20 pictures of a flat chessboard where it appears in whole with as many different sizes, positions and orientations as possible.


  • The exposure of the chessboard should be correctly set and it is better to avoid any lighting variation such as the projection of shadows by other elements across its surface in order to facilitate its detection.


  • These reference pictures do not need to be taken with the same lighting condition as the scene.


  • These reference pictures must be taken with the same camera and lens settings (focal length, zoom, focus distance, aperture) that will be used.

The second kinds are reference pictures used to determine the camera position :


  • The same chessboard and camera (with its lens) used for the intrinsic calibration must be used and the camera position must be exactly the same as in the final image for which one wants to perform the iso-photography.


  • The chessboard must appear free of occlusions by any other element in the scene.


  • If multiple camera positions are needed, this reference picture of the chessboard must be taken each time the camera is moved.


  • The exposure of the chessboard should be correctly set.


  • Beware that the chessboard is better detected facing the camera and may probably not be detected at a shallow angle.

Calibrate the camera with Predictive Cam

Calibrate the camera in Predictive Cam for Unity

Once you have taken all the required references pictures, you can calibrate the camera within Unity by following the next steps :

  • Create a dedicated folder in your Unity project and import all the camera reference pictures,

  • Instantiate the chessboard GameObject by right clicking anywhere in your Unity scene and selecting the menu "PredictiveSuite/Camera/Instantiate Chessboard",

  • The opened Chessboard Editor enables you to create a geometry with a custom chessboard. The chessboard should match the one you used to take the reference pictures with your camera. Fill in the required fields :

      • the Square Count field is the number of squares (width*height) in the chessboard

      • the Square Dimension field is the dimension of one square (width*height) in Unity units

      • the Margin Dimension field is the dimension of the white margin around the chessboard (X is left, Y is right, Z is bottom, W is top) in Unity units

      • the Origin Color field is the color of the square in the bottom left corner

      • the Depth field is the actual depth of the geometry that will be created, in Unity units

  • Click the "Create Chessboard" button to create and instantiate the geometry in your scene.

  • Place the chessboard in the 3D scene.


NB : you can also create a chessboard manually with your own geometry and texture, but you must make sure that the origin of your GameObject is placed on a white corner of the chessboard

  • Right click on the camera in your Unity scene (or create a new one if there is no camera in the scene) and select the menu "PredictiveSuite/Camera/Calibrate Camera (intrinsics)",

  • In the "Camera Intrinsic Calibration" window, fill in the required fields :

      • the Camera field should already be set with the camera you selected,

      • the Use Fish Eye field should be enabled if the camera you want to calibrate uses a fish eye lens,

      • the Chessboard field should contain the chessboard GameObject,

      • the Chessboard square count field should contain the number of squares in the chessboard (width*height),

      • the Square dimension field should contain the dimension of one square of the chessboard in Unity units,

  • List all the reference pictures you've imported in the Reference Pictures field (you can either list them one by one or select them all in the Project tab and drag them onto the "Reference Pictures" label),

  • Click on "Calibrate Camera" : the intrinsic settings of the camera (field of view, optical center, focal length) are computed and directly set on the camera,

  • Right click on the camera in your Unity scene and select the menu "PredictiveSuite/Camera/Position Camera with chessboard (extrinsics)",

  • In the "Camera Extrinsic Calibration" window, fill in the required fields :

      • the Element to place field defines whether the Camera should be placed relative to the chessboard or the Chessboard should be placed relative to the camera (the selected element will be moved in the scene, the other one will stay in place),

      • the Camera field should already be set with the selected camera,

      • the Chessboard field should contain the chessboard GameObject,

      • the Chessboard square count field should contain the number of squares in the chessboard (width*height),

      • the Square dimension field should contain the dimension of one square of the chessboard in Unity units,

      • the Add Reference Plane field defines whether the reference picture should be placed as a transparent overlay in front of the camera once the calibration is done (this is required to be able to place the geometries in the scene in the following steps),

  • Set the right picture in the Reference Picture field,

  • Click on "Calibrate Camera" : the position of the camera and the chessboard are computed and applied directly in the scene,

You can ignore the two next sections ("Calibrate the camera in Predictive Cam (standalone) and export the calibration in Unity" and "Import the Predictive Cam (standalone) camera in Unity").

Calibrate the camera in Predictive Cam (standalone) and export the calibration for Unity

Intrinsic camera properties

In order to find and export the camera intrinsic parameters from a series of photographs with Predictive cam (standalone):

  • Run camera.exe

  • Choose Track Object -> Chessboard.

  • Set the settings of the chessboard (width, height, edge length, margin, depth).

  • Choose Resize -> No in order to use the full image resolution.

  • Open a set of images. A number of images between 3 to 12 is needed to get a fine calibration: The more the better. The pictures should be focused on the chessboard with no motion blur so that the edges appear sharp and the borders are fully visible. The viewpoints of the chessboard should cover the whole field of view with multiple inclinations and rotations.

  • Press Camera View in order to see the superposition of the virtual chessboard over the picture.

  • Before calibration, the alignment is not right.

  • Go through each image using the Next Frame button (or the right arrow) and press Push to Buffer. If the chessboard is correctly detected, the Buffer Size is incremented and the alignment between the virtual chessboard and the picture is updated.

  • Save the calibration data to an .intrinsic file using Save Intrinsic in the File section.


Extrinsic camera properties

In order to find and export the camera extrinsic parameters from a photograph with Predictive cam (standalone):

  • Run camera.exe

  • Load an intrinsic calibration file using Load Intrinsic (see the previous section on how to compute the intrinsic parameters if needed).

  • Choose Track Object -> Camera so that the position of the chessboard is fixed and the camera is moving.

  • Set the following settings of the chessboard: width, height, edge length, margin, depth.

  • Open the image Camera -> Stream -> Single Image. Once loaded, the virtual chessboard should coincide with the real chessboard. You may have to make the plane of the image closer or further by changing the value of Image Plane Distance. If the chessboard is not detected, try changing the resolution of the input image in Tracking-> Resize or toggle the Tracking-> Fast checkbox.

  • Load a mesh representing the scene using File -> Import Geometry and place the virtual chessboard at the correct position in the virtual scene. Note that only OBJ format is supported. From the camera viewpoint, the geometry of the scene should coincide with the picture.

  • Save the extrinsic parameters using Save Extrinsic as well as the settings of the chessboard using Save Chessboard.

  • Export the camera to the UVR format using Export Camera.


Import the Predictive Cam (standalone) camera in Unity

Once you have exported the camera from Predictive Cam (standalone):

  • Open your Unity project.

  • Create a new scene for your iso-photography setup.

  • Delete the default camera and light in the scene.

  • Import the camera via the menu PredictiveSuite/Import/Camera”. The camera will be imported in the current scene with all the required assets (observers).

      • The camera should be stored in a *.xml file containing the <Camera> tag only or in a *.osf file containing the <OmenEngine> tag only.


NB: Usually, Predictive Engine standalone scenes and Predictive Cam cameras are defined with a Z-up convention: the Z axis is used as the up vector in the coordinates system. In Unity, the convention is to use the Y axis as the up vector. If your camera was created with a Z-up convention, you should enable the “Convert to Y up” option.

For optimization, make sure you delete or disable all the other cameras in the scene.

Import the geometries in Unity

To create an iso-photography, you must have all the objects and lights modelized in 3D in your scene.

Import all the corresponding geometries in Unity and place them roughly in the scene you created. Make sure you don't move the camera in the process.

Place the geometries precisely in the scene

Add the reference image as a backplane

You can ignore this section if you already added a reference picture while calibrating the camera with Predictive Cam.

To be able to place the geometries precisely in the scene, import the reference picture and place it as a transparent overlay in front of your camera :

  • Import the reference picture in your Unity project.

  • Right-click on your camera in the "Hierarchy" tab, menu "PredictiveSuite/Camera/Add reference picture plane".

  • In the selection window, select your reference picture (the reference picture must be in the Assets folder).

Position the geometries

Once the overlay is ready, move your geometries in the scene to fit precisely their position in the reference picture.

You can change the transparency of the plane by changing the "Color" field on the "Image" component.


When all the geometries are placed precisely, make sure to disable the Canvas GameObject. Otherwise it will be visible when you start the Predictive Engine render.

Set the right materials

Measurement

All materials and objects in the scene shall to be measured to output iso-photographic image of high quality and accuracy:

  • Glass and metal-like materials are usually measured using an ellipsometer which outputs spectral complex index of refraction.
    Measures of optical indexes for standard glass and metallic materials like BK7 glass can be obtained from databases but may not correspond to actual materials which can sometimes include additional coatings (for protection for instance).

  • For opaque specular and diffuse materials, it is the reflective properties of the material (the BRDF) that shall be measured. This property is usually measured with a specific device like a goniometer.

It is recommended to use measurements that are as close as possible to the true materials used in the scene.

It is strongly recommended to use spectral measurement instead of RGB to gain in accuracy.

See the Physics section for more details on spectral rendering.

Association

In Predictive Engine, a material defines the separation between two mediums on each side of the mesh the material is applied on.


When an object is made of multiple parts with different materials, its mesh must be separated into separate sub-meshes such that each sub-mesh is associated with a single material.


In the specific case of a Dielectric of Conductor partially contained within another Dielectric, the surfaces of the meshes should be cut along the change of mediums. For instance, a partially filled glass of water needs 3 materials and sub-meshes for each combination of mediums between air, water and glass: air-water, air-glass, water-glass.

It is the direction of the normal of the surface that indicates the external medium. See the detailed documentation for more details on normals.

Some material require properly defined texture coordinates on the mesh to function as expected.

Set the right lights

Lights are usually built from AreaLight with a specific mesh as a support for the shape of the light source. They need to be positioned and oriented in the same way as their real counterpart and be associated with the correct material.

The measurement of light properties encompasses their spectral and angular distributions. It is of extreme importance that it is done accurately since it has a large impact on the colors and brightness of the rendered scene.

Rendering the iso-photography

The last step before starting the rendering is to define the camera as a Photographic Camera or Bayer Camera and use the measured response curves of the camera sensor for the channel R, G and B. Once your scene is ready, you can start the application to get the Predictive Engine render. Don't forget to set the correct resolution and aspect ratio in the Unity Game tab.