13. Projector Simulation

Learn how to use Projector Simulation to set up virtual projectors in the Stage level, specify projector and lens equipment, and utilise the manual and automatic calibration tools to fine tune the setup onsite.

13.1 Projector Simulation overview

How does Projector Simulation work?

In d3’s visualiser, you can create and place virtual projectors. The virtual projector object shares the same properties as a physical projector such as resolution, position, rotation, aiming-point and lens size.

The content is being applied directly onto the object as onto any other video surface, independent of where the physical projectors are placed (however, the resolution is very much based on the physical amount of projectors and their location).

d3 used to visualise, simulate and calculate how physical projectors should be positioned for the Bentley Continental GT UK Tour

Note that the Virtual projectors do not project content onto the projection surface. Quite the opposite, the Virtual projectors render the content applied onto the screen from their respective point of view, sub-sampling it to whatever resolution is being set and sending the respective image to the physical projectors, up to 60 frames/sec.

Simply speaking, a Virtual projector behaves like a camera.

If the physical projector has to move a couple of meters to the side, there is no need to re-render any content, just move the Virtual projector correspondingly. It will then render a new image from the point of view of the new position. The actual fine-tuned onsite configuration of the Virtual projectors is done through a manual or automatic calibration process.

Outputs that the three virtual projectors from the above image see in the Output Feeds level

Note that the content creation is now completely separate from the output sent to the physical projectors. The content only needs to be made for the 3D model while the output is based on the properties of the Virtual projectors. There is therefore no need to re-render any content.

Projector Simulation as a decision-making tool

Projector simulation does not only have to be used for output purposes. During the early production stages, Projector Simulation can be used to explore projector positions, aiming points, appropriate lense sizes etc; it is a tool for specing the correct hardware for a show.

3D models preparation

As always when working in d3, you need to model an exact virtual replica of the real-world screens. If only standard rectangular screens are being used, these can be created directly in d3. However, for more complex three-dimensional objects, it is required that the screen is buily and exported from a 3D software such as 3ds Max, Maya, Cinema4D or similar. It is crucial to not only model and export the 3D models, but to also properly texture-map (UV-map) them and to carefully optimize the polygon count of the model.

The more accurate the 3D model the better. The level of accuracy usually depends on the distance between the audience and the object. For building projections, it is recommended to laser scan the building. A laser scan can generate a highly accurate 3D mesh (+-5mm accuracy) that you can import into a 3D software application from which the optimization, UV-mapping and content template creation process can take place. The mesh is then exported as a texture mapped .obj file into d3.

For smaller objects, where the audience stands much closer to the object and the pixel density is higher (for example for car projections), it is even more important that the 3D mesh is as accurate as possible. However, as this usually is not the case, d3 has a number of 2D-based warping tools allowing you to compensate for the inaccuracies between the virtual and the real-world model. Please see the sub-chapter Warping outputs for more information.

Content creation

Based on 3D models, animators can create content either in 2D or in 3D, or both. For simple setups, a 2D content template generated from the UV-coordinates of the 3D model can be used as a background image in for example After Effects.

Video files are rendered and imported into d3. In d3, the content is then applied back onto the same 3D models, allowing the content to be pre-visualises in 3D and in real-time, from any point of view.

As the output to the physical projectors is handled by the virtual projectors, the animators only need to think about how to create content that looks good on the screen in d3.

How do I learn about preparing a projection-based 3D model for d3?

The 3D modelling and UV-mapping processes are currently not covered in detail in this User Guide. Please contact the d3 Technologies consultancy team for advice.

Back to top ↑

13.2 Creating/removing virtual projectors

A virtual projector is categorised as a type of screen. This makes sense from the point of view that a virtual projector, just like any other screen, uses a system to output its data, in this case the Feed system. However, this may also appear confusing since the projector renders content from its point of view, rather than having content mapped to it, and in this respect functions more as a camera. Therefore how the virtual projector is categorised will change in a future release. The advantage of this is that the processes used for creating/removing virtual projectors are the same as the other screen types. For information on the other screen types please see the Screen types overview sub-chapter.

Creating a virtual projector for a stage

Please see the sub-chapter Creating/removing screens for step-by-step instructions on how to add a screen to a stage. Select Projector instead of Screen from the menu of screen types.

  • Select Projector from the menu of screen types. The default position of the virtual projector is (0, 3, -10) and the default lookAt position is (0,3,0).

The below image shows an example of two virtual projectors added to a stage. The stage is in fact a section of the d3 studio and a wooden sculpture which will be used in this chapter to demonstrate the Projector Simulation features.

Stage Visualiser being used to visualise two virtual projectors and a sculpture in a section of the d3 studio

Removing a virtual projector from a stage

Please see the sub-chapter Creating/removing screens for step-by-step instructions on how to remove a screen from a stage.

Back to top ↑

13.3 Virtual projector properties

This sub-chapter explains the definition of the virtual projector properties. These properties can be edited using the Virtual Projector editor. To understand how to use the properties please see the sub-chapters Manual calibration and Auto-calibration.

  • Open the Virtual Projector editor by either right-clicking a virtual projector directly in the stage, or by right-clicking the virtual projector from the screens list in the Stage editor.

General properties

General virtual projector properties

Resolution

The x and y resolution of the virtual projector, set in pixels.

The virtual projector resolution has to be the same as the resolution of the output head intended to be assigned to the virtual projector.

Screens

This property refers to which video screens the virtual projector should render content from. More than one screen can be added to the screens list.

Visualiser

Colour : colour of the projector beam in the Stage Visualiser.

Render mode : displays wireframe and/or beam elements in the Stage Visualiser.

Manual calibration properties

Virtual projector manual calibration properties

Pos

The x, y and z position of the virtual projector, set in meters.

Rotation

The x, y and z rotation of the virtual projector, set in degrees. The rotation is based around its lookAt position.

LookAt

The x, y and z position of the the virtual projector’s pivot point (the base point from which it rotates around), set in meters.

Look Distance

The absolute distance between the virtual projector position and its lookAt position, set in meters.

Roll angle

ThrowRatio

This value should be the same as the lens value of the physical projector.

Pixel aspect

Setting this value to 1 (default) means that the image aspect (below) is maintained as the perfect ratio between the x and y resolution.

Image aspect

The ratio between the x and y resolution of the virtual projector.

Automatic calibration properties

Virtual projector automatic calibration properties

Create reference point

This creates a virtual reference point on the d3 video screen. You can use this property to link a virtual reference point from the video screen to its corresponding physical point.

In the d3 visualiser, the reference point is represented by a yellow semi-transparent sphere, by default snapped to the most centrally positioned vertex of the object. It is important to zoom in closely to the centre of the screen to actually see the reference points.

To re-position the reference point on the screen, move it by selecting it and moving the mouse. It will snap to the vertices of the 3D object. You should create a minimum of six reference points for each virtual projector.

Please note: this feature can only be used when lining up virtual projectors using auto-calibration, not manual calibration.

Reference points visible

Shows/hides created reference points.

Add calibration points

Adds calibration points based on previously created reference points. A calibration point is visualised as a blue blinking line linking a reference point to a virtual projector. Reference points can be re-used to create calibration points among multiple virtual projectors.

Remove calibration points

To remove a calibration point, left-click the remove calibration points button and then left-click the calibration point itself.

Begin 2d alignment

This switches the Stage level to a 2D output mode. Left-click a green crosshair (representing a reference point on the video screen) and move the white cross in the output to match it with its physical corresponding point. To move the white cross pixel-per-pixel, use the keyboard arrows. Repeat the same process for all reference points.

Near and Far planes

The planes that cut the frustum perpendicular to the viewing direction are called the near plane and the far plane. Objects closer to the virtual projector than the near plane or beyond the far plane are not drawn.

Calibrate!

Automatically calibrates the properties of the virtual projector after completion of the 2D lineup.

Revert to manual

Reverts back to the manual settings of the virtual projector before the Automatic Calibration was executed.

Reset all calibration points

Back to top ↑

13.4 Initial virtual projector setup

Before lining up the projectors using manual and auto-calibration the virtual projectors have to be basically setup and their outputs assigned to their output heads. To do this:

  • Set the resolution of the virtual projector to match the corresponding output head’s resolution.
  • Change the pos (projector position), lookAt (pivot point) and throwRatio (lens value) properties so that the virtual projector covers the required part of the screen surface. A laser measurer may be required to calculate the correct position by measuring the distance from the physical projector to the video surface.

Please note: The x, y and z positions of the virtual projector are based on the origin point (0,0,0). Therefore it is important that the 3D objects are based around the same point.

  • If the default step value is too high (or too low), right-click the property fields and change the Step values. See the previous sub-chapter Virtual Projector properties for more information on the different virtual projector properties.

Properties from the Virtual Projector editor used so that the virtual projector covers the required part of the screen surface

  • Left-click + and add the screen that the particular virtual projector is covering.
  • Switch the Stage level to the Feed level by left-clicking Feed from the d3 State editor (bar at the top of the screen).
  • Ensure all of the Feed rectangles from the virtual projector outputs have been added to the output heads. Please see the sub-chapter Virtual projector outputs for information on how to do this.

Back to top ↑

13.5 Virtual projector outputs

Configuring an output display’s resolution and frame rate

d3 uses Feed scenes to output virtual projector outputs. The output displays contained in these Feed scenes can be configured to a specific resolution and frame rate. For more information please see the sub-chapter Configuring Feed scenes.

Creating a Feed rectangle from a virtual projector

  • Open the Feed level by left-clicking Feed from the d3 State editor (bar at the top of the screen). The top half of the screen displays all of the screens in the currently active stage in the Stage level. This section is commonly referred to as the screen section. For more information on the Feed level interface please see the sub-chapter Viewing the feed level.

Please note: the size of the screen rectangles in the Screen section of the Feed level is based on the pixel resolutions of the screens and virtual projectors.

  • Right-click the first virtual projector output, then left-click add feed rectangle. This will generate an output feed from the virtual projector and place it on head 2 (by default, output head 1 is represented by the GUI). This should send a feed to the first physical projector.

Add Feed Rectangle option is used to add a Feed rectangle and generate an output feed from a virtual projector

  • Repeat this step for each virtual projector that needs to be outputted.

Back to top ↑

13.6 Manual calibration

This sub-chapter explains the process of manually calibrating (lining up) the virtual projectors to perfectly map the virtual video screens.

Using manual calibration

  • Open the Virtual Projector editor by either right-clicking it directly in the stage, or by right-clicking the virtual projector from the screens list in the Stage editor.

Manual calibration properties of the Virtual Projector editor are used to lineup a virtual projector to the video screen(s) in the Stage level

LookAt position

The LookAt position of a virtual projector defines the centre point of its corresponding output feed. When lining up a virtual projector to the physical one, it is therefore crucial to match the virtual LookAt position to its corresponding point in the real world.

d3 has a built-in test pattern feature allowing the system to generate a line-drawing based on the 3D mesh of the virtual video screen. In the same output frame, d3 will also output a red cross positioned in the centre of the output feed. When the built-in test pattern is applied, d3 will always output this red cross, regardless of the orientation of the virtual projector. Consequently, it is required that the lookAt position of the virtual projector is aligned to the centre of the red cross.

To align the lookAt position of the virtual projector to the centre of the red cross:

  • Right-click the first output head (the output of the first virtual projector) and select use test pattern. A line drawing of the 3d mesh object will appear and a red cross will be placed in the centre of the output.

Use Test Pattern option being used to output a 3d mesh of the virtual video screen

  • Ensure the test pattern is also outputted from the physical projector.

Virtual 3d mesh test pattern outputted onto a wooden sculpture

  • Return to the Stage level. Set the step value of the lookAt position to 0.01.
  • Begin aligning the lookAt position of the virtual projector to the red cross by comparing the lookAt position’s crosshair in the Stage level and the red cross being outputted from the physical projector.

LookAt position of the virtual projector now matches its corresponding point on the physical video screen, in this example a wooden scuplture

Adjust the throwRatio

  • To zoom in on the physical video screen’s content, change the throwRatio value. This value corresponds to the lens size of the physical projector.
  • If the exact lens size is known (for example a fixed lens size is being used), type it in.
  • If a zoom lens is being used, set the start value to the lowest (or highest) value in the zoom range. Slowly change the value by scrolling the mousewheel in the property field. To do this the step value may need to be changed to 0.001.

Finetune properties

  • After setting the initial lens value, try not to edit the position properties. Instead, start adjusting the rotation parameters if needed, in particular the x and y rotations. Aim to establish parallel lines mapping onto the video screen globally, rather than focusing only on one part of the screen. Adjust the step value if needed.
  • Go back and finetune the values of the lookAt position to center the output to the physical video screen. Remember to establish parallel lines.
  • Adjust the throwRatio to zoom in/out of the content on the video screen. Assuming that the 3D mesh is accurate to the physical video screen, the mapping should gradually fall into place.

Virtual projector has been manually calibrated; the 3D mesh test pattern now lines up with the wooden sculpture

If the projection surface does not match the virtual 3D model after carrying out manual calibration the lineup may need to be fine-tuned. Please see the sub-chapter Warping outputs for more information.

Back to top ↑

13.7 Auto-calibration

This sub-chapter explains the process of auto-calibrating (lining up) the virtual projectors to perfectly map the virtual video screens.

3D mesh accuracy

When lining up virtual projectors using auto-calibration it is crucial that the 3D mesh object is modelled extremely accurately to the physical model. The reason for this is that d3’s calibration algorithm assumes that the virtual reference points link exactly to their corresponding real-world points.

The best way to generate a highly accurate 3D model is to laser scan the physical model, or to laser cut the physical model based on the same 3D file you later use in d3. For building projections we recommend a mesh with a +-5mm error margin.

Using auto-calibration

  • Open the Virtual Projector editor by either right-clicking it directly in the stage, or by right-clicking the virtual projector from the screens list in the Stage editor.

Automatic calibration properties of the Virtual Projector editor are used to lineup a virtual projector to the video screen(s) in the Stage level

  • Follow the steps outlined the sub-chapter Initial virtual projector setup. When adjusting the lookAt position and the throwRatio make sure the virtual projectors are covering the full surface of the projection object. Make sure to over-throw a bit.
  • Activate the reference points visible button by left-clicking it. When activated it will be highlighted in light grey.

Create reference points

  • Zoom in on the projection object and left-click create reference point from the Virtual Projector editor. This creates a reference point on the video screen, linking a virtual reference point of the video screen to its corresponding physical point. As can be seen in the image below, the reference point is represented by a yellow half-transparent sphere in the Stage Visualiser,

Please note: in order to create new reference points it is important that you zoom in very closely to the object. This deficiency will change in a future release.

  • To re-position the reference point on the screen, select it and move it using the mouse. It will snap to the closest vertex.
  • Create a minimum of 6 reference points and place them on clearly identifiable real-world reference points. This is to make it easier to link the virtual reference points to the corresponding real-world points at a later stage in the lineup process. Usually, there is no need to create more than 6-12 reference points per virtual projector but for more complex geometry it may be necessary to create more. Some complex building mapping projects have required up to 30 points.

Reference points are represented by yellow half transparent spheres

Create calibration points

  • After creating and placing the reference points left-click add calibration points from the Virtual Projector editor. Based on the previously created reference points, calibration points associate the reference points with a specific virtual projector. The calibration points are visualised as blue blinking lines but these are only visible if the render mode of the virtual projector is set to wire or beam+wire.

Please note: multiple virtual projectors can use the same reference points. However, new calibration points have to be created for every new virtual projector when auto-calibrating.

Blue blinking lines represent calibration points which associate the virtual model’s reference points with a virtual projector

Remove calibration and reference points

  • To remove a calibration point, left-click remove calibration points from the Virtual Projector editor and then left-click the calibration point itself.
  • To remove a reference point, right-click the reference point and then select remove in the pop-up window.

WARNING: in the current release of d3 it is very important to first remove a calibration point, then remove the reference point, otherwise you may need to re-do the entire process.

Begin 2d alignment

  • To start mapping the virtual reference points to the real-world corresponding points left-click begin 2d alignment from the Virtual Projector editor. This switches the Stage level to a 2D output mode. In this mode d3 will generate a line-drawing based on the 3D mesh of the virtual video screen to be used as a test pattern.
  • Left-click a green crosshair (representing a reference point on the video screen), move the white cross in the output and match it to its physical corresponding point.
  • To move the white cross pixel-per-pixel use the keyboard arrows.

Aligning a virtual reference point to its corresponding point on a real world object, in this example a wooden sculpture

Aligning a virtual reference point to its corresponding point on a real world object, in this example a wooden sculpture

  • Repeat the same process for all reference points. The more accurate you are in finding the exact physical corresponding point, the better the lineup.

Calibrate!

  • Use this property to calibrate the properties of the virtual projector, i.e. lining the content up to the physical screen.

Virtual projector has been automatically calibrated; the 3D mesh test pattern now lines up with the wooden sculpture

Please note: assuming that the 3D model is modelled accurately to the real-world screen, the virtual projector should now be set. Auto-calibration usually generates a better lineup than the manual alternative.

  • Click Esc to exit auto-calibration and return to the Stage level.

Tips and tricks when using auto-calibration

Auto-calibration works extremely well in cases where the 3D model is very accurate. If the 3D model is not accurate enough, auto-calibration will not work. However, there are a few points to consider if the auto-calibration lineup was unsuccessful.

  • Ensure the virtual projector is covering the entire virtual video screen surface.
  • Ensure that the screen has been added to the virtual projector.
  • Add a minimum of 6 to 9 reference points and distribute them on clear physical points on the projection object.
  • Be as accurate as possible when doing the 2D alignment. Use a spotter and/or a pair of binoculars if needed.
  • Sometimes when left-clicking Calibrate! the virtual projector rotates 180 degrees. If this happens, invert the values of the near and far planes by inserting a negative sign ‘-‘ before them. Left-click Calibrate! again.

If the projection surface still does not match the virtual 3D model the lineup may need to be fine-tuned. Please see the sub-chapter Warping outputs for more information.

Back to top ↑

13.8 Warping outputs

Overview

In cases where the projection surface does not match the virtual 3D model (i.e. where the lineup from the manual or automatic calibration needs a slight fine-tune), d3 provides a number of powerful warping layers that can be applied directly to the output. By using these tools there is no requirement for projector keystoning or any other warping inside the physical projector.

Please note: reset all digitial and optical warps inside the projector, including lens-shift, otherwise d3’s lineup features will be interrupted.

Adding/using warping layers

  • To add a warping layer to an output left-click feed in the d3 State editor (bar at the top of the screen), right-click the appropriate output and select edit feed warping from the options list.

Edit Feed Warping option being used to add a warping layer to an output feed

  • In the editing window, left-click the + button under the Deformation layers title. This will display a list of deformation layers.

Deformation layers for warping an output feed, opened by left-clicking + under the Deformation Layers title

4-point feedwarp

Drag any of the 4 points inwards to globally adjust the warping of the output. Adjust the horizontal and vertical bias to shift-scale the content.

9-point feedwarp

9-point warp allows you to drag 9 points inwards and adjust the pinch and straightness between them. This feed warp is useful when projecting onto a cylindrical screen when not using Projector Simulation. It easily conforms a rectangular output to fit to a cylindrical screen.

Custom feedwarp

Creates an arbitrary number of grid points on the output which can be group-selected. You can use this when local warping on an object is required.

Mesh deform

d3 automatically generates warping points based on every vertex point of the 3D object. To compensate for local in-accuracies between the 3D model and the real object, drag the mesh points inwards and line the content up.

WARNING: Mesh deform is an alpha feature which does not behave well when 4-point feedwarp is applied on top of the mesh deform. Pixels at the sides of the output may be cropped away.

Mesh Deform layer uses warping points that can be dragged to match the virtual model to the real world object

Lens deform

This is an alpha feature which is intended to compensate for projector lens characteristics. The feature is currently not active and should not be used in projects.

Reference to another projector

  • If there are two physical projectors close to each other, they may have different outputs generated from the corresponding virtual projectors, but they may share the same feedwarp. Therefore, apply the same feedwarp on projector B as on projector A using the Reference to another projector feedwarp.

Back to top ↑

13.9 Blending/masking outputs

d3 allows the user to generate custom softedge textures and masks inside d3. You can then apply the softedge texture directly on the output and update the shape, gradient and gamma settings in real-time.

Create a softedge mask on the output

  • Open the Feed level by left-clicking Feed from the d3 State editor (bar at the top of the screen), right-click the appropriate output and select select mask from the options list.

Select Mask option is used to create a custom SoftEdgeTexture mask

This will open the Texture object library which shows all of the still image files saved on your local hard-drive in the DxTexture folder. For more information on the DxTexture folder and other folders please see the sub-chapter Placing media files for a project.

  • In the top-left corner of the library, where it says new texture , type in the name of the softedge mask, for example Output1_softedge.
  • Hit Enter and select the texture type SoftEdgeTexture.

Texture object library being used to create a SoftEdgeTexture mask called Output1_softedge

  • Right-click the output and select edit mask (MaskName) from the options list, in this example edit mask (output1softedge). This will open the SoftEdge editor (explained below). The SoftEdge editor contains a series of points which can be dragged to adjust the gradient of the SoftEdgeTexture mask.

Points used to warp a mask, in this example the newly created Output1_Softedge mask

Softedge editor

Resolution

The resolution values are shown in the top right corner of the softedge editor. The resolution is automatically set to the resolution of the output head.

Adjust global gradients through gamma

The gamma value adjusts the global gradient between the inner and outer points in the softedge mask.

Select and move points

  • Select individual points, pairs of inner and outer points, or drag-select multiple points to adjust the blend level.

Add and remove points

  • Add more points by holding down Ctrl and left-clicking between existing points.
  • Remove points by right-clicking a point.

Adjust local gradients

  • Adjust the gradient between an inner and outer point by dragging them further apart, or closer together.

Please note: press Esc to exit texture editing. Right-click the output feed and select ‘Edit mask (Name)’ to edit the texture again.

Create a mask rather than a softedge

  • To create a mask rather than a softedge, set either gamma to 0 (this will set a sharp edge between the inner and outer points regardless of spacing between them) or move the inner and outer points closer together.

Composite multiple textures

d3 allows the user to generate two different textures and composite them into one single texture. To do this:

  • Create two different textures (they do not have to be of the type SoftEdgeTexture but can also be of the types GradientTexture and SolidTexture.
  • Repeat the same process as above when creating a texture but this time choose the texture type CompositeTexture and name it, for example, SoftEdge_output1 and Gradient_output1.
  • After creating the the CompositeTexture texures, their corresponding outputs will turn grey. This is because the CompositeTexture textures need to be told what textures they contain.
  • Right-click the output head and select edit mask (SoftEdge_output1) and edit mask (Gradient_output1) in the options window.
  • Set the resolution of the CompositeTexture texture.
  • Add the first texture by left-clicking add new layer, and then selecting the image from the Texture library.
  • Click add new layer again and select the second image.
  • To multiply the two images together, set the blend mode of image two to multiply.
  • Press Esc to exit texture editing.

Back to top ↑