ImageTracking_Coloring3D

Demonstrate how to create a coloring book and “convert” book image into 3D at real-time.

This sample can capture camera image as model texture, to achieve texture update when drawing.

How to Use

FindBear/Freeze/Thaw: FindBear when a bear target image is not found. Freeze means the bear texture will be frozen after press the button. Thaw will unfreeze the texture to make it change dynamically again.

How It Works

Map camera image in target region to model

Use CameraImageRenderer.RequestTargetTexture to register handler to get camera image texture and use it in the shader.

CameraRenderer.RequestTargetTexture((camera, texture) => { renderTexture = texture; });

Get world transform of the four corners of the target image in the scene.

var halfWidth = 0.5f;
var halfHeight = 0.5f / imageTarget.Target.aspectRatio();
Matrix4x4 points = Matrix4x4.identity;
Vector3 targetAnglePoint1 = imageTarget.transform.TransformPoint(new Vector3(-halfWidth, halfHeight, 0));
Vector3 targetAnglePoint2 = imageTarget.transform.TransformPoint(new Vector3(-halfWidth, -halfHeight, 0));
Vector3 targetAnglePoint3 = imageTarget.transform.TransformPoint(new Vector3(halfWidth, halfHeight, 0));
Vector3 targetAnglePoint4 = imageTarget.transform.TransformPoint(new Vector3(halfWidth, -halfHeight, 0));
points.SetRow(0, new Vector4(targetAnglePoint1.x, targetAnglePoint1.y, targetAnglePoint1.z, 1f));
points.SetRow(1, new Vector4(targetAnglePoint2.x, targetAnglePoint2.y, targetAnglePoint2.z, 1f));
points.SetRow(2, new Vector4(targetAnglePoint3.x, targetAnglePoint3.y, targetAnglePoint3.z, 1f));
points.SetRow(3, new Vector4(targetAnglePoint4.x, targetAnglePoint4.y, targetAnglePoint4.z, 1f));
material.SetMatrix("_UvPints", points);
...

To render the model with this part of image, calculate screen position of pixels and get texture value from that point in shader.

v2f vert (appdata_base v)
{
    v2f o;
    o.pos = UnityObjectToClipPos(v.vertex);
    o.uv = TRANSFORM_TEX(v.texcoord,_MainTex);

    float4 top = lerp(_UvPints[0], _UvPints[2], o.uv.x);
    float4 bottom = lerp(_UvPints[1], _UvPints[3], o.uv.x);
    float4 fixedPos = lerp(bottom, top, o.uv.y);
    float4x4 Rendering_Matrix_VP = mul(_RenderingProjectMatrix, _RenderingViewMatrix);
    o.fixedPos = ComputeGrabScreenPos(mul(Rendering_Matrix_VP, fixedPos));
    return o;
}

float4 frag (v2f i) : COLOR
{
    float2 coord = i.fixedPos.xy / i.fixedPos.w;
    ...
    return tex2D(_MainTex, coord);
}

There is also a special problem to handle. Unity is not able to hide the differences between platforms when using the RenderTexture created from the plugin, so you need to ensure that the differences of render texture coordinates are correctly handled for different rendering APIs when using this texture in render pipeline.

float4 frag (v2f i) : COLOR
{
    float2 coord = i.fixedPos.xy / i.fixedPos.w;
#if SHADER_API_METAL
#if UNITY_UV_STARTS_AT_TOP
    if (_MainTex_TexelSize.y < 0.0)
        coord.y = 1.0 - coord.y;
#endif
    coord.x = 1.0 - coord.x;
#else
#if UNITY_UV_STARTS_AT_TOP
    coord.y = 1.0 - coord.y;
#endif
#endif

    return tex2D(_MainTex, coord);
}

Stop real-time update of texture

There are some cases when you want to use the image in some viewpoint or at a specific time point to be the texture.

freezedTexture = new RenderTexture(renderTexture.width, renderTexture.height, 0);
Graphics.Blit(renderTexture, freezedTexture);
material.SetTexture("_MainTex", freezedTexture);