Depth Estimation Using Stereo Imaging in Matlab

Depth Estimation Using Stereo Imaging in Matlab Object Tracking in 3D

Disparity Equation P(X,Y,Z)

Stereo system with parallel optical axes

Depth

Disparity: dx = xr - xl

B Z D f dx Image plane f = focal length

Image plane pl(xl,yl)

pr(xr,yr)

Optical Center Ol

f = focal length

Optical Center Or B = Baseline

LEFT CAMERA

RIGHT CAMERA

Disparity vs. Baseline P(X,Y,Z)

Stereo system with parallel optical axes

Depth

Disparity dx = xr - xl

B Z D f dx Image plane f = focal length

Image plane pl(xl,yl)

Optical Center Ol

pr(xr,yr)

f = focal length

Optical Center Or B = Baseline

LEFT CAMERA

RIGHT CAMERA

Depth Accuracy › Given the same image localization error – Angle of cones in the figure › Depth Accuracy (Depth Resolution) vs. Baseline – Depth Error  1/B (Baseline length) – PROS of Longer baseline, › better depth estimation

Ol

Two viewpoints Or

Z1 Z2

Z1

– CONS › smaller common FOV

› Depth Accuracy (Depth Resolution) vs. Depth – Disparity (>0)  1/ Depth – Depth Error  Depth2 – Nearer the point, better the depth estimation

› An Example – f = 16 x 512/8 pixels, B = 0.5 m – Depth error vs. depth

Z2>Z1

Absolute error

Z2 Z   (dx) fB Relative error Z Z   (dx) Z fB

Depth from disparity image I(x,y)

Disparity map D(x,y)

image I´(x´,y´)

(x´,y´)=(x+D(x,y), y) So if we could find the corresponding points in two images, we could estimate relative depth…

Find the corners. points1 = detectHarrisFeatures(I1); points2 = detectHarrisFeatures(I2);

Extract the neighborhood features. [features1,valid_points1] = extractFeatures(I1,points1); [features2,valid_points2] = extractFeatures(I2,points2);

Match the features. indexPairs = matchFeatures(features1,features2);

Retrieve the locations of the corresponding points for each image. matchedPoints1 = valid_points1(indexPairs(:,1),:); matchedPoints2 = valid_points2(indexPairs(:,2),:);

Visualize the corresponding points. You can see the effect of translation between the two images despite several erroneous matches. figure; showMatchedFeatures(I1,I2,matchedPoints1,matchedPoints2);

Camera parameters Camera frame 2

Extrinsic parameters: Camera frame 1  Camera frame 2

Camera frame 1

Intrinsic parameters: Image coordinates relative to camera  Pixel coordinates

• Extrinsic parameters: rotation matrix and translation vector • Intrinsic parameters: focal length, pixel sizes (mm), image center point, radial distortion parameters

load('webcamsSceneReconstruction.mat');

Read in the stereo pair of images. I1 = imread('sceneReconstructionLeft.jpg'); I2 = imread('sceneReconstructionRight.jpg');

Rectify the images. [J1, J2] = rectifyStereoImages(I1,I2,stereoParams);

Display the images after rectification. figure imshow(stereoAnaglyph(I1,I2));

Compute the disparity. disparityMap = disparity(rgb2gray(J1), rgb2gray(J2)); figure imshow(disparityMap,[0,64],'InitialMagnification',50);

xyzPoints = reconstructScene(disparityMap,stereoParams); points3D = xyzPoints; points3D = points3D ./ 1000; ptCloud = pointCloud(points3D, 'Color', J1); % Create a streaming point cloud viewer player3D = pcplayer([-3, 3], [-3, 3], [0, 8], 'VerticalAxis', 'y', ... 'VerticalAxisDir', 'down'); % Visualize the point cloud view(player3D, ptCloud);

%Undistort the images. I1 = undistortImage(I1,stereoParams.CameraParameters1); I2 = undistortImage(I2,stereoParams.CameraParameters2); [nBW1,maskedImage1] = createMask3(I1); [nBW2,maskedImage2] = createMask3(I2); BW1 = bwareaopen(nBW1,600); BW2 = bwareaopen(nBW2,600); stats1 = regionprops(BW1,'Centroid','BoundingBox'); stats2 = regionprops(BW2,'Centroid','BoundingBox'); bc1 = stats1.Centroid; bc2 = stats2.Centroid; bb1 = stats1.BoundingBox; bb2 = stats2.BoundingBox;

%Compute the distance from camera 1 to the face. point3d = triangulate(bc1, bc2, stereoParams); distanceInMeters = norm(point3d)/1000;

colorDevice = imaq.VideoDevice('kinect',1); depthDevice = imaq.VideoDevice('kinect',2); step(colorDevice); step(depthDevice); colorImage = step(colorDevice); depthImage = step(depthDevice); ptCloud = pcfromkinect(depthDevice, depthImage, colorImage); % Initialize a player to visualize 3-D point cloud data. The axis is % set appropriately to visualize the point cloud from Kinect. player = pcplayer(ptCloud.XLimits, ptCloud.YLimits, ptCloud.ZLimits,... 'VerticalAxis', 'y', 'VerticalAxisDir', 'down'); xlabel(player.Axes, 'X (m)'); ylabel(player.Axes, 'Y (m)'); zlabel(player.Axes, 'Z (m)');

view(player, ptCloud);

ptCloud = pcfromkinect(depthDevice, depthImage, colorImage);

XYZPoints