# preface

Recently, I was studying the book "Multiple view Geometry". I finally felt shallow on the paper. Looking at the theory alone, I found it very empty and difficult to master. Therefore, I consolidated my knowledge through programming operation. If there are mistakes, thank you for criticism and correction.

This article is mainly about the idea and programming implementation of recovering affine properties from projected images.

## principle

There are various distortions in the pictures taken by the camera. Among them, the projection distortion makes the originally parallel straight lines no longer parallel, which will produce the effect of near large and far small in the photos. To correct this distortion, the book gives many methods. Here is one of them.

We can divide the projection transformation into three parts: similarity transformation, affine transformation and projection transformation, as shown in the following figure,

The similarity transformation and affine transformation will not change the infinite line, only the projection transformation will change. Therefore, as long as this line in the distorted image is found, the affine characteristics of the image can be restored (equivalent to reversing the projection transformation). To determine the position of this line, we must know at least two points on the line. We know that the intersections of all parallel lines are above the infinite line. Therefore, we only need to find two pairs of parallel lines on the image (originally parallel, but no longer parallel on the image) and find the corresponding two intersections to find the infinite line, as shown in the following figure

Then the affine properties of the image can be restored.

# Realization idea

First, our distorted image is shown below,

Using the formula:

l = x1 × x2

The homogeneous coordinates of the two-point line l can be obtained from the homogeneous coordinates of x1 and x2. In the figure, we find two pairs of parallel lines l1, l2, l3 and l4, as shown below

Using the formula:

x = l1 × l2

The intersections A12 and A34 of two pairs of parallel lines can be obtained from the homogeneous coordinates of l1, l2, l3 and l4 respectively. The straight line A12A34 is the infinite line we are looking for. Assuming that the homogeneous coordinates of the line are (l1, l2, l3), then through the matrix:

H = ((1，0，0)，(0，1，0)，(l1，l2，l3))

The straight line (l1, l2, l3) can be transformed into (0, 0, 1), that is, the straight line can be restored to infinite line. Similarly, we can also use the H matrix through the formula:

x = Hx'

Restore projection distortion.

## code

The code needs to be run twice

Main function of the first run:

int main() { Mat src = imread("distortion.jpg", IMREAD_GRAYSCALE); IplImage *src1 = cvLoadImage("distortion.jpg"); //In the first step, obtain the coordinates of a point in the picture through the mouse, and comment out Rectify(points_3d, src, src1) when running the first step;, Write the obtained eight points to //points_ In the 3D [8] coordinate array, because it is homogeneous, x3 = 1 GetMouse(src1); //8 key points on input distortion diagram Point3d points_3d[8] = { Point3d(99, 147, 1), Point3d(210, 93, 1), Point3d(144, 184, 1), Point3d(261, 122, 1), Point3d(144, 184, 1), Point3d(99, 147, 1), Point3d(261, 122, 1), Point3d(210, 93, 1) }; //Step 2: correct the image and comment out GetMouse(src1) when running this step;, Uncomment Rectify(points_3d, src, src1); //Rectify(points_3d, src, src1); imshow("yuantu", src); waitKey(0); }

Other functions:

void on_mouse(int event, int x, int y, int flags, void* ustc) { CvFont font; cvInitFont(&font, CV_FONT_HERSHEY_SIMPLEX, 0.5, 0.5, 0, 1, CV_AA); if (event == CV_EVENT_LBUTTONDOWN) { CvPoint pt = cvPoint(x, y); char temp[16]; sprintf(temp, "(%d,%d)", pt.x, pt.y); cvPutText(src, temp, pt, &font, cvScalar(255, 255, 255, 0)); cvCircle(src, pt, 2, cvScalar(255, 0, 0, 0), CV_FILLED, CV_AA, 0); cvShowImage("src", src); } } void GetMouse(IplImage *img) { src = img; cvNamedWindow("src", 1); cvSetMouseCallback("src", on_mouse, 0); cvShowImage("src", src); waitKey(0); }

Click anywhere in the pop-up picture to obtain the image coordinates (x1, x2) of the modified point, as shown in the following figure:

I selected four points a, b, c and d, among which:

ab // cd ac // bd

Fill the coordinates of the four points in the order of a, b, c, d, c, a, d and b_ In the 3d [8] coordinate array, the first run ends.

Main function of the second run:

int main() { Mat src = imread("distortion.jpg", IMREAD_GRAYSCALE); IplImage *src1 = cvLoadImage("distortion.jpg"); //In the first step, obtain the coordinates of a point in the picture through the mouse, and comment out Rectify(points_3d, src, src1) when running the first step;, Write the obtained eight points to //points_ In 3D [8] matrix, because it is homogeneous coordinate, x3 = 1 //GetMouse(src1); //8 key points on input distortion diagram Point3d points_3d[8] = { Point3d(99, 147, 1), Point3d(210, 93, 1), Point3d(144, 184, 1), Point3d(261, 122, 1), Point3d(144, 184, 1), Point3d(99, 147, 1), Point3d(261, 122, 1), Point3d(210, 93, 1) }; //Step 2: correct the image and comment out GetMouse(src1) when running this step;, Uncomment Rectify(points_3d, src, src1); Rectify(points_3d, src, src1); imshow("yuantu", src); waitKey(0); }

Correction function:

void Rectify(Point3d* points, Mat src, IplImage* img) { //Four lines are obtained through the input 8 points vector<vector<float>> lines; int num_lines = 4; for(int i = 0; i < num_lines; i++) { //Get two-point connection GetLineFromPoints(points[2 * i], points[2 * i + 1], lines); } //Find two intersections respectively vector<Point3f> intersect_points; int num_intersect_points = 2; for (int i = 0; i < num_intersect_points; i++) { //Calculate intersection GetIntersectPoint(lines[2 * i], lines[2 * i + 1], intersect_points); } //Find the vanishing line through the connecting line of two intersections vector<vector<float>> vanishing_line; GetLineFromPoints(intersect_points[0], intersect_points[1], vanishing_line); //Recovery matrix float H[3][3] = {{1, 0, 0}, {0, 1, 0}, {vanishing_line[0][0], vanishing_line[0][1], vanishing_line[0][2]}}; Mat image = Mat::zeros(src.rows, src.cols, CV_8UC1); GetRectifingImage(vanishing_line[0], src, image); int i = 0; } void GetLineFromPoints(Point3d point1, Point3d point2, vector<vector<float>> &lines) { vector<float> line; //Defines the three homogeneous coordinates of a line float l1 = 0; float l2 = 0; float l3 = 0; l1 = (point1.y * point2.z - point1.z * point2.y); l2 = (point1.z * point2.x - point1.x * point2.z); l3 = (point1.x * point2.y - point1.y * point2.x); //normalization l1 = l1 / l3; l2 = l2 / l3; l3 = 1; line.push_back(l1); line.push_back(l2); line.push_back(l3); lines.push_back(line); } void GetIntersectPoint(vector<float> line1, vector<float> line2, vector<Point3f> &intersect_points) { Point3f intersect_point; //Three homogeneous coordinates defining the intersection float x1 = 0; float x2 = 0; float x3 = 0; x1 = (line1[1] * line2[2] - line1[2] * line2[1]); x2 = (line1[2] * line2[0] - line1[0] * line2[2]); x3 = (line1[0] * line2[1] - line1[1] * line2[0]); //normalization x1 = x1 / x3; x2 = x2 / x3; x3 = 1; intersect_point.x = x1; intersect_point.y = x2; intersect_point.z = x3; intersect_points.push_back(intersect_point); } int Round(float x) { return (x > 0.0) ? floor(x + 0.5) : ceil(x - 0.5); } void GetRectifingImage(vector<float> line, Mat src, Mat dst) { Size size_src = src.size(); for (int i = 0; i < size_src.height; i++) { for (int j = 0; j < size_src.width; j++) { float x3 = line[0] * j + line[1] * i + line[2] * 1; int x1 = Round(j / x3); int x2 = Round(i / x3); if (x1 < size_src.width && x1 >= 0 && x2 < size_src.height && x2 >= 0) { dst.at<uint8_t>(x2, x1) = src.at<uint8_t>(i, j); } } } imshow("src", src); imshow("dst", dst); waitKey(0); }

The operation results are as follows:

The correction effect is related to the selection of points, because the point clicked by the mouse is not necessarily the point we really want. It is suggested that the distance between the two points of a straight line should be as large as possible.

I put the complete code on the network disk. You need to get the link yourself: https://pan.baidu.com/s/1c1EqaWmBseGKHurynedBlw

Extraction code: qltt