Size and direction in Android camera development

In the development of Android Camera, two more disturbing problems are size and direction.

Where dimensions refer to:

  • Camera display preview frame size
  • Camera frame size
  • Android display camera preview content control size

And direction means

  • Camera display preview frame direction
  • Direction of camera frame shooting
  • Android phone's own direction

In the development process, we need to deal with the relationship between the three directions and the three dimensions. Here, we take the Camera 1.0 API as an example and refer to Google's open source project: cameraview and android-Camera2Basic .

Size and size

As a hardware device, the camera can provide two types of dimensions:

  • Preview frame size
  • Capture frame size

Preview frame size

Through the getSupportedPreviewSizes method, you can get the size set of preview frames supported.

        private final SizeMap mPreviewSizes = new SizeMap();
        mCamera = Camera.open(mCameraId);
        mCameraParameters = mCamera.getParameters();
        // Supported preview sizes
        mPreviewSizes.clear();
        for (Camera.Size size : mCameraParameters.getSupportedPreviewSizes()) {
            mPreviewSizes.add(new Size(size.width, size.height));
        }

While mPreviewSizes is of SizeMap type. To view the source code, in fact, when adding the length and width of preview frame size, their length width ratio is also calculated and saved. The structure of storage length width ratio can be one to many, that is, the length width ratio is the same, and there can be many sizes of length and width, as long as their final proportion is the same.

    // Same aspect ratio, corresponding to multiple dimensions
    private final ArrayMap<AspectRatio, SortedSet<Size>> mRatios = new ArrayMap<>();

For example, the aspect ratios of 1920 * 1080 and 1280 * 720 are 16:9, while those of 800 * 600 and 640 * 480 are 4:3.

In this case, the ratio of length to width is used, and in some cases, the ratio of width to height is used. In fact, they are the same. They are the values of the longer side and the shorter side when the mobile phone is placed horizontally.

When calculating the aspect ratio, it is necessary to find the greatest common divisor of the width and height values, so as to carry out the reduction calculation. According to the Euclidean algorithm, it is also called the rolling phase division method: the greatest common divisor of two integers is equal to the greatest common divisor of the smaller one and the division remainder of two integers. The conversion code is as follows:

    // a > b
    private static int gcd(int a, int b) {
        while (b != 0) {
            int c = b;
            b = a % b;
            a = c;
        }
        return a;
    }

Capture frame size

Through getsupportedpicturesize method, we can get the size set of supported shot frames.

        // Supported picture sizes;
        private final SizeMap mPictureSizes = new SizeMap();
        mPictureSizes.clear();
        for (Camera.Size size : mCameraParameters.getSupportedPictureSizes()) {
            mPictureSizes.add(new Size(size.width, size.height));
        }

The storage structure is similar to the preview frame, and their length width ratio is calculated when the size set is obtained.

Android displays the control size of the camera preview content. You can get its Width and Height in the method corresponding to the control.

Calculate aspect ratio

With these three types of sizes, the next step is how to deal with them.

In order to avoid stretch phenomenon during preview and shooting, the aspect ratio of preview frame is the best consistent with that of display control, and the aspect ratio of shot frame is also the same with that of preview frame and display control. In a word, the aspect ratio of three frames is the best consistent, so as to have the best preview and shooting effect

Because the image of the preview control of the mobile phone is scaled by the camera preview frame according to the size of the control, when the aspect ratio is inconsistent, the preview image will inevitably be deformed. If the aspect ratio of preview frame is not the same as that of shot frame, the shot image will be deformed and stretched.

stay cameraview First of all, the default aspect ratio is set to 4: 3.

AspectRatio DEFAULT_ASPECT_RATIO = AspectRatio.of(4, 3);

According to this aspect ratio, you can get a list of the sizes that match from the size set of the preview frame, and then find that the width and height of the size list are just larger than the width and height of the preview control. If it is smaller than the width and height of the preview control, the image will be stretched.

        SortedSet<Size> sizes = mPreviewSizes.sizes(mAspectRatio);
        if (sizes == null) { // Not supported
            mAspectRatio = chooseAspectRatio();
            // According to the selected aspect ratio, the corresponding set of supported dimensions is obtained
            sizes = mPreviewSizes.sizes(mAspectRatio);
        }
        // Find the appropriate size from the size set compared to the size of the preview control
        Size size = chooseOptimalSize(sizes);
        // Set the appropriate size to the preview frame of the camera
        mCameraParameters.setPreviewSize(size.getWidth(), size.getHeight());

The code to find the appropriate preview frame size is as follows:

    private Size chooseOptimalSize(SortedSet<Size> sizes) {
        if (!mPreview.isReady()) { // Not yet laid out
            return sizes.first(); // Return the smallest size
        }
        int desiredWidth;
        int desiredHeight;
        // Size of preview interface
        final int surfaceWidth = mPreview.getWidth();
        final int surfaceHeight = mPreview.getHeight();
        // Whether it is a horizontal screen or not. If it is a horizontal screen, the width and height are exchanged
        if (isLandscape(mDisplayOrientation)) {
            desiredWidth = surfaceHeight;
            desiredHeight = surfaceWidth;
        } else {
            desiredWidth = surfaceWidth;
            desiredHeight = surfaceHeight;
        }

        // From the dimensions supported by the selected aspect ratio, find that both the length and width are greater than or equal to the control size
        Size result = null;
        for (Size size : sizes) { // Iterate from small to large
            if (desiredWidth <= size.getWidth() && desiredHeight <= size.getHeight()) {
                return size;
            }
            result = size;
        }
        // If there is no matching condition, select the largest return in the support size.
        return result;
    }

Note that when the screen is in horizontal mode, the width and height of the preview control will be changed, and they will be exchanged.

Once you find the right size for the preview frame, you can set it to the camera.

The size of the camera frame is also determined by the aspect ratio.

        final Size pictureSize = mPictureSizes.sizes(mAspectRatio).last();

In the case of a certain aspect ratio, the shot frame is often the largest size, so the shot picture is clearer, which is why the last method is used.

In this way, when the aspect ratio is determined, the corresponding size can be set.

At Google android-Camera2Basic In the project, there is also such a section of code for setting the size. Different from that, it determines the aspect ratio according to the maximum size of the picture taken, rather than the general 4:3 ratio by default, which is then set on this basis.

       Size largest = Collections.max(Arrays.asList(map.getOutputSizes(ImageFormat.JPEG)),
                        new CompareSizesByArea());

It can be seen that setting the aspect ratio in the camera is still a very important part.

Direction of direction

We've fixed the size problem, and we have direction left.

The camera has two directions to deal with:

  • Preview frame direction
  • Shooting frame direction

In order to get a better camera experience, it is necessary to deal with the preview frame and shooting frame direction, and ensure that the content seen through the mobile phone screen is in the normal direction.

First of all, the natural direction of mobile phones should be clear:

  • The natural direction when the mobile phone screen is upright, at this time, the coordinate origin is located in the upper left corner, the positive X-axis direction is to the right, the positive Y-axis direction is to the down, and the width ratio is short.
  • When the mobile phone screen is horizontally placed in the natural direction, at this time, the coordinate origin is located in the upper left corner, the positive X-axis direction is to the right, the positive Y-axis direction is to the down, and the width ratio is longer than the height.

Preview frame direction

The image data of the camera is from the camera hardware image sensor. After the sensor is fixed on the mobile phone, there is a default viewing direction: the coordinate origin is located in the upper left corner of the mobile phone when it is horizontally placed anticlockwise, that is, it is consistent with the screen X direction of the horizontal screen application. That is to say, it is 90 degrees to the screen X direction of the vertical screen application.

Here are some pictures:

Therefore, for horizontal screen applications, the natural direction of the screen is the same as that of the camera's image sensor, so the image you see is positive. For vertical applications, the preview image is turned sideways. You need to rotate the preview image 90 degrees clockwise to preview the image normally.

Horizontal screen shooting results:

Vertical screen shooting results:

There is a 90 degree angle deviation between the preview direction of the Camera and the natural direction of the screen, which is also explained in the orientation attribute of the Camera:

orientation indicates the direction of the camera image. Its value is the image when the camera image rotates clockwise to the same natural direction of the device. It may be 0, 90, 180, 270.

For vertical screen applications, the rear camera sensor is installed horizontally. When you face the screen, if the top edge of the rear camera sensor is parallel to the right side of the natural direction of the device, the orientation of the rear camera is 90. If the top edge of the front camera sensor is parallel to the right side of the natural direction of the device, the orientation of the front camera is 270.

For the front and rear camera sensor orientation is different, and there may be different values on different devices.

When the Activity direction is not limited, the official recommended code is used to set the direction:

  public static void setCameraDisplayOrientation(Activity activity, int cameraId, android.hardware.Camera camera) {
        android.hardware.Camera.CameraInfo info =
                new android.hardware.Camera.CameraInfo();
        android.hardware.Camera.getCameraInfo(cameraId, info);
        int rotation = activity.getWindowManager().getDefaultDisplay()
                .getRotation();
        int degrees = 0;
        switch (rotation) {
            case Surface.ROTATION_0:
                degrees = 0;
                break;
            case Surface.ROTATION_90:
                degrees = 90;
                break;
            case Surface.ROTATION_180:
                degrees = 180;
                break;
            case Surface.ROTATION_270:
                degrees = 270;
                break;
        }

        int result;
        if (info.facing == Camera.CameraInfo.CAMERA_FACING_FRONT) {
            result = (info.orientation + degrees) % 360;
            result = (360 - result) % 360;  // compensate the mirror
        } else {  // back-facing
            result = (info.orientation - degrees + 360) % 360;
        }
        camera.setDisplayOrientation(result);
    }

First, calculate the anti clockwise rotation angle of the device. For the calculation of the rear camera sensor: (info. Orientation - degrees + 360)% 360.

Because the camera image direction needs to be rotated clockwise to restore to the natural direction, and the screen rotation counter clockwise just offsets the camera rotation, so the two subtract, and then add 360 modulus operation.

For the front camera sensor, when using the front camera, what you can see from the vertical direction of the screen is often a mirror image. This is because the camera hardware has turned the image horizontally, that is to say, the image content has been adjusted in the vertical direction, which is equivalent to 180 degrees of rotation in advance. After that, you only need to rotate 90 degrees to get to the natural direction. It's just a mirror image, that is, turning left and right.

It is important to note that before API 14, the setDisplayOrientation method is called with the preview turned off.

At last, the stolen picture is clearer:

Shooting frame direction

It determines the direction of preview and shooting.

With the Camera.Parameters.setRotation function, you can set the direction of the final picture taken by the camera.

Official recommendation Code:

    public void onOrientationChanged(int orientation) {
        if (orientation == ORIENTATION_UNKNOWN) {
            return;
        }
        android.hardware.Camera.CameraInfo info = new android.hardware.Camera.CameraInfo();
        android.hardware.Camera.getCameraInfo(cameraId, info);

        orientation = (orientation + 45) / 90 * 90;
        int rotation = 0;

        if (info.facing == CameraInfo.CAMERA_FACING_FRONT) {
            rotation = (info.orientation - orientation + 360) % 360;
        } else {  // back-facing camera
            rotation = (info.orientation + orientation) % 360;
        }
        mParameters.setRotation(rotation);
    }

To calculate the direction of rotation, you need to consider the direction of the current screen and the direction of the camera.

OrientationEventListener works with Camera.orientation. When the screen direction changes, OrientationEventListener will receive a corresponding notice. In the callback method of onOrientationChanged, change the camera shooting direction. In fact, the change of camera preview direction is also in the callback method.

The return value of the onOrientationChanged method is from 0 to 359. The setRotation value can only be 0, 90, 180, 270. So we need to do a similar rounding operation for orientation of screen direction.

Of course, you can also get the direction according to the getRotation method of Display class in this callback. In short, there is a callback notification, and then change the direction of screen shooting and preview here.

For the front camera, the difference between the orientation of the camera and the orientation of the screen is the angle to be rotated; for the rear camera, the sum of the two is the angle to be rotated.

Here, we have a clearer understanding of the size and direction settings in camera development.

Reference for reference

  1. https://blog.csdn.net/Tencent_Bugly/article/details/53375311
  2. https://blog.csdn.net/daiqiquan/article/details/40650055
  3. https://zhuanlan.zhihu.com/p/20559606
  4. http://javayhu.me/blog/2017/09/25/camera-development-experience-on-android/

Welcome to attention, keep updating, official account reply: OpenGL, receive learning resource gift.

Tags: Android Mobile Google Attribute

Posted on Sat, 14 Mar 2020 03:02:04 -0400 by fuii_koh