One line of code to realize Android screen acquisition coding

 

 

More and more apps need to share mobile screens for others to watch, especially in the online education industry. Android has supported MediaProjection since 5.0. With MediaProjection, screen capture can be realized.

This library encapsulates the screen acquisition code. With a simple call, MediaProject permission application, H264 hard coding, error handling and other functions can be realized.

 

 

characteristic

  • Adapt to Android higher version
  • Asynchronous hard coding using MediaCodec
  • Coding information configurable
  • Notification bar display
  • call chaining

use

ScreenShareKit.init(this)
          .onH264{ buffer, isKeyFrame, ts ->

         }.start()

Github

Source address

realization

1 request user authorization screen acquisition

@TargetApi(Build.VERSION_CODES.M)
    fun requestMediaProjection(encodeBuilder: EncodeBuilder){
        this.encodeBuilder = encodeBuilder;
        mediaProjectionManager  = activity?.getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
        startActivityForResult(mediaProjectionManager?.createScreenCaptureIntent(), 90000)
    }

startActivityForResult needs to be used in Activity or Fragment, and the authorization result will be recalled in onActivityResult. Therefore, we need to encapsulate this step so that it can get the result by callback to. Here we use an interface free Fragment, which is used by many libraries.

private val invisibleFragment : InvisibleFragment
        get() {
            val existedFragment = fragmentManager.findFragmentByTag(FRAGMENT_TAG)
            return if (existedFragment != null) {
                existedFragment as InvisibleFragment
            } else {
                val invisibleFragment = InvisibleFragment()
                fragmentManager.beginTransaction()
                    .add(invisibleFragment, FRAGMENT_TAG)
                    .commitNowAllowingStateLoss()
                invisibleFragment
            }
        }

fun start(){
    invisibleFragment.requestMediaProjection(this)
}

In this way, we can get the authorization result and MediaProjection object in onActivityResult in an interfaced Fragment.

2. Android 10

If targetSdkVersion is set at 29 and above, it will receive an exception after calling createVirtualDisplay after getting MediaProjection.

java.lang.SecurityException: Media projections require a foreground service of type ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION

This means that this operation needs to be performed in the foreground service.

Then we write a service and pass all the results obtained by onActivityResult.

override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        intent?.let {
            if(isStartCommand(it)){
                val notification = NotificationUtils.getNotification(this)
                startForeground(notification.first, notification.second) //Notification bar display
                startProjection(
                    it.getIntExtra(RESULT_CODE, RESULT_CANCELED), it.getParcelableExtra(
                        DATA
                    )!!
                )
            }else if (isStopCommand(it)){
                stopProjection()
                stopSelf()
            }
        }
        return super.onStartCommand(intent, flags, startId)
    }

In the startProjection method, we need to get the MediaProjectionManager, get the MediaProjection, and then create a virtual display.

private fun startProjection(resultCode: Int, data: Intent) {
        val mpManager = getSystemService(MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
        if (mMediaProjection == null) {
            mMediaProjection = mpManager.getMediaProjection(resultCode, data)
            if (mMediaProjection != null) {
                mDensity = Resources.getSystem().displayMetrics.densityDpi
                val windowManager = getSystemService(WINDOW_SERVICE) as WindowManager
                mDisplay = windowManager.defaultDisplay
                createVirtualDisplay()
                mMediaProjection?.registerCallback(MediaProjectionStopCallback(), mHandler)
            }
        }
    }

private fun createVirtualDisplay() {
        mVirtualDisplay = mMediaProjection!!.createVirtualDisplay(
            SCREENCAP_NAME,
            encodeBuilder.encodeConfig.width,
            encodeBuilder.encodeConfig.height,
            mDensity,
            DisplayManager.VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY or DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC,
            surface,
            null,
            mHandler
        )
    }

In the createVirtualDisplay method, there is a Surface parameter. All actions on the screen will be mapped to this Surface. Here, we use MediaCodec to create an input Surface to receive the output of the screen and encode it.

3.MediaCodec code

 private fun initMediaCodec() {
        val format = MediaFormat.createVideoFormat(MIME, encodeBuilder.encodeConfig.width, encodeBuilder.encodeConfig.height)
        format.apply {
            setInteger(MediaFormat.KEY_COLOR_FORMAT,MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface) //Color format
            setInteger(MediaFormat.KEY_BIT_RATE, encodeBuilder.encodeConfig.bitrate) //Code stream
            setInteger(MediaFormat.KEY_BITRATE_MODE, MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_VBR)
            setInteger(MediaFormat.KEY_FRAME_RATE, encodeBuilder.encodeConfig.frameRate) //Number of frames
            setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
        }
        codec = MediaCodec.createEncoderByType(MIME)
        codec.apply {
            setCallback(object : MediaCodec.Callback() {
                override fun onInputBufferAvailable(codec: MediaCodec, index: Int) {
                }
                override fun onOutputBufferAvailable(
                    codec: MediaCodec,
                    index: Int,
                    info: MediaCodec.BufferInfo
                ) {
                    val outputBuffer:ByteBuffer?
                    try {
                        outputBuffer = codec.getOutputBuffer(index)
                        if (outputBuffer == null){
                            return
                        }
                    }catch (e:IllegalStateException){
                        return
                    }
                    val keyFrame = (info.flags and  MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0
                    if (keyFrame){
                        configData = ByteBuffer.allocate(info.size)
                        configData.put(outputBuffer)
                    }else{
                        val data = createOutputBufferInfo(info,index,outputBuffer!!)
                        encodeBuilder.h264CallBack?.onH264(data.buffer,data.isKeyFrame,data.presentationTimestampUs)
                    }
                    codec.releaseOutputBuffer(index, false)

                }

                override fun onError(codec: MediaCodec, e: MediaCodec.CodecException) {
                    encodeBuilder.errorCallBack?.onError(ErrorInfo(-1,e.message.toString()))
                }

                override fun onOutputFormatChanged(codec: MediaCodec, format: MediaFormat) {
                }

            })
            configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
            surface = createInputSurface()
            codec.start()
        }
    }

Some conventional configurations are made above. MediaFormat can set some parameters for the encoder, such as code rate, frame rate, key frame interval, etc.

MediaCodec coding provides two modes: synchronous and asynchronous. Here, the callback is set asynchronously (asynchronous API 21 and above are available)

4. Encapsulation

In the onoutbufferavailable callback, I have called back the encoded data and determined whether it is a key frame or an ordinary frame. What's the use of encapsulating this library????
In fact, some third-party audio and video SDKs can be combined to directly stream the encoded screen stream data through the third-party SDK to realize the screen sharing function.

Here, take the pushExternalVideoFrame method of anyRTC audio and video SDK as an example

        val rtcEngine = RtcEngine.create(this,"",RtcEvent())
        rtcEngine.enableVideo()
        rtcEngine.setExternalVideoSource(true,false,true)
        rtcEngine.joinChannel("","111","","")
        ScreenShareKit.init(this)
            .onH264 {buffer, isKeyFrame, ts ->
                rtcEngine.pushExternalVideoFrame(ARVideoFrame().apply {
                    val array = ByteArray(buffer.remaining())
                    buffer.get(array)
                    bufType = ARVideoFrame.BUFFER_TYPE_H264_EXTRA
                    timeStamp = ts
                    buf = array
                    height = Resources.getSystem().displayMetrics.heightPixels
                    stride = Resources.getSystem().displayMetrics.widthPixels
                })
            }.start()

A few lines of code can realize screen acquisition, coding and transmission ~ very convenient
Pay attention to me and share knowledge every day~

 

Posted on Sat, 04 Dec 2021 15:34:28 -0500 by arundathi