Case study of intelligent recognition of voice synchronization in wechat applet

catalog

1, Background

In some application scenarios of the applet, there will be a need for voice to text. The original method is to record the voice file through the recording function of the small program, and then transfer the voice file to the text message by calling the voice intelligent recognition WebApi (for example, baidu cloud AI platform, iFLYTEK platform). The above method is cumbersome and the user experience is poor.
In order to solve this problem, wechat directly opens the plug-in of simultaneous interpretation, which can be directly used by the author of small program to develop simultaneous interpretation. This article will complete the real-time conversion of voice through a complete case of front and back-end integrated application, and upload the voice to the server background backup.

2, Introduction to simultaneous interpretation plug-ins

WeChat simultaneous interpreting is a simultaneous transmission interface launched by WeChat Chi Ling's voice team, WeChat translation team and public platform. The first phase opens the interface of voice to text, text translation and speech synthesis, and provides energy for developers.

1. Add plug-ins in the background of wechat applet

Enter wechat applet background - > Enter settings - > third party settings - > Add plug-ins - > search simultaneous interpretation - > finish adding.

2. Wechat applet enabling plug-in

In applet app.json Add plug-in version and other information to the file:

"plugins": {
    "WechatSI": {
      "version": "0.3.3",
      "provider": "wx069ba97219f66d99"
    }
  },

To introduce a plug-in into a pager file:

/* index.js */

const plugin = requirePlugin("WechatSI")

// Get * * globally unique * * voice recognition manager * * recordRecoManager**
const manager = plugin.getRecordRecognitionManager()

List of methods for the recordRecoManager object:

method parameter explain
start options Start identifying
stop End identification
onStart callback This event is called when recording recognition starts normally
onRecognize callback This event will be called if there is a new identification returned
onStop callback Identify end events
onError callback Identify error events

Official development documents: Speech recognition manager for plug-ins

3, Front end implementation of voice synchronous conversion

1. Interface UI and operation

The UI refers to the official DEMO of wechat: press and hold the button for recording, and release the button to convert the recording into text in real time.

Users can edit the synchronously converted text and upload the original voice file and text to the background server.

2. Code implementation

The main code of voice synchronization conversion:

//Import plug-ins
const plugin = requirePlugin("WechatSI");
// Get * * globally unique * * voice recognition manager * * recordRecoManager**
const manager = plugin.getRecordRecognitionManager();

/**
   * Load for initialization
   */
 onLoad: function () {
 	//Get recording permission
	app.getRecordAuth();
	//Initialize speech recognition callback
    this.initRecord();
  },

 ...
 
/**
   * Initialize speech recognition callback
   * Bind voice playback start event
   */
  initRecord: function () {
    //This event will be called if there is a new identification returned
    manager.onRecognize = (res) => {
      let currentData = Object.assign({}, this.data.currentTranslate, {
        text: res.result,
      });
      this.setData({
        currentTranslate: currentData,
      });
      this.scrollToNew();
    };

    // Identify end events
    manager.onStop = (res) => {
      let text = res.result;

      console.log(res.tempFilePath);

      if (text == "") {
        this.showRecordEmptyTip();
        return;
      }

      let lastId = this.data.lastId + 1;

      let currentData = Object.assign({}, this.data.currentTranslate, {
        text: res.result,
        translateText: "Identifying",
        id: lastId,
        voicePath: res.tempFilePath,
        duration: res.duration
      });

      this.setData({
        currentTranslate: currentData,
        recordStatus: 1,
        lastId: lastId,
      });
      //Add the current recognition content and voice file to the list
      this.addRecordFile(currentData, this.data.dialogList.length);
      //Refresh List 
	  this.scrollToNew();
    };

    // Identify error events
    manager.onError = (res) => {
      this.setData({
        recording: false,
        bottomButtonDisabled: false,
      });
    };

  },

  /**
   * Press and hold the button to start speech recognition
   */
  streamRecord: function (e) {
    let detail = e.detail || {};
    let buttonItem = detail.buttonItem || {};
    //Start Chinese recording
    manager.start({
      lang: buttonItem.lang,
    });

    this.setData({
      recordStatus: 0,
      recording: true,
      currentTranslate: {
        // Current voice input
        create: util.recordTime(new Date()),
        text: "Listening",
        lfrom: buttonItem.lang,
        lto: buttonItem.lto,
      },
    });
    //Refresh List 
    this.scrollToNew();
  },

  /**
   * Release button to end speech recognition
   */
  streamRecordEnd: function (e) {
    let detail = e.detail || {}; // The detail object provided when a custom component triggers an event
    let buttonItem = detail.buttonItem || {};

    // Prevent repeated triggering of stop function
    if (!this.data.recording || this.data.recordStatus != 0) {
      console.warn("has finished!");
      return;
    }

    manager.stop();

    this.setData({
      bottomButtonDisabled: true,
    });
  },

Edit the identification text and finish uploading the main code:

 /**
   * Initial data of the page
   */
  data: {
    edit_text_max: 200,
    remain_length: 200,
    edit_text: "",
    is_focus: false,
    tips: "",
    index: -1,
    voicePath: "",
    
  },

/**
   * Load initialization
   */
 onLoad: function (options) {
    //Fill in the edit box based on the incoming text content
    this.setEditText(options.content)
    
    this.setData({
        index: index,
        oldText:options.content,
        voicePath: options.voicePath
    })
    
  },

 /**
   * Edit text
   */
  editInput: function (event) {
    console.log(event)
    if (event.detail.value.length > this.getEditTextMax()) {

    } else {
      this.data.edit_text = event.detail.value
      this.updateRemainLength(this.data.edit_text)
    }
  },

 /**
   * Upload text and voice files
   */
  editConfirm: function (event) {
    let json=this.data.edit_text
    //Call the wechat upload file api to upload the information to the server webApi
    wx.uploadFile({
      url: api.wxFileUploadUrl,
      filePath: this.data.voicePath,
      name: "file",
      header: {
        Authorization: wx.getStorageSync("loginFlag"),
        "Content-Type": "multipart/form-data",
      },
      formData: {
        openId: app.globalData.userInfo.openId,
        realName: "Voice files",
        json: JSON.stringify(json),
      },
      success: (result) => {
        console.log("success:", result);
        if (result.statusCode == "200") {
          let data = JSON.parse(result.data);
          console.log("data", data);
          if (data.success == true) {
            let module = data.module;
            console.log("module", module);
            app.showInfo("Upload succeeded");            
            setTimeout( ()=>{
              wx.navigateBack();
            }, 2000)
                      
          } else {
            app.showInfo("Unexpected error" + data.errMsg + ",Please re-enter");
            wx.navigateTo({
              url: "/pages/index/index",
            });
          }
        } else {
          app.showInfo("Access background exception,Re entering the system");
          wx.navigateTo({
            url: "/pages/index/index",
          });
        }
      },
      fail: (result) => {
        console.log("fail", result);
        wx.navigateTo({
          url: "/pages/index/index",
        });
      },
      complete: () => {},
    });

  },

4, Back end SpringBoot to upload voice files to webApi

1. Spring boot project API related structure tree

2. Implementation of file upload tool class

The tool class package mainly stores the general file upload tool class, which will upload the file to the folder specified by the configuration and write the file information to upload_file table.

  • File information entity class: upload with table in database_ File correspondence;
  • File storage warehouse class: CRUD of data is realized through Spring Data JPA interface;
  • File upload tool interface: encapsulate the file upload method in a unified way;
  • File upload tool implementation class: implements the file upload method interface.

Document information entity class: UploadFile.java

/**
 * Document information sheet
 *
 * @author zhuhuix
 * @date 2020-04-20
 */
@Entity
@Getter
@Setter
@Table(name = "upload_file")
public class UploadFile {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @NotNull(groups = Update.class)
    private Long id;

    /**
     * Actual name of document
     */
    @Column(name = "real_name")
    private String realName;

    /**
     * file name
     */
    @NotNull
    @Column(name = "file_name")
    private String fileName;

    /**
     * File master name
     */
    @NotNull
    @Column(name = "primary_name")
    private String primaryName;

    /**
     * File extension
     */
    @NotNull
    private String extension;

    /**
     * Storage path
     */
    @NotNull
    private String path;

    /**
     * file type
     */
    private String type;

    /**
     * file size
     */
    private Long size;

    /**
     * Uploaded by
     */
    private String uploader;

    @JsonIgnore
    @Column(name = "create_time")
    @CreationTimestamp
    private Timestamp createTime;

    public UploadFile(String realName, @NotNull String fileName, @NotNull String primaryName, @NotNull String extension, @NotNull String path, String type, Long size, String uploader) {
        this.realName = realName;
        this.fileName = fileName;
        this.primaryName = primaryName;
        this.extension = extension;
        this.path = path;
        this.type = type;
        this.size = size;
        this.uploader = uploader;
    }

    @Override
    public String toString() {
        return "UploadFile{" +
                "fileName='" + fileName + '\'' +
                ", uploader='" + uploader + '\'' +
                ", createTime=" + createTime +
                '}';
    }
}

File storage warehouse class: UploadFileRepository.java

/**
 * Upload file DAO interface layer
 *
 * @author zhuhuix
 * @date 2020-04-03
 */
public interface UploadFileRepository extends JpaRepository<UploadFile, Long>, JpaSpecificationExecutor<UploadFile> {
//This interface inherits the JpaRepository and CrudRepository interfaces, and has implemented CRUD methods such as findById,save,delete, etc
}

The UploadFileRepository interface inherits the JpaRepository and CrudRepository interfaces, and has implemented CRUD methods such as findById,save,delete, etc

File upload tool interface: UploadFileTool.java

/**
 * File upload interface definition
 *
 * @author zhuhuix
 * @date 2020-04-20
 */
public interface UploadFileTool {

    /**
     * File upload
     * @param multipartFile file
     * @return Upload information
     */
   UploadFile upload(String uploader,String realName,MultipartFile multipartFile);
}

File upload tool implementation class: UploadFileToolImpl.java

/**
 * File upload implementation class
 *
 * @author zhuhuix
 * @date 2020-04-20
 */
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class UploadFileToolImpl implements UploadFileTool {

    private final UploadFileRepository uploadFileRepository;

    @Value("${uploadFile.path}")
    private String path;

    @Value("${uploadFile.maxSize}")
    private long maxSize;

    public UploadFileToolImpl(UploadFileRepository uploadFileRepository) {
        this.uploadFileRepository = uploadFileRepository;
    }

    @Override
    @Transactional(rollbackFor = Exception.class)
    public UploadFile upload(String uploader, String realName, MultipartFile multipartFile) {
        //Check file size
        if (multipartFile.getSize() > maxSize * Constant.MB) {
            throw new RuntimeException("File upload size limit exceeded" + maxSize + "MB");
        }
        //Get the main file name and extension of the uploaded file
        String primaryName = FileUtil.mainName(multipartFile.getOriginalFilename());
        String extension = FileUtil.extName(multipartFile.getOriginalFilename());
        //Get file type based on file extension
        String type = getFileType(extension);
        //Time stamp the uploaded file
        LocalDateTime date = LocalDateTime.now();
        DateTimeFormatter format = DateTimeFormatter.ofPattern("yyyyMMddhhmmssS");
        String nowStr = "-" + date.format(format);
        String fileName = primaryName + nowStr + "." + extension;

        try {
            String filePath = path + type + File.separator + fileName;
            File dest = new File(filePath).getCanonicalFile();
            if (!dest.getParentFile().exists()) {
                dest.getParentFile().mkdirs();
            }
            multipartFile.transferTo(dest);
            if (ObjectUtil.isNull(dest)) {
                throw new RuntimeException("Failed to upload file");
            }

            UploadFile uploadFile = new UploadFile(realName, fileName, primaryName, extension, dest.getPath(), type, multipartFile.getSize(), uploader);
            return uploadFileRepository.save(uploadFile);

        } catch (Exception e) {
            e.printStackTrace();
            throw new RuntimeException(e.getMessage());
        }

    }

    /**
     * File type based on file extension
     *
     * @param extension File extension
     * @return file type
     */
    private static String getFileType(String extension) {
        String document = "txt doc pdf ppt pps xlsx xls docx csv";
        String music = "mp3 wav wma mpa ram ra aac aif m4a";
        String video = "avi mpg mpe mpeg asf wmv mov qt rm mp4 flv m4v webm ogv ogg";
        String image = "bmp dib pcp dif wmf gif jpg tif eps psd cdr iff tga pcd mpt png jpeg";
        if (image.contains(extension)) {
            return "image";
        } else if (document.contains(extension)) {
            return "document";
        } else if (music.contains(extension)) {
            return "music";
        } else if (video.contains(extension)) {
            return "video";
        } else {
            return "other";
        }
    }
}

Note that the @ Value annotation is used in the program code to get the uploadFile.path and uploadFile.maxsize Parameters, usually written in the project static configuration file as follows (yml configuration file).

# Test environment file storage path
uploadFile:
  path: C:\startup\file\
  # File size / M
  maxSize: 50
3. The implementation of the interface for uploading files in small programs

The Wx miniprogram package defines the interface of CRM webApi, and the applet calls webApi to upload files and other functions.

  • Wechat app webApi: provide external app upload file webApi;
  • Wechat applet service interface: encapsulate the service interface for uploading files of applets;
  • Wechat applet service implementation: the implementation of applet upload file service, which calls the UploadFile interface in tools package to upload files.

Wechat app CRM webApi: WxMiniCrmController.java

/**
 * Wechat applet Crm webApi
 *
 * @author zhuhuix
 * @date 2020-03-30
 */
@Slf4j
@RestController
@RequestMapping("/api/wx-mini")
@Api(tags = "Wechat applet Crm Interface")
public class WxMiniCrmController {

    private final WxMiniCrm wxMiniCrm;

    public WxMiniCrmController(WxMiniCrm wxMiniCrm) {
        this.wxMiniCrm = wxMiniCrm;
    }

    @ApiOperation(value = "Upload files on wechat applet")
    @PostMapping(value = "/fileUpload")
    public ResponseEntity fileUpload(HttpServletRequest request) {
        MultipartHttpServletRequest req = (MultipartHttpServletRequest) request;

        MultipartFile multipartFile = req.getFile("file");
        String openId = req.getParameter("openId");
        String realName = req.getParameter("realName");
        String json = req.getParameter("json");

        return ResponseEntity.ok(wxMiniCrm.uploadFile(json, openId,realName, multipartFile));

    }
}

Wechat applet CRM service interface: WxMiniCrm.java

/**
 * Definition of CRM service interface of wechat applet
 *
 * @author zhuhuix
 * @date 2020-04-20
 */
public interface WxMiniCrm {

    /**
     * Write the json object passed in by the wechat applet to the database, and upload the file to the server at the same time
     *
     * @param json          Incoming json object from wechat
     * @param openId        Uploaded by
     * @param realName      Actual name of document
     * @param multipartFile Upload file
     * @return Return to upload information
     */
    Result<UploadFile> uploadFile(String  json, String openId, String realName,MultipartFile multipartFile);
}

Wechat applet CRM service implementation: WxMiniCrmImpl.java

/**
 * Wechat applet CRM implementation class
 *
 * @author zhuhuix
 * @date 2020-04-20
 */
@Slf4j
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class WxMiniCrmImpl implements WxMiniCrm {

    private final UploadFileTool uploadFileTool;

    public WxMiniCrmImpl(UploadFileTool uploadFileTool) {
        this.uploadFileTool = uploadFileTool;
    }

    @Override
    @Transactional(rollbackFor = Exception.class)
    public Result<UploadFile> uploadFile(String  json, String openId,String realName, MultipartFile multipartFile) {
        return new Result<UploadFile>().ok(uploadFileTool.upload(openId,realName, multipartFile));
    }
}
4. View of the interface for uploading files of applets

Visit Swagger2 to see the interface, and refer to the integration of Swagger2 and SpringBoot SpringBoot JWT authentication mechanism project integration Swagger2

5, Actual test

Voice test OK

Upload file to background:

View the uploaded log information:

Tags: Java JSON SpringBoot Spring

Posted on Fri, 29 May 2020 02:03:51 -0400 by 051119