Case study of intelligent recognition of voice synchronization in wechat applet

• 1, Background
• 2, Introduction to simultaneous interpretation plug-ins
  • 1. Add plug-ins in the background of wechat applet
  • 2. Wechat applet enabling plug-in
• 3, Front end implementation of voice synchronous conversion
  • 1. Interface UI and operation
  • 2. Code implementation
• 4, Back end SpringBoot to upload voice files to webApi
  • 1. Spring boot project API related structure tree
  • 2. Implementation of file upload tool class
  • 3. The implementation of the interface for uploading files in small programs
  • 4. View of the interface for uploading files of applets
• 5, Actual test

In some application scenarios of the applet, there will be a need for voice to text. The original method is to record the voice file through the recording function of the small program, and then transfer the voice file to the text message by calling the voice intelligent recognition WebApi (for example, baidu cloud AI platform, iFLYTEK platform). The above method is cumbersome and the user experience is poor.
In order to solve this problem, wechat directly opens the plug-in of simultaneous interpretation, which can be directly used by the author of small program to develop simultaneous interpretation. This article will complete the real-time conversion of voice through a complete case of front and back-end integrated application, and upload the voice to the server background backup.

WeChat simultaneous interpreting is a simultaneous transmission interface launched by WeChat Chi Ling's voice team, WeChat translation team and public platform. The first phase opens the interface of voice to text, text translation and speech synthesis, and provides energy for developers.

Enter wechat applet background - > Enter settings - > third party settings - > Add plug-ins - > search simultaneous interpretation - > finish adding.

In applet app.json Add plug-in version and other information to the file:

"plugins": { "WechatSI": { "version": "0.3.3", "provider": "wx069ba97219f66d99" } },

To introduce a plug-in into a pager file:

/* index.js */ const plugin = requirePlugin("WechatSI") // Get * * globally unique * * voice recognition manager * * recordRecoManager** const manager = plugin.getRecordRecognitionManager()

List of methods for the recordRecoManager object:

Official development documents: Speech recognition manager for plug-ins

The UI refers to the official DEMO of wechat: press and hold the button for recording, and release the button to convert the recording into text in real time.

Users can edit the synchronously converted text and upload the original voice file and text to the background server.

The main code of voice synchronization conversion:

//Import plug-ins const plugin = requirePlugin("WechatSI"); // Get * * globally unique * * voice recognition manager * * recordRecoManager** const manager = plugin.getRecordRecognitionManager(); /** * Load for initialization */ onLoad: function () { //Get recording permission app.getRecordAuth(); //Initialize speech recognition callback this.initRecord(); }, ... /** * Initialize speech recognition callback * Bind voice playback start event */ initRecord: function () { //This event will be called if there is a new identification returned manager.onRecognize = (res) => { let currentData = Object.assign({}, this.data.currentTranslate, { text: res.result, }); this.setData({ currentTranslate: currentData, }); this.scrollToNew(); }; // Identify end events manager.onStop = (res) => { let text = res.result; console.log(res.tempFilePath); if (text == "") { this.showRecordEmptyTip(); return; } let lastId = this.data.lastId + 1; let currentData = Object.assign({}, this.data.currentTranslate, { text: res.result, translateText: "Identifying", id: lastId, voicePath: res.tempFilePath, duration: res.duration }); this.setData({ currentTranslate: currentData, recordStatus: 1, lastId: lastId, }); //Add the current recognition content and voice file to the list this.addRecordFile(currentData, this.data.dialogList.length); //Refresh List this.scrollToNew(); }; // Identify error events manager.onError = (res) => { this.setData({ recording: false, bottomButtonDisabled: false, }); }; }, /** * Press and hold the button to start speech recognition */ streamRecord: function (e) { let detail = e.detail || {}; let buttonItem = detail.buttonItem || {}; //Start Chinese recording manager.start({ lang: buttonItem.lang, }); this.setData({ recordStatus: 0, recording: true, currentTranslate: { // Current voice input create: util.recordTime(new Date()), text: "Listening", lfrom: buttonItem.lang, lto: buttonItem.lto, }, }); //Refresh List this.scrollToNew(); }, /** * Release button to end speech recognition */ streamRecordEnd: function (e) { let detail = e.detail || {}; // The detail object provided when a custom component triggers an event let buttonItem = detail.buttonItem || {}; // Prevent repeated triggering of stop function if (!this.data.recording || this.data.recordStatus != 0) { console.warn("has finished!"); return; } manager.stop(); this.setData({ bottomButtonDisabled: true, }); },

Edit the identification text and finish uploading the main code:

/** * Initial data of the page */ data: { edit_text_max: 200, remain_length: 200, edit_text: "", is_focus: false, tips: "", index: -1, voicePath: "", }, /** * Load initialization */ onLoad: function (options) { //Fill in the edit box based on the incoming text content this.setEditText(options.content) this.setData({ index: index, oldText:options.content, voicePath: options.voicePath }) }, /** * Edit text */ editInput: function (event) { console.log(event) if (event.detail.value.length > this.getEditTextMax()) { } else { this.data.edit_text = event.detail.value this.updateRemainLength(this.data.edit_text) } }, /** * Upload text and voice files */ editConfirm: function (event) { let json=this.data.edit_text //Call the wechat upload file api to upload the information to the server webApi wx.uploadFile({ url: api.wxFileUploadUrl, filePath: this.data.voicePath, name: "file", header: { Authorization: wx.getStorageSync("loginFlag"), "Content-Type": "multipart/form-data", }, formData: { openId: app.globalData.userInfo.openId, realName: "Voice files", json: JSON.stringify(json), }, success: (result) => { console.log("success:", result); if (result.statusCode == "200") { let data = JSON.parse(result.data); console.log("data", data); if (data.success == true) { let module = data.module; console.log("module", module); app.showInfo("Upload succeeded"); setTimeout( ()=>{ wx.navigateBack(); }, 2000) } else { app.showInfo("Unexpected error" + data.errMsg + ",Please re-enter"); wx.navigateTo({ url: "/pages/index/index", }); } } else { app.showInfo("Access background exception,Re entering the system"); wx.navigateTo({ url: "/pages/index/index", }); } }, fail: (result) => { console.log("fail", result); wx.navigateTo({ url: "/pages/index/index", }); }, complete: () => {}, }); },

The tool class package mainly stores the general file upload tool class, which will upload the file to the folder specified by the configuration and write the file information to upload_file table.

• File information entity class: upload with table in database_ File correspondence;
• File storage warehouse class: CRUD of data is realized through Spring Data JPA interface;
• File upload tool interface: encapsulate the file upload method in a unified way;
• File upload tool implementation class: implements the file upload method interface.

Document information entity class: UploadFile.java

/** * Document information sheet * * @author zhuhuix * @date 2020-04-20 */ @Entity @Getter @Setter @Table(name = "upload_file") public class UploadFile { @Id @GeneratedValue(strategy = GenerationType.IDENTITY) @NotNull(groups = Update.class) private Long id; /** * Actual name of document */ @Column(name = "real_name") private String realName; /** * file name */ @NotNull @Column(name = "file_name") private String fileName; /** * File master name */ @NotNull @Column(name = "primary_name") private String primaryName; /** * File extension */ @NotNull private String extension; /** * Storage path */ @NotNull private String path; /** * file type */ private String type; /** * file size */ private Long size; /** * Uploaded by */ private String uploader; @JsonIgnore @Column(name = "create_time") @CreationTimestamp private Timestamp createTime; public UploadFile(String realName, @NotNull String fileName, @NotNull String primaryName, @NotNull String extension, @NotNull String path, String type, Long size, String uploader) { this.realName = realName; this.fileName = fileName; this.primaryName = primaryName; this.extension = extension; this.path = path; this.type = type; this.size = size; this.uploader = uploader; } @Override public String toString() { return "UploadFile{" + "fileName='" + fileName + '\'' + ", uploader='" + uploader + '\'' + ", createTime=" + createTime + '}'; } }

File storage warehouse class: UploadFileRepository.java

/** * Upload file DAO interface layer * * @author zhuhuix * @date 2020-04-03 */ public interface UploadFileRepository extends JpaRepository<UploadFile, Long>, JpaSpecificationExecutor<UploadFile> { //This interface inherits the JpaRepository and CrudRepository interfaces, and has implemented CRUD methods such as findById,save,delete, etc }

The UploadFileRepository interface inherits the JpaRepository and CrudRepository interfaces, and has implemented CRUD methods such as findById,save,delete, etc

File upload tool interface: UploadFileTool.java

/** * File upload interface definition * * @author zhuhuix * @date 2020-04-20 */ public interface UploadFileTool { /** * File upload * @param multipartFile file * @return Upload information */ UploadFile upload(String uploader,String realName,MultipartFile multipartFile); }

File upload tool implementation class: UploadFileToolImpl.java

/** * File upload implementation class * * @author zhuhuix * @date 2020-04-20 */ @Service @Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class) public class UploadFileToolImpl implements UploadFileTool { private final UploadFileRepository uploadFileRepository; @Value("$") private String path; @Value("$") private long maxSize; public UploadFileToolImpl(UploadFileRepository uploadFileRepository) { this.uploadFileRepository = uploadFileRepository; } @Override @Transactional(rollbackFor = Exception.class) public UploadFile upload(String uploader, String realName, MultipartFile multipartFile) { //Check file size if (multipartFile.getSize() > maxSize * Constant.MB) { throw new RuntimeException("File upload size limit exceeded" + maxSize + "MB"); } //Get the main file name and extension of the uploaded file String primaryName = FileUtil.mainName(multipartFile.getOriginalFilename()); String extension = FileUtil.extName(multipartFile.getOriginalFilename()); //Get file type based on file extension String type = getFileType(extension); //Time stamp the uploaded file LocalDateTime date = LocalDateTime.now(); DateTimeFormatter format = DateTimeFormatter.ofPattern("yyyyMMddhhmmssS"); String nowStr = "-" + date.format(format); String fileName = primaryName + nowStr + "." + extension; try { String filePath = path + type + File.separator + fileName; File dest = new File(filePath).getCanonicalFile(); if (!dest.getParentFile().exists()) { dest.getParentFile().mkdirs(); } multipartFile.transferTo(dest); if (ObjectUtil.isNull(dest)) { throw new RuntimeException("Failed to upload file"); } UploadFile uploadFile = new UploadFile(realName, fileName, primaryName, extension, dest.getPath(), type, multipartFile.getSize(), uploader); return uploadFileRepository.save(uploadFile); } catch (Exception e) { e.printStackTrace(); throw new RuntimeException(e.getMessage()); } } /** * File type based on file extension * * @param extension File extension * @return file type */ private static String getFileType(String extension) { String document = "txt doc pdf ppt pps xlsx xls docx csv"; String music = "mp3 wav wma mpa ram ra aac aif m4a"; String video = "avi mpg mpe mpeg asf wmv mov qt rm mp4 flv m4v webm ogv ogg"; String image = "bmp dib pcp dif wmf gif jpg tif eps psd cdr iff tga pcd mpt png jpeg"; if (image.contains(extension)) { return "image"; } else if (document.contains(extension)) { return "document"; } else if (music.contains(extension)) { return "music"; } else if (video.contains(extension)) { return "video"; } else { return "other"; } } }

Note that the @ Value annotation is used in the program code to get the uploadFile.path and uploadFile.maxsize Parameters, usually written in the project static configuration file as follows (yml configuration file).

# Test environment file storage path uploadFile: path: C:\startup\file\ # File size / M maxSize: 50

The Wx miniprogram package defines the interface of CRM webApi, and the applet calls webApi to upload files and other functions.

• Wechat app webApi: provide external app upload file webApi;
• Wechat applet service interface: encapsulate the service interface for uploading files of applets;
• Wechat applet service implementation: the implementation of applet upload file service, which calls the UploadFile interface in tools package to upload files.

Wechat app CRM webApi: WxMiniCrmController.java

/** * Wechat applet Crm webApi * * @author zhuhuix * @date 2020-03-30 */ @Slf4j @RestController @RequestMapping("/api/wx-mini") @Api(tags = "Wechat applet Crm Interface") public class WxMiniCrmController { private final WxMiniCrm wxMiniCrm; public WxMiniCrmController(WxMiniCrm wxMiniCrm) { this.wxMiniCrm = wxMiniCrm; } @ApiOperation(value = "Upload files on wechat applet") @PostMapping(value = "/fileUpload") public ResponseEntity fileUpload(HttpServletRequest request) { MultipartHttpServletRequest req = (MultipartHttpServletRequest) request; MultipartFile multipartFile = req.getFile("file"); String openId = req.getParameter("openId"); String realName = req.getParameter("realName"); String json = req.getParameter("json"); return ResponseEntity.ok(wxMiniCrm.uploadFile(json, openId,realName, multipartFile)); } }

Wechat applet CRM service interface: WxMiniCrm.java

/** * Definition of CRM service interface of wechat applet * * @author zhuhuix * @date 2020-04-20 */ public interface WxMiniCrm { /** * Write the json object passed in by the wechat applet to the database, and upload the file to the server at the same time * * @param json Incoming json object from wechat * @param openId Uploaded by * @param realName Actual name of document * @param multipartFile Upload file * @return Return to upload information */ Result<UploadFile> uploadFile(String json, String openId, String realName,MultipartFile multipartFile); }

Wechat applet CRM service implementation: WxMiniCrmImpl.java

/** * Wechat applet CRM implementation class * * @author zhuhuix * @date 2020-04-20 */ @Slf4j @Service @Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class) public class WxMiniCrmImpl implements WxMiniCrm { private final UploadFileTool uploadFileTool; public WxMiniCrmImpl(UploadFileTool uploadFileTool) { this.uploadFileTool = uploadFileTool; } @Override @Transactional(rollbackFor = Exception.class) public Result<UploadFile> uploadFile(String json, String openId,String realName, MultipartFile multipartFile) { return new Result<UploadFile>().ok(uploadFileTool.upload(openId,realName, multipartFile)); } }

Visit Swagger2 to see the interface, and refer to the integration of Swagger2 and SpringBoot SpringBoot JWT authentication mechanism project integration Swagger2

Voice test OK

Upload file to background:

View the uploaded log information:

29 May 2020, 02:03 | Views: 1524