This article mainly introduces how to use nsq in nodeJs, and other implementations will be output in subsequent articles.

cause

Some time ago, I made a node service to generate pdf from web pages. In the process of generating puppeter and canvas, the consumption of memory is relatively large, the generation time of pages with large internal capacity is too long, and the third-party components sometimes generate problems.

nsq is introduced to realize load balancing and eliminate single point fault.

But after searching on the Internet, we found that there are few schemes to introduce nsq into node. After soft film and hard bubble, we finally introduced nsq into node. I hope we can talk about our harvest with you.

Get to know nsq

NSQ is a distributed real-time message platform based on Go language. It has a distributed and decentralized topology and supports infinite level expansion. There is no single point of failure, fault tolerance, high availability and reliable transmission of messages. In addition, NSQ is very easy to configure and deploy, and supports many message protocols. Support multiple clients, simple protocol.

nsq design is very simple, need to understand the following core concepts.

1. nsqd: a daemons responsible for receiving, queuing and forwarding messages to clients
2. nsqlookupd: manages topology information, collects topic s and channel s reported by nsqd, and provides the final consistent discovery service daemons.
3. Topic: a topic is a logical key for a program to publish messages. When a program publishes messages for the first time, it will create a topic.
4. Channels: channel group is related to consumers, which is the load balance between consumers. In a sense, channel is a "queue". Every time a publisher sends a message to a topic, the message will be copied to all channels connected by consumers. Consumers read the message through this special channel. In fact, channels will be created when consumers first subscribe.
Topic can only have one channel, and different channels can be used to distribute different tasks

A schematic diagram of nsq of Jindian is attached below:

Problems encountered during installation

It is found that the gcc version is too low during use, which results in an error.
because node.js 4. The v8 engine has been upgraded, which requires gcc version above 4.8.

Start of actual combat

This project is based on eggjs. First you need to install nsqjs in the project

$ npm install nsqjs --save

In the root file app.js nsq control, nsq configuration is very simple, nsq is divided into write and read two independent processes.

 
const nsq = require('nsqjs')

module.exports = app => {
  app.beforeStart(async () => {
    // Write operation of instantiating nsq
    // Configure the host and port of nsq in config. Here is the nsq address you configured
    const writerNsq = new nsq.Writer(app.config.nsq.nsqHostWriter, app.config.nsq.writePort)

    // Write function of connecting nsq
    writerNsq.connect()

    // When the write operation is connected successfully, assign it to the global app to write information
    writerNsq.on('ready', () => {
      app.writerNsq = writerNsq
    })

When nsq write function is implemented, we can write information to nsq queue through publish method

ctx.app.writerNsq.publish(config.nsq.topic, {
        // Parameters you need to pass
      })

When the server is idle, nsq will be randomly assigned to the idle thread to implement the read operation. Our core business is implemented after the read function is enabled.
The initialization process of read and write is performed when the project starts.
During the operation of the project, we just kept reading and writing.

    // Read operation of instantiating nsq
    // The parameter is the address corresponding to the topic to be read, the channel to be read, and the nsq to be read
    const client = new nsq.Reader(app.config.nsq.topic, app.config.nsq.channel, { 
      lookupdHTTPAddresses: app.config.nsq.nsqHostReader,
      maxInFlight: 1
    })

    // Read function of connecting nsq
    client.connect()
 
    // Called whenever a message queue comes in client.on method
    // Message is the message passed in by our door during the process of writing
    client.on method('message', async msg => {
      // Format written information
      let data = JSON.parse(msg.body.toString())
      try {
      // To maintain the connection state, handle the timeout condition
        const touch = () => {
          if (!msg.hasResponded) {
            msg.touch()

            // Touch the message again a second before the next timeout.
            setTimeout(touch, msg.timeUntilTimeout() - 1000)
          }
        }

        let timeTouch =  setTimeout(touch, msg.timeUntilTimeout() - 1000)

        let timeFinish = setTimeout(msg.finish.bind(msg), msg.timeUntilTimeout() * 3 + 1000)
        
        // Here is the core processing part of the project. The specific content will be explained later. The url returned here is the network address for generating pdf

        let url = await ctx.service.pdf.index.generate(data)

        clearTimeout(timeTouch)
        clearTimeout(timeFinish)
        
        // This is the end of the queue. Tell nsq to put down a brother and come in
        msg.finish()

      } catch (error) {
        // In case of any mistake, do not block, nsq will rejoin the team after failure
        msg.finish()
        // You can join the network log here
        console.log(error)
      }
    });
 
    client.on('error', function(err) {
      // Error handling occurs when listening to read operation here
      // Here you can do some error handling and add the error log
      console.log(err)
    });
  });
};

At the beginning, I was a bit flustered to use nsq. After all, as a front-end engineer, I was confused. However, after careful study and consultation with the back-end tycoons, I found that nsq is actually an efficient queue, easy to use, and relatively simple to use.
In the following articles, the core business of generating PDF service by puppeter will be introduced in detail, that is, the above ctx.service.pdf.index.generate(data) specific implementation process.
The above is just my learning summary. If you have any questions, please let me know.