nginx worker process loop

When the worker process starts, it first initializes the environment it needs to run, then it enters a cycle in which it checks for events to execute and then processes them.In this process, the worker process also needs to interact with the master process, and more importantly, as a child process, the worker process can also receive command line instructions (such as kill, and so on) for the corresponding logical processing.How do worker processes interact with masters or command line instructions?This paper first explains how the worker process interacts with the master process and how the worker process handles command line instructions. Then, the whole workflow of worker process interaction is introduced from the source code.

1. How the worker interacts with the master process

The first thing to note here is that nginx handles instructions either by master or by external commands, that is, when a command is received, whether master or external, the worker sets the corresponding flag bit in its callback method, and then the worker process itselfAfter the event is processed in the loop, these flags are checked for true in turn, and the logic is executed based on the role of the flags.

The worker process interacts with the master process through the socket pipeline.A ngx_process_t structure is declared in the ngx_process.h file, where we focus on its channel property:

typedef struct {
  	// The rest of the properties...
    
    ngx_socket_t channel[2];
} ngx_process_t;

The ngx_process_t structure here stores information about a process, such as pid, channel, status, and so on.Each process has an array of ngx_processes, where the array element is the ngx_process_t structure, meaning that each process stores basic information about the remaining processes through the ngx_processes array.It is declared as follows:

// Stores an array of all subprocesses in nginx, each of which is marked with a corresponding ngx_process_t structure
extern ngx_process_t ngx_processes[NGX_MAX_PROCESSES];

Here we can see that each process has a corresponding channel array, which is 2 in length and is the pipe flow that interacts with the master process.Before the master process creates each child process, a channel array is created by:

int socketpair(int domain, int type, int protocol, int sv[2]);

The primary function of this method is to create an anonymous pair of connected sockets, that is, if data is written to one socket, the written data can be received to the other socket.In this way, if data is written to one side of the pipe in the parent process, data can be received on the other side of the pipe in the child process, which enables data communication between the parent and child processes.

After the master process has started the child process, the child process retains the corresponding data in the master process, including the channel array here.In this way, the master process can communicate with the child process through a channel array.

2. worker processes external commands

External commands are essentially handled by the individual signals and callback methods defined in the signals array.When the master process initializes the basic environment, it sets the signal callback method specified in the signals array to the corresponding signal.Since the worker process inherits the basic environment of the master process, the worker process also calls back methods when it receives the signal set here.The main logic of this callback method is simply to set the value of the corresponding flag bit.You can refer to my previous article (nginx master work cycle hyperlink) on how to set the corresponding flag bit after nginx receives the signal, which is not repeated here.

3. Source explanation

The master process starts each child process by using the ngx_start_worker_processes() method, as follows:

/**
 * Start n worker child processes and set the socketpair between each child process and master parent process
 * socket Handle Communication Mechanism Established by System Calls
 */
static void ngx_start_worker_processes(ngx_cycle_t *cycle, ngx_int_t n, ngx_int_t type) {
  ngx_int_t i;
  ngx_channel_t ch;
  
  ngx_memzero(&ch, sizeof(ngx_channel_t));
  ch.command = NGX_CMD_OPEN_CHANNEL;

  for (i = 0; i < n; i++) {

    // spawn means spawning, which means generating a child process whose event cycle is
    // The ngx_worker_process_cycle() method, where ngx_worker_process_cycle is the loop in which the worker process handles events.
    // The worker process in an infinite for loop continuously checks for events in the corresponding event model.
    // It then separates accept events from read and write events into two queues, and finally continuously handles events in the event loop
    ngx_spawn_process(cycle, ngx_worker_process_cycle, 
                      (void *) (intptr_t) i, "worker process", type);

    // The main purpose of this code below is to notify other processes of the event of creating a new process.
    // ch.command = NGX_CMD_OPEN_CHANNEL; NGX_CMD_OPEN_CHANNEL in indicates that a new process is currently being created.
    // The ngx_process_slot store is the array location where the new process is stored. The reason for broadcasting is that,
    // After each child process is created, its memory data is the replicated parent process, but the ngx_processes array is one for each process.
    // Thus, the sub-processes created first in the array have no data for the sub-processes created later, but the master process has data for all the sub-processes.
    // So once the master process creates a child process here, it goes to channel[0] for each process in the ngx_processes array
    // Writes the currently broadcast event, the ch here, in which way each subprocess receives the event,
    // Will attempt to update its saved ngx_processes data information
    ch.pid = ngx_processes[ngx_process_slot].pid;
    ch.slot = ngx_process_slot;
    ch.fd = ngx_processes[ngx_process_slot].channel[0];

    // Broadcast events
    ngx_pass_open_channel(cycle, &ch);
  }
}

The main concern here is the method call above to start the child process, the ngx_spawn_process() method here, whose second parameter is a method that will enter the loop specified by the method after the child process is started.In the ngx_spawn_process() method, the master process creates a channel array for the currently created subprocess to communicate with the current subprocess.The following is the source code for the ngx_spawn_process() method:

ngx_pid_t ngx_spawn_process(ngx_cycle_t *cycle, ngx_spawn_proc_pt proc, void *data, char *name, ngx_int_t respawn) {
  u_long on;
  ngx_pid_t pid;
  ngx_int_t s;

  if (respawn >= 0) {
    s = respawn;

  } else {
    // All currently created processes are stored in the ngx_processes array, and ngx_last_process is the last of the current records
    // Index of the next location of process in ngx_processes, except that processes recorded in ngx_processes may be partially
    // It is no longer valid.The current loop is to find out from scratch if a process has failed and, if it has, to reuse the process location.
    // Otherwise, use the location pointed to by ngx_last_process directly
    for (s = 0; s < ngx_last_process; s++) {
      if (ngx_processes[s].pid == -1) {
        break;
      }
    }

    // This means that the maximum number of processes has been created
    if (s == NGX_MAX_PROCESSES) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, 0,
                    "no more than %d processes can be spawned",
                    NGX_MAX_PROCESSES);
      return NGX_INVALID_PID;
    }
  }

  // The NGX_PROCESS_DETACHED flag indicates that the current fork process has nothing to do with the original parent process, for example, when upgrading nginx,
  // The newly generated master process has nothing to do with the original master process
  if (respawn != NGX_PROCESS_DETACHED) {

    /* Solaris 9 still has no AF_LOCAL */

    // The main function of the socketpair() method here is to generate a pair of socket streams for communication between the main process and the child process, which will
    // Stored in ngx_processes[s].channel, this field is essentially an integer array of length 2.Primary and subprocesses
    // Before communicating, the main process shuts down one, while the child process shuts down the other, and then goes to another file descriptor that is not closed.
    // Communication can be achieved by writing or reading data.
    // AF_UNIX indicates that the socket address family in UNIX file form is currently in use
    // SOCK_STREAM specifies that the communication established by the current socket is a pipeline flow, which is bidirectional.
    // That is, both sides of the pipe can read and write
    // The third parameter, protocol, must be zero
    if (socketpair(AF_UNIX, SOCK_STREAM, 0, ngx_processes[s].channel) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "socketpair() failed while spawning \"%s\"", name);
      return NGX_INVALID_PID;
    }

    ngx_log_debug2(NGX_LOG_DEBUG_CORE, cycle->log, 0,
                   "channel %d:%d",
                   ngx_processes[s].channel[0],
                   ngx_processes[s].channel[1]);

    // Set ngx_processes[s].channel[0] to non-blocking mode
    if (ngx_nonblocking(ngx_processes[s].channel[0]) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    ngx_nonblocking_n
                        " failed while spawning \"%s\"",
                    name);
      ngx_close_channel(ngx_processes[s].channel, cycle->log);
      return NGX_INVALID_PID;
    }

    // Set ngx_processes[s].channel[1] to non-blocking mode
    if (ngx_nonblocking(ngx_processes[s].channel[1]) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    ngx_nonblocking_n
                        " failed while spawning \"%s\"",
                    name);
      ngx_close_channel(ngx_processes[s].channel, cycle->log);
      return NGX_INVALID_PID;
    }

    on = 1;
    // Set the ngx_processes[s].channel[0] socket pipe to asynchronous mode
    if (ioctl(ngx_processes[s].channel[0], FIOASYNC, &on) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "ioctl(FIOASYNC) failed while spawning \"%s\"", name);
      ngx_close_channel(ngx_processes[s].channel, cycle->log);
      return NGX_INVALID_PID;
    }

    // It is currently in the main process, where ngx_pid points to the process id of the main process. The main purpose of the current method is to
    // The operation privileges of ngx_processes[s].channel[0] are set to the main process, that is, the main process passes to the
    // ngx_processes[s].channel[0] Writes and reads data to communicate with subprocesses
    if (fcntl(ngx_processes[s].channel[0], F_SETOWN, ngx_pid) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "fcntl(F_SETOWN) failed while spawning \"%s\"", name);
      ngx_close_channel(ngx_processes[s].channel, cycle->log);
      return NGX_INVALID_PID;
    }

    // FD_CLOEXEC indicates that the socket pipe currently specified can be used in a child process, but not in a program execl() executes
    if (fcntl(ngx_processes[s].channel[0], F_SETFD, FD_CLOEXEC) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "fcntl(FD_CLOEXEC) failed while spawning \"%s\"",
                    name);
      ngx_close_channel(ngx_processes[s].channel, cycle->log);
      return NGX_INVALID_PID;
    }

    // FD_CLOEXEC indicates that the socket pipe currently specified can be used in a child process, but not in a program execl() executes
    if (fcntl(ngx_processes[s].channel[1], F_SETFD, FD_CLOEXEC) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "fcntl(FD_CLOEXEC) failed while spawning \"%s\"",
                    name);
      ngx_close_channel(ngx_processes[s].channel, cycle->log);
      return NGX_INVALID_PID;
    }

    // ngx_processes[s].channel[1] is used to listen for related events for a child process when the parent process directs
    // After the event is published by ngx_processes[s].channel[0], it will be received in ngx_processes[s].channel[1]
    // Corresponding events for processing
    ngx_channel = ngx_processes[s].channel[1];

  } else {
    // In the case of NGX_PROCESS_DETACHED mode, it indicates that another master process is currently emerging, so its pipe values are set to -1
    ngx_processes[s].channel[0] = -1;
    ngx_processes[s].channel[1] = -1;
  }

  ngx_process_slot = s;


  // The fork() method will result in a new process whose relationship to the parent process is that the memory data of the child process will replicate the parent process completely.
  // It is also important to note that the code executed by the child process from fork() starts after fork(), whereas for the parent process,
  // The return value of this method is the parent process id, and 0 for the child process, so if-else statements allow the parent process
  // Call subsequent different snippets of code separately from the child process
  pid = fork();

  switch (pid) {

    case -1:
      // fork error
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "fork() failed while spawning \"%s\"", name);
      ngx_close_channel(ngx_processes[s].channel, cycle->log);
      return NGX_INVALID_PID;

    case 0:
      // The branch of the subprocess execution where the proc() method is passed in externally, that is, the current method simply creates a new process.
      // Specific process processing logic that will be handed over to an external code block to define the ngx_getpid() method takes the process id of the currently created child process
      ngx_pid = ngx_getpid();
      proc(cycle, data);
      break;

    default:
      // The parent process will come here
      break;
  }

  ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "start %s %P", name, pid);

  // The parent process will come here and the current pid is the pid of the newly created child process that the parent process got after fork()
  ngx_processes[s].pid = pid;
  ngx_processes[s].exited = 0;

  if (respawn >= 0) {
    return pid;
  }

  // Set the properties of the current process and store them in the corresponding location in the ngx_processes array
  ngx_processes[s].proc = proc;
  ngx_processes[s].data = data;
  ngx_processes[s].name = name;
  ngx_processes[s].exiting = 0;

  switch (respawn) {

    case NGX_PROCESS_NORESPAWN:
      ngx_processes[s].respawn = 0;
      ngx_processes[s].just_spawn = 0;
      ngx_processes[s].detached = 0;
      break;

    case NGX_PROCESS_JUST_SPAWN:
      ngx_processes[s].respawn = 0;
      ngx_processes[s].just_spawn = 1;
      ngx_processes[s].detached = 0;
      break;

    case NGX_PROCESS_RESPAWN:
      ngx_processes[s].respawn = 1;
      ngx_processes[s].just_spawn = 0;
      ngx_processes[s].detached = 0;
      break;

    case NGX_PROCESS_JUST_RESPAWN:
      ngx_processes[s].respawn = 1;
      ngx_processes[s].just_spawn = 1;
      ngx_processes[s].detached = 0;
      break;

    case NGX_PROCESS_DETACHED:
      ngx_processes[s].respawn = 0;
      ngx_processes[s].just_spawn = 0;
      ngx_processes[s].detached = 1;
      break;
  }

  if (s == ngx_last_process) {
    ngx_last_process++;
  }

  return pid;
}

The ngx_spawn_process() method eventually fork() a subprocess to execute the callback method specified by its second parameter.But before that, we need to note that it creates a pair of anonymous sockets through a socketpair() method call, then stores them in the channel array of the current process, which completes the creation of the channel array.

The ngx_worker_process_cycle() method is executed after the worker process starts, which first initializes the worker process, including the processing of inherited channel arrays.Since both master and worker processes hold the socket descriptor referred to by the channel array, master processes and worker processes essentially only need the descriptor of one side of the array.Thus, the worker process closes its saved descriptor on the other side during initialization.In nginx, the master process unify keeps the 0 bit socket descriptor of the channel array, closes the 1 bit socket descriptor, and the worker process closes the 0 bit socket descriptor, leaving the 1 bit descriptor.In this way, when the master process needs to communicate with the worker process, it only needs to write data to channel[0], while the worker process listens to channel[1] and receives data writes from the master process.Let's first look at the source code for the worker process initialization method ngx_worker_process_init():

/**
 * This is mainly to initialize the current process, set its priority and open file limit parameters.
 * Finally, a connection to listen on channel[1] is added to the current process to constantly read messages from the master process for processing
 */
static void ngx_worker_process_init(ngx_cycle_t *cycle, ngx_int_t worker) {
  sigset_t set;
  ngx_int_t n;
  ngx_time_t *tp;
  ngx_uint_t i;
  ngx_cpuset_t *cpu_affinity;
  struct rlimit rlmt;
  ngx_core_conf_t *ccf;
  ngx_listening_t *ls;

  // Set up time zone related information
  if (ngx_set_environment(cycle, NULL) == NULL) {
    /* fatal */
    exit(2);
  }

  ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);

  // Set the priority of the current process
  if (worker >= 0 && ccf->priority != 0) {
    if (setpriority(PRIO_PROCESS, 0, ccf->priority) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "setpriority(%d) failed", ccf->priority);
    }
  }

  // Set the number of file handles that the current process can open
  if (ccf->rlimit_nofile != NGX_CONF_UNSET) {
    rlmt.rlim_cur = (rlim_t) ccf->rlimit_nofile;
    rlmt.rlim_max = (rlim_t) ccf->rlimit_nofile;

    if (setrlimit(RLIMIT_NOFILE, &rlmt) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "setrlimit(RLIMIT_NOFILE, %i) failed",
                    ccf->rlimit_nofile);
    }
  }

  // Changes the limit on the largest size of a core file(RLIMIT_CORE) for worker processes.
  // In short, set the maximum size a core file can use
  if (ccf->rlimit_core != NGX_CONF_UNSET) {
    rlmt.rlim_cur = (rlim_t) ccf->rlimit_core;
    rlmt.rlim_max = (rlim_t) ccf->rlimit_core;

    if (setrlimit(RLIMIT_CORE, &rlmt) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "setrlimit(RLIMIT_CORE, %O) failed",
                    ccf->rlimit_core);
    }
  }

  // geteuid() returns the user id to execute the current program, where 0 indicates whether or not the root user is
  if (geteuid() == 0) {
    // The setgid() method changes the group's id
    if (setgid(ccf->group) == -1) {
      ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
                    "setgid(%d) failed", ccf->group);
      /* fatal */
      exit(2);
    }

    // initgroups() is the id to change additional groups
    if (initgroups(ccf->username, ccf->group) == -1) {
      ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
                    "initgroups(%s, %d) failed",
                    ccf->username, ccf->group);
    }

    // Change the user's id
    if (setuid(ccf->user) == -1) {
      ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_errno,
                    "setuid(%d) failed", ccf->user);
      /* fatal */
      exit(2);
    }
  }

  // Note that for the cache manager and cache loader processes, the worker here passes in -1,
  // Indicates that the two processes do not need to be nucleophilic
  if (worker >= 0) {
    // Get the CPU nucleophilicity of the current worker
    cpu_affinity = ngx_get_cpu_affinity(worker);

    if (cpu_affinity) {
      // Setting up worker's parent core
      ngx_setaffinity(cpu_affinity, cycle->log);
    }
  }

#if (NGX_HAVE_PR_SET_DUMPABLE)
  if (prctl(PR_SET_DUMPABLE, 1, 0, 0, 0) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "prctl(PR_SET_DUMPABLE) failed");
  }

#endif

  if (ccf->working_directory.len) {
    // The purpose of chdir() is to change the current working directory to the path passed in for its parameters
    if (chdir((char *) ccf->working_directory.data) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "chdir(\"%s\") failed", ccf->working_directory.data);
      /* fatal */
      exit(2);
    }
  }

  // Initialize an empty set of instructions
  sigemptyset(&set);

  // _SIG_BLOCK: Adds a set parameter to the signal in the signal set into the signal mask.
  // _SIG_UNBLOCK: Removes the signal in the set pointed to by the set parameter from the signal mask.
  // _SIG_SETMASK: Set the set parameter to point to the signal set as a signal mask.
  // Here is the direct initialization of the blocked semaphore set, which is empty by default
  if (sigprocmask(SIG_SETMASK, &set, NULL) == -1) {
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  "sigprocmask() failed");
  }

  tp = ngx_timeofday();
  srandom(((unsigned) ngx_pid << 16) ^ tp->sec ^ tp->msec);

  ls = cycle->listening.elts;
  for (i = 0; i < cycle->listening.nelts; i++) {
    ls[i].previous = NULL;
  }

  // Here the init_process() method of each module is called to initialize the process module
  for (i = 0; cycle->modules[i]; i++) {
    if (cycle->modules[i]->init_process) {
      if (cycle->modules[i]->init_process(cycle) == NGX_ERROR) {
        /* fatal */
        exit(2);
      }
    }
  }

  // This is mainly to close channel[1] pipe handles for other processes in the current process
  for (n = 0; n < ngx_last_process; n++) {

    if (ngx_processes[n].pid == -1) {
      continue;
    }

    if (n == ngx_process_slot) {
      continue;
    }

    if (ngx_processes[n].channel[1] == -1) {
      continue;
    }

    if (close(ngx_processes[n].channel[1]) == -1) {
      ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                    "close() channel failed");
    }
  }

  // Close channel[0] pipe handle for current process
  if (close(ngx_processes[ngx_process_slot].channel[0]) == -1) {
    ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                  "close() channel failed");
  }

#if 0
  ngx_last_process = 0;
#endif

  // ngx_channel points to the channel[1] handle of the current process, which is also the handle that listens for messages sent by the master process.
  // In the current method, a connection object is first created for the current handle, encapsulated as an event, and then added to
  // The corresponding event model queue listens for events with the current handle, and the main logic for handling events is ngx_channel_handler()
  // Method proceed.The main processing logic of ngx_channel_handler here is to set some flags for the current process based on the message currently received.
  // Or update some of the cached data so that in the current event cycle, these flags are constantly checked to make sense during the event process
  // Handle real logic.Therefore, the processing efficiency of ngx_channel_handler is very high here
  if (ngx_add_channel_event(cycle, ngx_channel, NGX_READ_EVENT,
                            ngx_channel_handler)
      == NGX_ERROR) {
    /* fatal */
    exit(2);
  }
}

This method primarily initializes the worker process, where we focus on the last ngx_processes array that holds information about each process in the current nginx.During traversal, the channel[1] handles of the remaining processes held by the current process are closed while the channel[0] handles are retained, so that if the current process needs to communicate with other processes, it only needs to write data to the channel[0] of the target process.After the traversal is complete, the current process closes its channel[0] handle, leaving the channel[1] handle.Finally, the ngx_add_channel_event() method adds a listening event to channel[1] for the current process. The second parameter passed in when calling the ngx_add_channel_event() method is ngx_channel, which is assigned in the previous ngx_spawn_process() method and points to the socket handle of channel[1] for the current process.

The essence of the ngx_add_channel_event() method is to create an event of the ngx_event_t structure and then add it to the event model (such as epoll) handle currently in use.The implementation source for this method is no longer detailed here, but what we need to focus on is the callback method when the event is triggered, the third parameter, the ngx_channel_handler() method, passed in when the ngx_add_channel_event() method is called.The source code for this method is as follows:

static void ngx_channel_handler(ngx_event_t *ev) {
  ngx_int_t n;
  ngx_channel_t ch;
  ngx_connection_t *c;

  if (ev->timedout) {
    ev->timedout = 0;
    return;
  }

  c = ev->data;

  for (;;) {

    // Keep reading messages from the master process in an infinite for loop
    n = ngx_read_channel(c->fd, &ch, sizeof(ngx_channel_t), ev->log);

    // If an error occurs reading the message, indicating that the current handle may be invalid, the current connection needs to be closed
    if (n == NGX_ERROR) {
      if (ngx_event_flags & NGX_USE_EPOLL_EVENT) {
        ngx_del_conn(c, 0);
      }

      ngx_close_connection(c);
      return;
    }

    if (ngx_event_flags & NGX_USE_EVENTPORT_EVENT) {
      if (ngx_add_event(ev, NGX_READ_EVENT, 0) == NGX_ERROR) {
        return;
      }
    }

    if (n == NGX_AGAIN) {
      return;
    }

    // Process incoming messages
    switch (ch.command) {
      // If it is a quit message, set the quit flag bit
      case NGX_CMD_QUIT:
        ngx_quit = 1;
        break;

        // If terminate message, set terminate flag bit
      case NGX_CMD_TERMINATE:
        ngx_terminate = 1;
        break;

        // If it is a reopen message, set the reopen flag bit
      case NGX_CMD_REOPEN:
        ngx_reopen = 1;
        break;

        // If a new process message is created, update the data for the corresponding location of the current ngx_processes array
      case NGX_CMD_OPEN_CHANNEL:
        ngx_processes[ch.slot].pid = ch.pid;
        ngx_processes[ch.slot].channel[0] = ch.fd;
        break;

        // If the message closes the channel, close the handle to the corresponding location of the ngx_processes array
      case NGX_CMD_CLOSE_CHANNEL:
        if (close(ngx_processes[ch.slot].channel[0]) == -1) {
          ngx_log_error(NGX_LOG_ALERT, ev->log, ngx_errno,
                        "close() channel failed");
        }

        ngx_processes[ch.slot].channel[0] = -1;
        break;
    }
  }
}

In the ngx_channel_handler() method, the data in the socket handle being listened for is mainly read, and the data is hosted by a ngx_channel_t structure, which is the structure used by nginx to communicate with the worker process by the master, which specifies the type of event currently occurring and the process information for the event.The following is the declaration of the ngx_channel_t structure:

typedef struct {
  	// Current Event Type
    ngx_uint_t command;
  	// The pid of the event
    ngx_pid_t pid;
  	// The subscript of the process in which the event occurred in the ngx_processes array
    ngx_int_t slot;
  	// Value of channel[0] descriptor for the process in which the event occurred
    ngx_fd_t fd;
} ngx_channel_t;

After reading the data of the ngx_channel_t structure from the current process's channel[1], the ngx_channel_handler() method updates the status of the corresponding flag bits based on the type of event that occurs, and updates the status information of the corresponding event-occurring process in the current process's ngx_processes array.

After handling the events sent by the master process, the worker process continues its cycle, in which it checks the status of the flags it is interested in, and then executes the corresponding logic based on those states.The source code for the loop in which the worker process works is as follows:

/**
 * Enter a loop of worker process work
 */
static void ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data) {
  ngx_int_t worker = (intptr_t) data;

  ngx_process = NGX_PROCESS_WORKER;
  ngx_worker = worker;

  // Initialize the worker process, which was previously explained in the source code
  ngx_worker_process_init(cycle, worker);

  ngx_setproctitle("worker process");

  for (;;) {

    if (ngx_exiting) {
      // This is to check if there are any events that are not cancelable, that is, if all events have been cancelled, if cancelled,
      // NGX_OK is returned.The logic here can be understood as, if marked as ngx_exiting, then at this point, if there are still uncancelled
      // If an event exists, it will go to the ngx_process_events_and_timers() method below, which will handle the incomplete event.
      // Then walk to this location again in the loop, and the final if condition is true, which executes the work of exiting the worker process
      if (ngx_event_no_timers_left() == NGX_OK) {
        ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
        ngx_worker_process_exit(cycle);
      }
    }

    ngx_log_debug0(NGX_LOG_DEBUG_EVENT, cycle->log, 0, "worker cycle");

    // Here, by checking if there is a corresponding event in the corresponding event model and queuing it for processing,
    // This is the core way the worker process handles events
    ngx_process_events_and_timers(cycle);

    // Here ngx_terminate is the option to force nginx to close. If the command to force nginx to close is sent to nginx, the current process will exit directly
    if (ngx_terminate) {
      ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
      ngx_worker_process_exit(cycle);
    }

    // Here ngx_quit is the elegant exit option.This is mainly to set ngx_exiting to 1 to indicate that the current process needs to exit.
    // The following three tasks are then performed:
    // 1. Add an event to the event queue to handle the connection that is currently active, place its close flag at 1, and execute the connection
    //    Current handling methods to complete connection events as soon as possible;
    // 2. Close the socket handle listening in the current cycle;
    // 3. Mark the close state of all currently idle connections as 1, and then call their connection handling method.
    if (ngx_quit) {
      ngx_quit = 0;
      ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "gracefully shutting down");
      ngx_setproctitle("worker process is shutting down");

      if (!ngx_exiting) {
        ngx_exiting = 1;
        ngx_set_shutdown_timer(cycle);
        ngx_close_listening_sockets(cycle);
        ngx_close_idle_connections(cycle);
      }
    }

    // ngx_reopen mainly reopens all files of nginx, such as switching log files of nginx, etc.
    if (ngx_reopen) {
      ngx_reopen = 0;
      ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "reopening logs");
      ngx_reopen_files(cycle, -1);
    }
  }
}

You can see that the worker process mainly handles whether nginx exits the associated flags and whether nginx reads the configuration file flags again.

4. Summary

This paper first explains the basic principles of master-worker process interaction, then goes deep into the source code to explain how nginx implements the communication between master and worker processes.

Tags: Programming socket Nginx REST Unix

Posted on Mon, 03 Feb 2020 23:18:28 -0500 by varun_146100