Design logger log system
The circle in the figure is the mprpc framework we implemented. This framework is for others to use. It publishes local services as remote RPC services. The two most important members of the framework are RpcProvider and RpcChannel. They will have a lot of normal output information and error information during operation. We can't cout them on the screen because the running time is long, There is a lot of information output on the screen. If there is any problem, it is difficult for us to locate it and it is inconvenient to use it. Therefore, the most direct way to solve problems is to read the log.
The log can record the information and error information during the normal operation of the software. When we locate the problem, we open the corresponding log to check and find it.
If we output a log module behind the framework, what should we do if we want to record the normal information and error information during the operation of the framework in the log file?
The two arrows on the left represent RPC requests and RPC responses. When RPC requests come, our framework will generate many log files during execution. We need to write logs. The process of writing log information is disk I/O. the speed of disk I/O is not fast. We can't count the cost of disk I/O in the business execution department of RPC requests (otherwise, the efficiency of RPC request processing will be slow), We can't count the time spent on logging into the execution time of the framework business. Therefore, generally, adding a kafka to our server can be used as a Message Queuing Middleware in the logging system. We write logs in a cache queue (this queue is equivalent to an asynchronous log writing mechanism), What our framework does is write logs to the memory queue without disk I/O operations. Then we have a special log writing thread in the back, which is to do disk I/O operations, take the log information from the queue head, and then write the log information to the log file, so that its disk I/O will not be included in our RPC request business.
The RpcProvier end of our mprpc framework is implemented with the muduo library, using epoll and multithreading. It is likely that the RPC processing process will be done in multiple threads, and multiple threads will write logs, that is, multiple threads will add data to the cache queue, so our cache queue must ensure thread safety.
There is a queue in our c + +, that is, the C + + container. However, the C + + container only considers the use of applications and does not consider thread safety. Therefore, we use the thread mutual exclusion mechanism to maintain the threads entering and leaving the queue. According to, the thread writing logs is also an independent thread, which is implemented with a unique mutual exclusion lock.
If the queue is empty, that is, the previous logs are written to the log file. At this time, the thread writing the log does not have to grab this mutex because there is nothing to write. As a result, the thread writing to the queue cannot obtain the lock in time, and the information cannot be written to the queue in time, which destroys the efficiency of RPC request business.
Therefore, what we have to deal with is the communication between threads.
If the queue is empty, the log writing thread will wait all the time. When the queue is not empty, it is necessary for the log writing thread to grab the lock and write the log information to the log file.
Our log files are placed in the current directory.
If the capacity of our log files is too large, for example, more than 20M, new log files will be generated!
Implement logger logging system
lockqueue.h
#pragma once
#include <condition_variable>
#include <mutex>
#include <queue>
#include <thread>
//Log queue class for asynchronous log writing
template <typename T>
class LockQueue
{
public:
//Multiple worker threads will write log queue s
void Push(const T &data)
{
std::lock_guard<std::mutex> lock(m_mutex); //Obtain the lock and release the lock automatically at the end of the function
m_queue.push(data); //Push data into queue
m_condvariable.notify_one(); //Wake up a thread
}
//A thread reads the log queue and writes the log file
T Pop()
{
std::unique_lock<std::mutex> lock(m_mutex); //Obtain the lock and release the lock automatically at the end of the function
while (m_queue.empty()) //Queue is empty
{
m_condvariable.wait(lock); //Thread hang, block, release lock
}
T data = m_queue.front(); //Get queue header
m_queue.pop(); //Pop up queue header element
return data;
}
private:
std::queue<T> m_queue; //queue
std::mutex m_mutex; //mutex
std::condition_variable m_condvariable; //Conditional variable
};
logger.h
#pragma
#include "lockqueue.h"
#include <string>
//Define macro function LOG_INFO("xxx %d %s", 20, "xxxx")
//Use gcc variable parameters_ VA_ARGS__， Provide users with easier use of logger
//Snprintf (buffer, buffer length, written format string, ###u VA_ARGS_)
//The parameter list representing the variable parameter is filled into the buffer, and then logger.Log(c)
#define LOG_INFO(logmsgformat, ...) \
do \
{ \
Logger &logger = Logger::GetInstance(); \
logger.SetLogLevel(INFO); \
char c[1024] = {0}; \
snprintf(c, 1024, logmsgformat, ##__VA_ARGS__); \
logger.Log(c); \
} while (0)
#define LOG_ERR(logmsgformat, ...) \
do \
{ \
Logger &logger = Logger::GetInstance(); \
logger.SetLogLevel(ERROR); \
char c[1024] = {0}; \
snprintf(c, 1024, logmsgformat, ##__VA_ARGS__); \
logger.Log(c); \
} while (0)
//Define log level
enum LogLevel
{
INFO, //General information
ERROR, //error message
};
//The log class provided by Mprpc framework is implemented in singleton mode
class Logger
{
public:
//Initialize and get a single instance of the log
static Logger &GetInstance();
//Set log level
void SetLogLevel(LogLevel level);
//Write log
void Log(std::string msg);
private:
//Logging level
int m_loglevel;
//Log buffer queue
LockQueue<std::string> m_lckque;
//Constructor to create a background process dedicated to writing logs when constructing the Logger class
Logger();
Logger(const Logger &) = delete;
Logger(Logger &&) = delete;
};
logger.cc
#include "logger.h"
#include <iostream>
#include <time.h>
//Get single instance of log
Logger &Logger::GetInstance()
{
static Logger logger;
return logger;
}
//Constructor to create a background process dedicated to writing logs when constructing the Logger class
Logger::Logger()
{
std::thread WriteLogTask([&]()
{
for (;;)
{
//Get current date
time_t now = time(nullptr);
tm *nowtm = localtime(&now);
//Build the file name and write the log information of the current day in the file
char file_name[128];
sprintf(file_name, "%d-%d-%d-log.txt",
nowtm->tm_year + 1900,
nowtm->tm_mon + 1,
nowtm->tm_mday);
//Open the log file as an append
FILE *pf = fopen(file_name, "a+");
if (pf == nullptr)
{
std::cout << "logger file: " << file_name << " open error! " << std::endl;
exit(EXIT_FAILURE);
}
//Out of the queue, get the information in the log queue header
std::string msg = m_lckque.Pop();
//Insert the time information into the information in the log queue header
char time_buf[128] = {0};
sprintf(time_buf, "%d:%d:%d =>[%s]",
nowtm->tm_hour,
nowtm->tm_min,
nowtm->tm_sec,
(m_loglevel == INFO ? "info" : "error"));
msg.insert(0, time_buf);
msg.append("\n"); //Add a line break
//Writes the information in the log queue header to the log file
fputs(msg.c_str(), pf);
fclose(pf);
}
});
//Set this thread as a separate thread, which is equivalent to a daemon thread and runs the log writing operation in the background
WriteLogTask.detach();
}
//Set log level
void Logger::SetLogLevel(LogLevel level)
{
m_loglevel = level;
}
//Write log
void Logger::Log(std::string msg)
{
m_lckque.Push(msg);
}