Reference blogs / articles:
Detailed explanation of Apache Thrift series (I) - overview and introduction
acwing Linux basic course (mainly refer to the tutorial here)
You'll know what RPC is after reading it
thrift official website
c + + parallel programming
What is Thrift
Thrift is a lightweight, cross language remote service invocation framework. It generates RPC server / client template code of various mainstream languages through its own IDL intermediate language and with the help of code generation engine.
RPC (Remote Procedure Call) is a protocol that requests services from remote computers through the network without understanding the underlying network technology.
In short, it is convenient for you to call the program on the remote server
Learning Thrift framework through game matching cases
The game matching service is realized with the help of the server provided by acwing
Task list:
- Describe the interface
- Implement the game node (written in Python, match_client side)
- Realize matching system nodes (written in C + +, match_server side, save_client port)
match matching service and save data saving service are the client and server of the two services
acwing in the data storage center (save_server port) has been implemented with the help of the previous myserver server
Step.1 create corresponding project files
Create a project on gitlab, and then create three file directories under the project folder
match_system: match system
game: game
thrift: interface description
Step.2 create an interface description file
Two interface files
match.thrift:
Provide add_user interface and remove_user interface for matching clients
save.thrift:
Provide a save_data interface for the matching server to send data to the myserver server (data storage server) and save it
Enter thrift directory
Enter the following into match.thrift
namespace cpp match_service struct User{ 1:i32 id, 2:string name, 3:i32 score } service Match{ i32 add_user(1:User user,2:string info), i32 remove_user(1:User user,2:string info), }
The first line of namespace describes the language used. We choose c + +, so the syntax of thrift file is basically the same as that of c + +. There are several differences: int should be written as i32, and variable parameters should be labeled before
We write a structure User and describe the two interfaces add_user and remove_user as parameters
Enter the following into save.thrift
namespace cpp save_service service Save{ #username:myserver Name: the server used for data saving #Password: the md5sum first 8 digits of the password of Myserver #1 will be returned if the verification is successful, otherwise 0 will be returned i32 save_data(1:string username,2:string password,3:i32 player1_id,4:i32 player2_id) }
username and password are used for authentication
md5sum processing server password is used to facilitate code disclosure and ensure security
Step.3 Create matching client files
We use python to write the add_user and remove_user functions of the game matching client
Task:
- We should judge whether to add or remove according to the input, and convey relevant user parameters.
- We need to use the framework to make the client call the server program
Use the following command to generate a frame under the current folder
thrift -r --gen py tutorial.thrift
tutorial.thrift should be changed to its own match.thrift position
After Gen py is generated, rename it, and then find the PY file of the client. I need to modify it (header file and related functions)
Because we are writing to the client, we should pay attention to deleting the files related to the server (python can not be deleted, but the following c + + can not. See the thrift official website for details)
client.py file example
from match_client.match import Match from match_client.match.ttypes import User from thrift import Thrift from thrift.transport import TSocket from thrift.transport import TTransport from thrift.protocol import TBinaryProtocol from sys import stdin def operate(op,user_id,username,score): # Make socket transport = TSocket.TSocket('localhost', 9090) # Buffering is critical. Raw sockets are very slow transport = TTransport.TBufferedTransport(transport) # Wrap in a protocol protocol = TBinaryProtocol.TBinaryProtocol(transport) # Create a client to use the protocol encoder client = Match.Client(protocol) # Connect! transport.open() user=User(user_id,username,score) if op=="add": client.add_user(user,"") elif op=="remove": client.remove_user(user,"") # Close! transport.close() def main(): for line in stdin: op,user_id,username,score=line.split(" ") operate(op,int(user_id),username,int(score)) if __name__=="__main__": main()
Step.4 create matching server files
We use c + + to write the game matching server
Task:
- Handle add_user and remove_user calls
- Match for players
Similarly, use similar operations to generate the cpp framework of thrift and modify its contents (header file and function implementation)
thrift -r --gen cpp tutorial.thriftPart.2 handling interface calls and matching
We found that the processing needs to process two characters at the same time: interface call and player matching. We can consider that there is a parallel requirement here.
Next, we need to understand several concepts
We create a Pool class to describe a matching Pool
Take the user as mathematics and define methods such as save_result, add_user, remove_user and match
Concurrent design pattern: producer consumer
We regard the two tasks as producers and consumers respectively
Producer: interface calls for add_user and remove_user
Consumer: matching work
We add the interface call to a Message_Queue
And match according to the message queue
Because the two tasks are carried out at the same time, parallel programming is needed.
Multithreading is realized by calling thread related library, and then mutex (lock) is used to solve the problem of resource contention in parallel programming.
Since I don't know much about parallel programming, I don't talk about it here
At the beginning, we wrote the most original matching method: the matching pool matches when there are more than two players. However, the matching mechanism of the games we usually play is not so easy. We can consider adding the parameter of player waiting time and changing the matching range according to the waiting time to achieve more realistic matching.
Part.4 single threaded framework To multi-threaded frameworkThe initial framework for instruction generation is a single threaded framework. We refer to the case on the official website and modify it to a multi-threaded framework to accelerate the matching service at one time.
The following is the finished product code matching the server
// This autogenerated skeleton file illustrates how to build a server. // You should copy it to another filename to avoid overwriting it. #include "match_server/Match.h" #include "save_client/Save.h" #include <thrift/protocol/TBinaryProtocol.h> #include <thrift/server/TSimpleServer.h> #include <thrift/transport/TServerSocket.h> #include <thrift/transport/TBufferTransports.h> #include <thrift/transport/TTransportUtils.h> #include <thrift/transport/TSocket.h> #include <thrift/concurrency/ThreadManager.h> #include <thrift/concurrency/ThreadFactory.h> #include <thrift/server/TThreadedServer.h> #include <thrift/TToString.h> #include<iostream> #include<thread> #include<mutex> #include<condition_variable> #include<queue> #include<vector> #include<unistd.h> using namespace ::apache::thrift; using namespace ::apache::thrift::protocol; using namespace ::apache::thrift::transport; using namespace ::apache::thrift::server; using namespace ::match_service; using namespace ::save_service; using namespace std; struct Task{ User user; string type; }; struct MessageQueue{ queue<Task> q; mutex m; condition_variable cv; }message_queue; class Pool{ public: void save_result(int a,int b){ printf("Match Result:%d vs %d !!!\n",a,b); std::shared_ptr<TTransport> socket(new TSocket("123.57.47.211", 9090)); std::shared_ptr<TTransport> transport(new TBufferedTransport(socket)); std::shared_ptr<TProtocol> protocol(new TBinaryProtocol(transport)); SaveClient client(protocol); try { transport->open(); client.save_data("acs_152","1c915010",a,b); transport->close(); } catch (TException& tx) { cout << "ERROR: " << tx.what() << endl; } } bool check_match(uint32_t i, uint32_t j) { auto a = users[i], b = users[j]; int dt = abs(a.score - b.score); int a_max_dif = wt[i] * 50; int b_max_dif = wt[j] * 50; return dt <= a_max_dif && dt <= b_max_dif; } void match(){ for(uint32_t i = 0 ; i < wt.size() ; i ++ ){ wt[i]++; } while(users.size()>1){ sort(users.begin(),users.end(),[&](User& a,User b){ return a.score<b.score; }); bool flag=true; for(uint32_t i = 0; i < users.size(); i ++ ){ for(uint32_t j = i+1; j < users.size() ; j ++ ){ if(check_match(i,j)){ auto a=users[i],b=users[j]; users.erase(users.begin()+j); users.erase(users.begin()+i); wt.erase(wt.begin()+j); wt.erase(wt.begin()+i); save_result(a.id,b.id); flag=false; break; } } if(flag)break; } if(flag)break; } } void add(User user){ users.push_back(user); wt.push_back(0); } void remove(User user){ for (uint32_t i=0;i<users.size();i++) if(users[i].id==user.id){ users.erase(users.begin()+i); wt.erase(wt.begin()+i); break; } } private: vector<User>users; vector<int> wt;//Waiting time in seconds }pool; class MatchHandler : virtual public MatchIf { public: MatchHandler() { // Your initialization goes here } int32_t add_user(const User& user, const std::string& info) { // Your implementation goes here printf("add_user\n"); unique_lock<mutex>lck(message_queue.m); message_queue.q.push(); message_queue.cv.notify_all(); return 0; } int32_t remove_user(const User& user, const std::string& info) { // Your implementation goes here printf("remove_user\n"); unique_lock<mutex>lck(message_queue.m); message_queue.q.push(); message_queue.cv.notify_all(); return 0; } }; class MatchCloneFactory : virtual public MatchIfFactory { public: ~MatchCloneFactory() override = default; MatchIf* getHandler(const ::apache::thrift::TConnectionInfo& connInfo) override { std::shared_ptr<TSocket> sock = std::dynamic_pointer_cast<TSocket>(connInfo.transport); /*cout << "Incoming connection\n"; cout << "\tSocketInfo: " << sock->getSocketInfo() << "\n"; cout << "\tPeerHost: " << sock->getPeerHost() << "\n"; cout << "\tPeerAddress: " << sock->getPeerAddress() << "\n"; cout << "\tPeerPort: " << sock->getPeerPort() << "\n";*/ return new MatchHandler; } void releaseHandler(MatchIf* handler) override { delete handler; } }; void consume_task(){ while(true){ unique_lock<mutex>lck(message_queue.m); if(message_queue.q.empty()){ //message_queue.cv.wait(lck); //continue; lck.unlock(); pool.match(); sleep(1); } else{ auto task = message_queue.q.front(); message_queue.q.pop(); lck.unlock(); //do task if(task.type=="add")pool.add(task.user); else if(task.type=="remove")pool.remove(task.user); } } } int main(int argc, char **argv) { TThreadedServer server( std::make_shared<MatchProcessorFactory>(std::make_shared<MatchCloneFactory>()), std::make_shared<TServerSocket>(9090), //port std::make_shared<TBufferedTransportFactory>(), std::make_shared<TBinaryProtocolFactory>() ); cout<<"Start Match Server"<<endl; thread matching_thread(consume_task); server.serve(); return 0; }
Step.5 create data save client file
Since acwing on the data storage server has been implemented, you only need to generate the client as needed
Create a save_client in the match_system/src directory
thrift -r --gen cpp tutorial.thrift
Here, tutorial.thrift is written to the save.thrift location under the thrift folder
Delete the cpp file on the server side at the same time (because c + + can't have two main files at the same time, it may be ok if Python doesn't delete them)
Test whether the matching result data can be saved to the myserver server. Of course, before that, you should write the save_result method in the matching server file
harvest
The files of the project are placed on gitlab. You can have a look if you are interested, Project address
Through this study, many knowledge points have been added. Later, I will write some blogs to summarize them
- RPC framework taking thrift as an example
- c + + parallel programming
- Basic concepts of threaded process
- Proficient in git operation and related specifications
- c + + conditional variables, basic use of classes
- Concept of lock