When the dead wood is in spring, it is not luxuriant,
Young and cherish the people around the mirror.
Write in front
Today's exam was fairly smooth, but today's me, my heart is inexplicably uncomfortable, uncomfortable? If you feel uncomfortable, please write a blog. 3A requires the implementation of distributed kv server based on Raft
Implementation process
requirement:
- Single client, serial
- Multiple clients can initiate requests concurrently
- Once a client reads a new value, other clients should also read the value
Because of their own LAB3 code reference This brother This article is only used for personal review + learning. You can directly jump to the link to learn.
type Clerk struct { mu sync.Mutex servers []*labrpc.ClientEnd // You will have to modify this struct. // You only need to ensure that the clientId is not repeated every time you restart, so clientId + seqd is safe clientId int64 // Client unique ID seqId int64 // The client monotonically incremented request id leaderId int }
Seqd reflects the idempotency of a write request. If a write request has actually been written, but it dies when it returns, using this monotonically increasing request Id can avoid the problem of repeated writing.
func MakeClerk(servers []*labrpc.ClientEnd) *Clerk { ck := new(Clerk) ck.servers = servers // You'll have to add code here. ck.clientId = nrand() return ck }
This function is used to generate the client
// Put or Append type PutAppendArgs struct { Key string Value string Op string // "Put" or "Append" // You'll have to add definitions here. // Field names must start with capital letters, // otherwise RPC will break. ClientId int64 SeqId int64 } type PutAppendReply struct { Err Err }
For the PRC parameter of PutAppend, you need to take ClientId int64 and seqd Int64.
func (ck *Clerk) PutAppend(key string, value string, op string) { // You will have to modify this function. args := PutAppendArgs{ Key: key, Value: value, Op: op, ClientId: ck.clientId, SeqId: atomic.AddInt64(&ck.seqId, 1), } DPrintf("Client[%d] PutAppend, Key=%s Value=%s", ck.clientId, key, value) leaderId := ck.currentLeader() for { reply := PutAppendReply{} if ck.servers[leaderId].Call("KVServer.PutAppend", &args, &reply) { if reply.Err == OK { // success break } } leaderId = ck.changeLeader() time.Sleep(1 * time.Millisecond) } }
This function is the RPC initiator function. If an error is returned, it indicates that the leader has changed during the operation and needs to switch the leader to resend.
func (ck *Clerk) currentLeader() (leaderId int) { ck.mu.Lock() defer ck.mu.Unlock() leaderId = ck.leaderId return } func (ck *Clerk) changeLeader() (leaderId int) { ck.mu.Lock() defer ck.mu.Unlock() ck.leaderId = (ck.leaderId + 1) % len(ck.servers) return ck.leaderId }
type GetArgs struct { Key string // You'll have to add definitions here. ClientId int64 SeqId int64 } type GetReply struct { Err Err Value string } Request and response.
Get RPC call parameters and responses
func (ck *Clerk) Get(key string) string { // You will have to modify this function. args := GetArgs{ Key: key, ClientId: ck.clientId, SeqId: atomic.AddInt64(&ck.seqId, 1), } DPrintf("Client[%d] Get starts, Key=%s ", ck.clientId, key) leaderId := ck.currentLeader() for { reply := GetReply{} if ck.servers[leaderId].Call("KVServer.Get", &args, &reply) { if reply.Err == OK { // Hit return reply.Value } else if reply.Err == ErrNoKey { // non-existent return ""; } } leaderId = ck.changeLeader() time.Sleep(1 * time.Millisecond) } }
The processing of get is roughly the same as that of putappend. It should be noted that there may be two get errors, one is caused by the change of the leader, and the other is caused by the absence of the sent key value.
const ( OK = "OK" ErrNoKey = "ErrNoKey" ErrWrongLeader = "ErrWrongLeader" ) type Err string // Put or Append type PutAppendArgs struct { Key string Value string Op string // "Put" or "Append" // You'll have to add definitions here. // Field names must start with capital letters, // otherwise RPC will break. ClientId int64 SeqId int64 } type PutAppendReply struct { Err Err } type GetArgs struct { Key string // You'll have to add definitions here. ClientId int64 SeqId int64 } type GetReply struct { Err Err Value string }
Here is the structure to be used in lab3, which is defined in common.go
const ( OP_TYPE_PUT = "Put" OP_TYPE_APPEND = "Append" OP_TYPE_GET = "Get" ) type Op struct { // Your definitions here. // Field names must start with capital letters, // otherwise RPC will break. Index int Term int Type string Key string Value string SeqId int64 ClientId int64 } type OpContext struct { op *Op committed chan byte wrongLeader bool ignore bool keyExist bool value string } type KVServer struct { mu sync.Mutex me int rf *raft.Raft applyCh chan raft.ApplyMsg dead int32 // set by Kill() maxraftstate int // snapshot if log grows this big kvStore map[string]string // kv storage reqMap map[int]*OpContext // Log index - > request context seqMap map[int64]int64 // Client ID - > client seq // Your definitions here. lastAppliedIndex int }
Here, the first op operation and kvsever structure on the server side are given. reqMap stores the RPC calls in progress, and seqMap records the maximum request ID submitted by each clientId for write idempotency determination.
func StartKVServer(servers []*labrpc.ClientEnd, me int, persister *raft.Persister, maxraftstate int) *KVServer { // call labgob.Register on structures you want // Go's RPC library to marshall/unmarshall. labgob.Register(&Op{}) kv := new(KVServer) kv.me = me kv.maxraftstate = maxraftstate // You may need initialization code here. kv.applyCh = make(chan raft.ApplyMsg) kv.rf = raft.Make(servers, me, persister, kv.applyCh) // You may need initialization code here. kv.kvStore = make(map[string]string) kv.reqMap = make(map[int]*OpContext) kv.seqMap = make(map[int64]int64) go kv.applyLoop() return kv }
This function is responsible for starting the service. Here, the MAKE function of raft is called to initialize the raft cluster and start kvsever
func Make(peers []*labrpc.ClientEnd, me int, persister *Persister, applyCh chan ApplyMsg) *Raft { rf := &Raft{} rf.peers = peers rf.persister = persister rf.me = me // Your initialization code here (2A, 2B, 2C). rf.role = ROLE_FOLLOWER rf.leaderId = -1 rf.votedFor = -1 rf.lastActiveTime = time.Now() rf.lastIncludedIndex = 0 rf.lastIncludedTerm = 0 rf.applyCh = applyCh // rf.nextIndex = make([]int, len(rf.peers)) // rf.matchIndex = make([]int, len(rf.peers)) // initialize from state persisted before a crash rf.readPersist(persister.ReadRaftState()) //rf.installSnapshotToApplication() DPrintf("RaftNode[%d] Make again", rf.me) // start ticker goroutine to start elections go rf.electionLoop() go rf.appendEntriesLoop() go rf.applyLogLoop() //go rf.ticker() DPrintf("Raftnode[%d]start-up", me) return rf }
func (kv *KVServer) PutAppend(args *PutAppendArgs, reply *PutAppendReply) { // Your code here. reply.Err = OK op := &Op{ Type: args.Op, Key: args.Key, Value: args.Value, ClientId: args.ClientId, SeqId: args.SeqId, } // Write raft layer var isLeader bool op.Index, op.Term, isLeader = kv.rf.Start(op) if !isLeader { reply.Err = ErrWrongLeader return } opCtx := newOpContext(op) func() { kv.mu.Lock() defer kv.mu.Unlock() // Save the RPC context and wait for the callback to be submitted. The same Index may be overwritten because of the Leader change. However, the previous RPC will timeout and exit and make the client retry kv.reqMap[op.Index] = opCtx }() // Clean up context before RPC end defer func() { kv.mu.Lock() defer kv.mu.Unlock() if one, ok := kv.reqMap[op.Index]; ok { if one == opCtx { delete(kv.reqMap, op.Index) } } }() timer := time.NewTimer(2000 * time.Millisecond) defer timer.Stop() select { case <- opCtx.committed: // If submitted if opCtx.wrongLeader { // Similarly, the term in the index position is different, indicating that the leader has changed and the client needs to write to the new leader again reply.Err = ErrWrongLeader } else if opCtx.ignored { // Note the req id has expired and the request is ignored. For MIT lab, you only need to tell the client OK to skip } case <- timer.C: // If the submission fails for 2 seconds, ask the client to retry reply.Err = ErrWrongLeader } }
Here, the start function in raft is called. Here, the code is compared. The putappend logic is to write the op to raft, wait for applyloop to send the submission model, and then return the rpc call (ignore is the embodiment of idempotency)
func (rf *Raft) Start(command interface{}) (int, int, bool) { index := -1 term := -1 isLeader := true // Your code here (2B). rf.mu.Lock() defer rf.mu.Unlock() if rf.role != ROLE_LEADER { return -1, -1, false } logEntry := LogEntry{ Command: command, Term: rf.currentTerm, } rf.log = append(rf.log, logEntry) index = rf.lastIndex() term = rf.currentTerm rf.persist() DPrintf("RaftNode[%d] Add Command, logIndex[%d] currentTerm[%d]", rf.me, index, term) return index, term, isLeader }
Obviously, the server side writes the operation to the log
func newOpContext(op *Op) (opCtx *OpContext) { opCtx = &OpContext{ op: op, committed: make(chan byte), } return }
func (kv *KVServer) Get(args *GetArgs, reply *GetReply) { // Your code here. reply.Err = OK op := &Op{ Type: OP_TYPE_GET, Key: args.Key, ClientId: args.ClientId, SeqId: args.SeqId, } // Write raft layer var isLeader bool op.Index, op.Term, isLeader = kv.rf.Start(op) if !isLeader { reply.Err = ErrWrongLeader return } opCtx := newOpContext(op) func() { kv.mu.Lock() defer kv.mu.Unlock() // Save the RPC context and wait for the callback to be submitted. The same Index may be overwritten because of the Leader change. However, the previous RPC will timeout and exit and make the client retry kv.reqMap[op.Index] = opCtx }() // Clean up context before RPC end defer func() { kv.mu.Lock() defer kv.mu.Unlock() if one, ok := kv.reqMap[op.Index]; ok { if one == opCtx { delete(kv.reqMap, op.Index) } } }() timer := time.NewTimer(2000 * time.Millisecond) defer timer.Stop() select { case <-opCtx.committed: // If submitted if opCtx.wrongLeader { // Similarly, the term in the index position is different, indicating that the leader has changed and the client needs to write to the new leader again reply.Err = ErrWrongLeader } else if !opCtx.keyExist { // key does not exist reply.Err = ErrNoKey } else { reply.Value = opCtx.value // Return value } case <- timer.C: // If the submission fails for 2 seconds, ask the client to retry reply.Err = ErrWrongLeader } }
Get requests are as like as two peas in putappend, but the error handling state is somewhat different, and roughly logical thinking is exactly the same as putappend.
func (kv *KVServer) applyLoop() { for !kv.killed() { select { case msg := <- kv.applyCh: cmd := msg.Command index := msg.CommandIndex func() { kv.mu.Lock() defer kv.mu.Unlock() // Operation log op := cmd.(*Op) opCtx, existOp := kv.reqMap[index] prevSeq, existSeq := kv.seqMap[op.ClientId] kv.seqMap[op.ClientId] = op.SeqId if existOp { // If there are RPC s waiting for results, judge whether the status is consistent with that at the time of writing if opCtx.op.Term != op.Term { opCtx.wrongLeader = true } } // Only client write requests with monotonically increasing ID are processed if op.Type == OP_TYPE_PUT || op.Type == OP_TYPE_APPEND { if !existSeq || op.SeqId > prevSeq { // If it is an incremental request ID, accept its change if op.Type == OP_TYPE_PUT { // put operation kv.kvStore[op.Key] = op.Value } else if op.Type == OP_TYPE_APPEND { // Put append operation if val, exist := kv.kvStore[op.Key]; exist { kv.kvStore[op.Key] = val + op.Value } else { kv.kvStore[op.Key] = op.Value } } } else if existOp { opCtx.ignored = true } } else { // OP_TYPE_GET if existOp { opCtx.value, opCtx.keyExist = kv.kvStore[op.Key] } } DPrintf("RaftNode[%d] applyLoop, kvStore[%v]", kv.me, kv.kvStore) // Wake up pending RPC if existOp { close(opCtx.committed) } }() } } }
The logic of applyloop is to listen to the kv.applyCh pipeline. Once there is an embodied log. SEQID processing here is very wonderful. SEQID is added to the client atom, and then it is cleverly incremental. The server does not care about expired requests, and finally wakes up the blocked committed pipeline, that is, the putappend and get handle functions. Let them judge, and then pRC returns. marvellous!!!
Write it at the back
3A the overall difficulty is not very great, but there has been an item timeout in test. I really don't know what the problem is. Let's put this problem aside for the time being
- Linear consistency requires that the client must initiate OP serially, and the server cannot guarantee the effectiveness of the first op. in actual engineering, I think the client should queue the request number seqd and submit it serially to KV server.
- As long as the ID of each client is guaranteed to be unique, the seqd does not need to be persisted. It is relatively simple to ensure the uniqueness of the client ID, such as ip+pid+timestamp.
- Write idempotency determines whether the current OP can be executed by comparing the seqd of the last submitted log of clientId.
- The read consistency is achieved by writing a read log to raft. When the read log is submitted, the data of kvStore at this moment is returned to RPC. Then the data seen by RPC must comply with linear consistency, that is, subsequent read operations can continue to read this value.
- The RPC server needs to do timeout processing, because it is likely that the master selection occurs after the leader writes to the local log, then the new leader will truncate the log written by the old leader, resulting in the continuous failure to submit the corresponding index position. Therefore, the client should write to the log again through timeout to achieve submission and obtain the final result.