Play with Golang's channel and implement PubSub mode in 200 lines of code

introduction

PubSub(Publish/Subscribe) mode, which means "publish / subscribe" mode, is to solve the one to many dependency and enable multiple consumers to listen to a topic at the same time, which can not only decouple producers and consumers, but also decouple different consumers from each other (Note: some antipatterns rely on the order of execution of subscribers. It is necessary to avoid using shared data to transfer status, because this will make consumers coupled and cannot change independently). The key lies in the need for an intermediary to maintain the subscription relationship and deliver the production messages to subscribers.

In the language of Golang, channel is naturally suitable to be the intermediary. Let's implement the tool class EventBus step by step according to the PubSub pattern

Define type

First, let's define some basic types and core operations.

//EventID is the unique identifier of the Event
type EventID int64

//Event
type Event interface {
    ID() EventID
}

//EventHandler
type EventHandler interface {
    OnEvent(ctx context.Context, event Event) error
    CanAutoRetry(err error) bool
}

// JobStatus holds information related to a job status
type JobStatus struct {
    RunAt      time.Time
    FinishedAt time.Time
    Err        error
}

//EventBus ...
type EventBus struct {}

func (eb *EventBus) Subscribe(eventID EventID, handlers ...EventHandler) { }

func (eb *EventBus) Unsubscribe(eventID EventID, handlers ...EventHandler) { }

func (eb *EventBus) Publish(evt Event) <-chan JobStatus { }

Key disassembly

First, consumers need to Subscribe to related topics through Subscribe. The key point is to maintain the subscribed consumers according to the EventID. Naturally, we think of the map. We choose to use handlers map[EventID][]EventHandler for maintenance. Considering the concurrency problem, we also need to add a lock.

//Subscribe ...
func (eb *EventBus) Subscribe(eventID EventID, handlers ...EventHandler) {
    eb.mu.Lock()
    defer eb.mu.Unlock()

    eb.handlers[eventID] = append(eb.handlers[eventID], handlers...)
}

The implementation here is relatively simple, without considering the problem of repeated subscriptions by one consumer, which is left to the user to deal with. (however, why should the same consumer call subcribe multiple times to subscribe to the same topic? It feels like writing a bug.)

The following is the core Publish function. On the one hand, a channel (preferably with a buffer) must be required to transmit Event data. On the other hand, in order to ensure performance, some resident processes are required to listen to messages and start relevant consumers. The following is the relevant code (logs, error handling, etc. are added in the full version of the code. For the sake of showing the key points, they are omitted for the time being)

func (eb *EventBus) Start() {
    if eb.started {
        return
    }

    for i := 0; i < eb.eventWorkers; i++ {
        eb.wg.Add(1)
        go eb.eventWorker(eb.eventJobQueue)
    }

    eb.started = true
}


func (eb *EventBus) eventWorker(jobQueue <-chan EventJob) {
loop:
    for {
        select {
        case job := <-jobQueue:
            jobStatus := JobStatus{
                RunAt: time.Now(),
            }

            ctx, cancel := context.WithTimeout(context.Background(), eb.timeout)
            g, _ := errgroup.WithContext(ctx)
            for index := range job.handlers {
                handler := job.handlers[index]
                g.Go(func() error {
                    return eb.runHandler(ctx, handler, job.event)
                })
            }
            jobStatus.Err = g.Wait()

            jobStatus.FinishedAt = time.Now()

            select {
            case job.resultChan <- jobStatus:
            default:
            }
            cancel()
        }
    }
}

After making the above preparations, the following is the real Publish code.

// EventJob ...
type EventJob struct {
    event      Event
    handlers   []EventHandler
    resultChan chan JobStatus
}

//Publish ...
func (eb *EventBus) Publish(evt Event) <-chan JobStatus {
    eb.mu.RLock()
    defer eb.mu.RUnlock()
    if ehs, ok := eb.handlers[evt.ID()]; ok {
        handlers := make([]EventHandler, len(ehs))
        copy(handlers, ehs) //A snapshot of consumers at that time
        job := EventJob{
            event:      evt,
            handlers:   handlers,
            resultChan: make(chan JobStatus, 1),
        }

        var jobQueue = eb.eventJobQueue
        select {
        case jobQueue <- job:
        default:
        }

        return job.resultChan
    } else {
        err := fmt.Errorf("no handlers for event(%d)", evt.ID())
        resultChan := make(chan JobStatus, 1)
        resultChan <- JobStatus{
            Err: err,
        }
        return resultChan
    }
}

There is no way to get the relevant consumers directly from the handlers according to the ID in the eventWorker. On the one hand, it is to make the eventWorker more general, and on the other hand, it is also to reduce the blocking caused by lock operation.

So far, we have disassembled the core code one by one. For the complete code, please refer to the code in the channelx project event_bus.go

Use example

Tool classes without examples are incomplete. An example is provided below.

First define an event, where the id is defined as private, and then force it to be specified in the constructor.

const ExampleEventID channelx.EventID = 1

type ExampleEvent struct {
    id channelx.EventID
}

func NewExampleEvent() ExampleEvent {
    return ExampleEvent{id:ExampleEventID}
}

func (evt ExampleEvent) ID() channelx.EventID  {
    return evt.id
}

Next is the event handler. According to the actual needs, check whether the received event is a subscribed event in OnEvent and whether the received event structure can be converted to a specific type. After defense programming, you can process the event logic.

type ExampleHandler struct {
    logger channelx.Logger
}

func NewExampleHandler(logger channelx.Logger) *ExampleHandler {
    return &ExampleHandler{
        logger: logger,
    }
}

func (h ExampleHandler) Logger() channelx.Logger{
    return h.logger
}

func (h ExampleHandler) CanAutoRetry(err error) bool {
    return false
}

func (h ExampleHandler) OnEvent(ctx context.Context, event channelx.Event) error {
    if event.ID() != ExampleEventID {
        return fmt.Errorf("subscribe wrong event(%d)", event.ID())
    }

    _, ok := event.(ExampleEvent)
    if !ok {
        return fmt.Errorf("failed to convert received event to ExampleEvent")
    }

    // handle the event here
    h.Logger().Infof("event handled")

    return nil
}

Finally, the start of EventBus, the subscription and publication of events.

eventBus := channelx.NewEventBus(logger, "test", 4,4,2, time.Second, 5 * time.Second)
eventBus.Start()

handler := NewExampleHandler(logger)
eventBus.Subscribe(ExampleEventID, handler)
eventBus.Publish(NewExampleEvent())

Write at the end

I also wrote some articles about the use of channel before,

The lightweight util s implemented inside are open source channelx , you are welcome to review. If you have any tools you like, you are welcome to like them or star:)

Tags: Go

Posted on Tue, 21 Sep 2021 18:44:54 -0400 by jacomus