[reinforcement learning] example 1 of implementing q-learning in python

Author: hhh5460

Address: https://www.cnblogs.com/hhh5460/p/10134018.html

Problem situation

-o---T
#T is the location of the treasure, o is the location of the explorer

This time, we will use the q-learning method to implement a small example. The environment of the example is a one-dimensional world, and there are treasures on the right side of the world. As long as the Explorer gets the treasure and tastes the sweetness, he will remember the way to get the treasure. This is the behavior he learned with reinforcement learning.

Q-learning is a method of recording behavior value (Q value). Every behavior in a certain state has a value Q(s, a), that is, the value of behavior a in s state is Q(s, a). S, in the Explorer game above, is where o is. And each location Explorer can do two actions left/right, which is all the explorers can do.

Thanks: the above three paragraphs are from here: https://morvanzhou.github.io/tutorials/machine-learning/reinforcement-learning/2-1-general-rl/

 

To solve this problem, the following things should be clarified first:

0. Related parameters

epsilon = 0.9   # greediness  greedy
alpha = 0.1     # Learning rate
gamma = 0.8     # Diminishing reward value

 

1. State set

The seeker's state, that is, where it can reach, has six. So definition

states = range(6) # State set, from 0 to 5

So, how to determine the next state after performing an action in a certain state?

def get_next_state(state, action):
    '''After performing actions on the state, the next state is obtained'''
    global states
    
    # left, right = -1,+1 # Generally speaking, this is the case, but the two positions should be considered
    if action == 'right' and state != states[-1]: # Except for the last state (position), it can be right(+1)
        next_state = state + 1
    elif action == 'left' and state != states[0]: # Except for the first state (position), it can be left(-1)
        next_state = state -1
    else:
        next_state = state
    return next_state

 

2. Action set

When a seeker is in each state, there are only two possible actions: "left" or "right". So definition

actions = ['left', 'right'] # Action set. You can also add actions'none',It means staying

So, in a given state (position), how to determine all legal actions?

def get_valid_actions(state):
    '''Take the legal action set in the current state, and rewards irrelevant!'''
    global actions # ['left', 'right']
    
    valid_actions = set(actions)
    if state == states[-1]:             # Last state (position), then
        valid_actions -= set(['right']) # Remove the action to the right
    if state == states[0]:              # The most previous state (position), then
        valid_actions -= set(['left'])  # Remove left
    return list(valid_actions)
    

 

3. Reward collection

When a seeker reaches each state (position), there should be a reward. So definition

rewards = [0,0,0,0,0,1] # Reward collection. Only the location of the last treasure has a reward of 1, and all others are 0

Obviously, getting the reward under state state is simple: rewards[state]. According to the state, you can follow the diagram without defining an additional function.

 

4.Q table

Most important. Q table is a table that records state behavior values. The common q-table s are two-dimensional, and the basic length is as follows:

 (note that there are also three-dimensional Q table s)

So definition

q_table = pd.DataFrame(data=[[0 for _ in actions] for _ in states],
                       index=states, columns=actions)

 

5. Environment and its renewal

The purpose of thinking about the environment is to allow people to observe the explorers' exploration process through the screen, that's all.

The environment is very simple, just a string of characters' --- T '! When the seeker reaches the state (position), replace the character in the position with 'o', and finally reprint the whole string! therefore

def update_env(state):
    '''Update environment and print'''
    global states
    
    env = list('-----T')
    if state != states[-1]:
        env[state] = 'o'
    print('\r{}'.format(''.join(env)), end='')
    time.sleep(0.1)

 

6. Finally, Q-learning algorithm

Pseudo code of Q-learning algorithm

Pseudo code of Chinese version:

Source: https://www.hhyz.me/2018/08/05/2018-08-05-RL/

Q value is updated according to Behrman equation

$$Q(s_t,a_t) \leftarrow Q(s_t,a_t) + \alpha[r_{t+1} + \lambda \max _{a} Q(s_{t+1}, a) - Q(s_t,a_t)] \tag {1}$$

Well, it's time to do it:

# A total of 13 explorations were made
for i in range(13):
    # 0.Start from the leftmost position (not necessary)
    current_state = 0
    #current_state = random.choice(states) # Also random
    while current_state != states[-1]:
        # 1.Choose one of the legal actions in the current state randomly (or greedily) as the current action
        if (random.uniform(0,1) > epsilon) or ((q_table.ix[current_state] == 0).all()):  # explore
            current_action = random.choice(get_valid_actions(current_state))
        else:
            current_action = q_table.ix[current_state].idxmax() # Take advantage of (greed)
        # 2.Execute the current action to get the next state (position)
        next_state = get_next_state(current_state, current_action)
        # 3.Remove all of the Q value,The maximum value is to be taken
        next_state_q_values = q_table.ix[next_state, get_valid_actions(next_state)]
        # 4.According to the Behrman equation, update Q table Current status in-Action corresponding Q value
        q_table.ix[current_state, current_action] += alpha * (rewards[next_state] + gamma * next_state_q_values.max() - q_table.ix[current_state, current_action])
        # 5.Go to the next state (position)
        current_state = next_state

print('\nq_table:')
print(q_table)

Well, this is the famous Q-learning algorithm!

Notice that in Behrman's equation, rewards are used [next]_ State] again: next_state

 

Of course, we hope to see the explorers' exploration process. We can update (print) the environment at any time

for i in range(13):
    #current_state = random.choice(states)
    current_state = 0
    
    update_env(current_state) # Environment related
    total_steps = 0           # Environment related
    
    while current_state != states[-1]:
        if (random.uniform(0,1) > epsilon) or ((q_table.ix[current_state] == 0).all()):  # explore
            current_action = random.choice(get_valid_actions(current_state))
        else:
            current_action = q_table.ix[current_state].idxmax() # Take advantage of (greed)

        next_state = get_next_state(current_state, current_action)
        next_state_q_values = q_table.ix[next_state, get_valid_actions(next_state)]
        q_table.ix[current_state, current_action] += alpha * (reward[next_state] + gamma * next_state_q_values.max() - q_table.ix[current_state, current_action])
        current_state = next_state
        
        update_env(current_state) # Environment related
        total_steps += 1          # Environment related
        
    print('\rEpisode {}: total_steps = {}'.format(i, total_steps), end='') # Environment related
    time.sleep(1)                                                          # Environment related
    print('\r                                ', end='')                    # Environment related
        
print('\nq_table:')
print(q_table)

7. Complete code

'''
-o---T
# T is the location of the treasure, o is the location of the explorer
'''
# author: hhh5460
# Time: 20181217
import pandas as pd import random import time epsilon = 0.9 # greediness greedy alpha = 0.1 # Learning rate gamma = 0.8 # Diminishing reward value states = range(6) # State set. From 0 to 5 actions = ['left', 'right'] # Action set. You can also add actions'none',It means staying rewards = [0,0,0,0,0,1] # Reward collection. Only the location of the last treasure has a reward of 1, and all others are 0 q_table = pd.DataFrame(data=[[0 for _ in actions] for _ in states], index=states, columns=actions) def update_env(state): '''Update environment and print''' global states env = list('-----T') # Environment is such a string(list)!! if state != states[-1]: env[state] = 'o' print('\r{}'.format(''.join(env)), end='') time.sleep(0.1) def get_next_state(state, action): '''After performing actions on the state, the next state is obtained''' global states # l,r,n = -1,+1,0 if action == 'right' and state != states[-1]: # Except for the last state (position), to the right+1 next_state = state + 1 elif action == 'left' and state != states[0]: # Except for the first state (position), the left is on the left-1 next_state = state -1 else: next_state = state return next_state def get_valid_actions(state): '''Take the legal action set in the current state, and reward irrelevant!''' global actions # ['left', 'right'] valid_actions = set(actions) if state == states[-1]: # Last state (position), then valid_actions -= set(['right']) # Not to the right if state == states[0]: # The most previous state (position), then valid_actions -= set(['left']) # Not to the left return list(valid_actions) for i in range(13): #current_state = random.choice(states) current_state = 0 update_env(current_state) # Environment related total_steps = 0 # Environment related while current_state != states[-1]: if (random.uniform(0,1) > epsilon) or ((q_table.ix[current_state] == 0).all()): # explore current_action = random.choice(get_valid_actions(current_state)) else: current_action = q_table.ix[current_state].idxmax() # Take advantage of (greed) next_state = get_next_state(current_state, current_action) next_state_q_values = q_table.ix[next_state, get_valid_actions(next_state)] q_table.ix[current_state, current_action] += alpha * (rewards[next_state] + gamma * next_state_q_values.max() - q_table.ix[current_state, current_action]) current_state = next_state update_env(current_state) # Environment related total_steps += 1 # Environment related print('\rEpisode {}: total_steps = {}'.format(i, total_steps), end='') # Environment related time.sleep(2) # Environment related print('\r ', end='') # Environment related print('\nq_table:') print(q_table)

 

8. Real final, renderings

Tags: github Lambda

Posted on Mon, 29 Jun 2020 22:14:31 -0400 by Hardbyte