Article directory:

1. Gesture data collection

2. PaddleX training model

3. Test gesture recognition model

4. Test game gesture control

5. Success~

# Unzip code
!unzip /home/aistudio/data/data41298/code.zip -d /home/aistudio/work/

!pip install paddlex

Fist means go down:

Palms indicate walking up:

The following two are left and right:

Blank means fixed bit by bit:

# Set work path
import os
os.chdir('/home/aistudio/work/Pacman-master/')

1. Gesture data collection:

This step requires running the collection folder locally PalmTracker.py Hand gesture data collection is carried out for files;

When running the program, the camera will be turned on, make gestures in the designated area, and press s to save;

# !python collect/PalmTracker.py

collect    data     game.py  pacman.py	test.jpg  utils.py
config.py  demo.py  images   src	tools	  weights

2. PaddleX training model

This step uses ResNet18 provided by PaddleX for training;

The pre training model uses the weight of training on 'IMAGENET', and PaddleX selects the parameter pretrain_weights = IMAGENET;

I have collected about 40 pieces of each gesture here, and the accuracy of the training results is more than 93%;

2.1 defining data sets

from paddlex.cls import transforms
import os
import cv2
import numpy as np
import paddlex as pdx

base = './data'

with open(os.path.join('train_list.txt'), 'w') as f:
    for i, cls_fold in enumerate(os.listdir(base)):
        cls_base = os.path.join(base, cls_fold)
        files = os.listdir(cls_base)
        print('{} train num:'.format(cls_fold), len(files))
        for pt in files:
            img = os.path.join(cls_fold, pt)
            info = img + ' ' + str(i) + '\n'
            f.write(info)

with open(os.path.join('labels.txt'), 'w') as f:
    for i, cls_fold in enumerate(os.listdir(base)):
        f.write(cls_fold+'\n')

train_transforms = transforms.Compose([
    transforms.RandomCrop(crop_size=224),
    transforms.Normalize()
])

train_dataset = pdx.datasets.ImageNet(
    data_dir=base,
    file_list='train_list.txt',
    label_list='labels.txt',
    transforms=train_transforms,
    shuffle=True)

2.2 using ResNet18 training model

20 epoch s are trained here, and the initial learning rate is 2e-2

num_classes = len(train_dataset.labels)
model = pdx.cls.ResNet18(num_classes=num_classes)
model.train(num_epochs=20,
            train_dataset=train_dataset,
            train_batch_size=32,
            lr_decay_epochs=[5, 10, 15],
            learning_rate=2e-2,
            save_dir='w',
            log_interval_steps=5,
            save_interval_epochs=4)

3 test gesture recognition model:

from paddlex.cls import transforms
import matplotlib.pyplot as plt
import paddlex
import cv2
import warnings

warnings.filterwarnings('ignore')

train_transforms = transforms.Compose([
    transforms.RandomCrop(crop_size=224),
    transforms.Normalize()
])

model = paddlex.load_model('weights/final')
im = cv2.imread('test.jpg')
result = model.predict(im, topk=1, transforms=train_transforms)
print("Predict Result:", result)

%matplotlib inline
plt.imshow(im)
("Predict Result:", result)

%matplotlib inline
plt.imshow(im)
plt.show()

2020-06-23 09:27:29 [INFO]	Model[ResNet18] loaded.
Predict Result: [{'category_id': 1, 'category': 'left', 'score': 0.9999609}]

[failed to transfer the pictures in the external link. The source station may have anti-theft chain mechanism. It is recommended to save the pictures and upload them directly (img-xDGN7LG1-1592877702883)(output_13_1.png)]

4. Test gesture control in the game:

Local operation demo.py That's right;

!python demo.py

5. Success

Then embed the control into the game~

Game code from: https://github.com/hbokmann/Pacman

!python game.py

I put the demo video on Youtube

Link address: https://youtu.be/tlZT2WeaK1U

Update, station B approved!

Link address: https://www.bilibili.com/video/BV1xa4y1Y7Mb/

About the author:

contact us:

If you need weight files or source codes, and alchemists who share the same interests with AI, please privately stamp the author~

Please note AI fans when contacting me, I will pull you into the communication group~

The group will regularly share various source codes and models. Please download the previously shared ones from the group file~