Master the X3 series, enjoy fitness and gaming at the same time.

demo video

One Prepare

1.1 Install the Boxing Emperor game on the window computer

Because it is a key simulation, in theory, as long as it is a key controlled game can be played! I’ll use the mame emulator as an example)

Download MAME:MAMEdev.org | Home of The MAME Project.

unzip

commands

mame -keyboardprovider win32 sfiii3nr1 

To view the information about the control button, press the key to display the setting screen. Select (double click or press enter) Input Settings, Input Assignments(this system), and you can find all the buttons. For example, kbd up is the up button on the keyboard. To modify a key, double-click or enter, and then type the new key.

One thing to note here is that the default keys of the game are partially different from the keys given in the code and need to be modified in advance. Modify keys according to the following mappings.

ACTION2KEY = { 
    'COIN_P1': '5', 
    'COIN_P2': '6', 
    'P1_START': '1', 
    'P2_START': '2', 
    'P1_JPUNCH': 'L-CTRL', 
    'P2_JPUNCH': 'A', 
    'P1_RIGHT': 'RIGHT', 
    'P2_RIGHT': 'G', 
    'P1_LEFT': 'L', 
    'P2_LEFT': 'D', 
    'P1_DOWN': 'DOWN', 
    'P2_DOWN': 'F', 
    'P1_UP': 'UP', 
    'P2_UP': 'R', 
    'P1_SKICK': 'L-SHIFT', 
    'P2_SKICK': 'W', 
} 

1.2 Start the signal receiving program on the Windows PC

Since the Windows mame cannot be controlled by script (ubuntu can be controlled by mametoolkit), the game can only be controlled by analog buttons. The script is as follows. The startup command is:

python3 your_python_file --ip your_ip 
 

import os 
import re 
import socket 
import argparse 
import time 
from threading import Thread 
 
try: 
    import win32api 
    import win32con 
except ModuleNotFoundError: 
    os.system('pip3 install pywin32') 
    import win32api 
    import win32con 
 
key2code = {'0': 48, '1': 49, '2': 50, '3': 51, '4': 52, '5': 53, '6': 54, '7': 55, '8': 56, '9': 57, 'a': 65, 
            'b': 66, 'c': 67, 'd': 68, 'e': 69, 'f': 70, 'g': 71, 'h': 72, 'i': 73, 'j': 74, 'k': 75, 'l': 76, 
            'm': 77, 'n': 78, 'r': 82, 'o': 79, 'p': 80, 'q': 81, 's': 83, 't': 84, 'u': 85, 'v': 86, 'w': 87, 'x': 88, 
            'y': 89, 'z': 90, '/0': 96, 
            '/1': 97, '/2': 98, '/3': 99, '/4': 100, '/5': 101, '/6': 102, '/7': 103, '/8': 104, '/9': 105, '*': 106, 
            '+': 107, 'enter': 108, '-': 109, '.': 110, '/': 111, 'f1': 112, 'f2': 113, 'f3': 114, 'f4': 115, 'f5': 116, 
            'f6': 117, 'f7': 118, 'f8': 119, 'f9': 120, 'f10': 121, 'f11': 122, 'f12': 123, 'backspace': 8, 'tab': 9, 
            'clear': 12, 'l-shift': 16, 'r-shift': 16, 'l-ctrl': 17, 'r-ctrl': 17, 'alt': 18, 'capslock': 20, 'esc': 27, 
            'space': 32, 
            'pageup': 33, 'pagedown': 34, 'end': 35, 'home': 36, 'left': 37, 'up': 38, 'right': 39, 'down': 40, 
            'insert': 45, 'delete': 46, 'help': 47, 'numlock': 144} 
key2scan = {'numlock': [0x45, 0xc5], '/': [0xe035, 0xe0b5], '*': [0x09, 0x89], '-': [0x0c, 0x8c], '7': [0x08, 0x88], 
            '8': [0x09, 0x89], '9': [0x0a, 0x8a], '4': [0x05, 0x85], '5': [0x06, 0x86], '6': [0x07, 0x87], 
            '1': [0x02, 0x82], 'end': [0xe04f, 0xe0cf], '2': [0x03, 0x83], '3': [0x04, 0x84], 
            'pgdn': [0x51, 0xd1], '0': [0x0b, 0x8b], 'ins': [0x52, 0xd2], '.': [0x34, 0xb4], 'del': [0x53, 0xd3], 
            '+': [0x0d, 0x8d], 'enter': [0x1c, 0x9c], 'insert': [0xe052, 0xe0d2], 'page up': [0xe049, 0xe0c9], 
            'delete': [0xe053, 0xe0d3], 'page down': [0xe051, 0xe0d1], 'left': [0xe046, 0xe0c6], 
            'right': [0xe04d, 0xe0cd], 'up': [0xe048, 0xe0c8], 'down': [0xe050, 0xe0d0], 
            'esc': [0x01, 0x81], 'f1': [0x3b, 0xbb], 'f2': [0x3c, 0xbc], 'f3': [0x3d, 0xbd], 
            'f4': [0x3e, 0xbe], 'f5': [0x3f, 0xbf], 'f6': [0x40, 0xc0], 'f7': [0x41, 0xc1], 'f8': [0x42, 0xc2], 
            'f9': [0x43, 0xc3], 'f10': [0x44, 0xc4], 'f11': [0x57, 0xd7], 'f12': [0x58, 0xd8], '~': [0x29, 0xa9], 
            '·': [0x29, 0xa9], '<tab>': [0x0f, 0x8f], '!': [0x02, 0x82], 'q': [0x10, 0x90], '@': [0x03, 0x83], 
            'w': [0x11, 0x91], '#': [0x04, 0x84], 'e': [0x12, 0x12], '$': [0x05, 0x85], 'r': [0x13, 0x93], 
            '%': [0x06, 0x86], 't': [0x14, 0x94], '^': [0x07, 0x87], 'y': [0x15, 0x95], '&': [0x08, 0x88], 
            'u': [0x16, 0x96], 'i': [0x17, 0x97], '(': [0x0a, 0x8a], 'o': [0x18, 0x98], ')': [0x0b, 0x8b], 
            'p': [0x19, 0x99], '_': [0x0c, 0x8c], '{': [0x1a, 0x9a], '[': [0x1a, 0x9a], '=': [0x0d, 0x8d], 
            '}': [0x1b, 0x9b], ']': [0x1b, 0x9b], '|': [0x2b, 0xab], '\\': [0x2b, 0xab], 'backspace': [0x0e, 0x8e], 
            'capslock': [0x3a, 0xba], 'l-shift': [0x2a, 0xaa], 'a': [0x1e, 0x9e], 'z': [0x2c, 0xac], 
            's': [0x1f, 0x9f], 'x': [0x2d, 0xad], 'd': [0x20, 0xa0], 'c': [0x2e, 0xae], 'f': [0x21, 0xa1], 
            'v': [0x2f, 0xaf], 'g': [0x22, 0xa2], 'b': [0x30, 0xb0], 'h': [0x23, 0xa3], 'n': [0x31, 0xb1], 
            'j': [0x24, 0xa4], 'm': [0x32, 0xb2], 'k': [0x25, 0xa5], '<': [0x33, 0xb3], ',': [0x33, 0xb3], 
            'l': [0x26, 0xa6], '>': [0x34, 0xb4], ':': [0x27, 0xa7], ';': [0x27, 0xa7], '?': [0x35, 0xb5], 
            '': [0x35, 0xb5], '"': [0x28, 0xa8], '\'': [0x28, 0xa8], 'r-shift': [0x36, 0xb6], 'l-ctrl': [0x1d, 0x9d], 
            'l-alt': [0x38, 0xb8], 'space': [0x39, 0xb9], 'r-alt': [0xe038, 0xe0b8], 'r-ctrl': [0xe01d, 0xe09d], } 
 
 
class SocketKeyBoard(object): 
    def __init__(self, ip, port=8000): 
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
        self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 
        print("socket bind " + ip + ":" + str(port)) 
        self.socket.bind((ip, port)) 
        self.socket.listen(128) 
        self.client_socket, self.client_addr = None, None 
        self.auto_thread = Thread(target=self.auto_run, args=()) 
        self.auto_thread.start() 
        self.hold_key_list = {} 
 
    def auto_run(self): 
        while True: 
            time.sleep(0.01) 
            remove_key_list = [] 
            for key in self.hold_key_list.keys(): 
                if self.hold_key_list[key] < time.time(): 
                    self.press_key(key, win32con.KEYEVENTF_KEYUP) 
                    remove_key_list.append(key) 
            for key in remove_key_list: 
                self.hold_key_list.pop(key) 
 
    @staticmethod 
    def press_key(key, code=0): 
        if code == 0: 
            print("press", key, time.time()) 
        else: 
            print("release", key, time.time()) 
        if key in key2code.keys(): 
            if key2scan[key][0] & 0xff00 == 0xe000: 
                win32api.keybd_event(key2code[key], key2scan[key][0] & 0x00ff, 
                                     win32con.KEYEVENTF_EXTENDEDKEY | code, 0) 
            else: 
                win32api.keybd_event(key2code[key], key2scan[key][0], 0 | code, 0) 
        else: 
            print("get unexpected key: ", key) 
 
    def run(self): 
        total_len = 0 
        while True: 
            if self.client_socket: 
                rev_info = self.client_socket.recv(1024) 
                if not rev_info: 
                    self.client_socket = None 
                    continue 
                key_list = re.findall("\"(.*?)\"", rev_info.decode()) 
                key_list = [key.strip().lower() for key in key_list] 
                for key in key_list: 
                    total_len += 1 
                    if 'hold' in key: 
                        key_temp = key.split(':')[0] 
                        keep_time = float(re.findall("hold(.*?)s", key.split(':')[1])[0]) 
                        self.hold_key_list[key_temp] = time.time() + keep_time 
                        key = key_temp 
                    self.press_key(key) 
                time.sleep(0.01) 
                for key in key_list: 
                    if ':hold' in key: 
                        continue 
                    if key in key2code.keys(): 
                        self.press_key(key, win32con.KEYEVENTF_KEYUP) 
            else: 
                print("wait for connecting ...") 
                self.client_socket, self.client_addr = self.socket.accept() 
                print("connect success from: ", self.client_addr) 
 
 
def parse_args(): 
    description = "start web socket keyboard application" 
    parser = argparse.ArgumentParser(description=description) 
    parser.add_argument('--ip', help="the local ip number") 
    parser.add_argument('--port', help='set listen port ', default=8000) 
    args_ = parser.parse_args() 
    return args_ 
 
 
if __name__ == "__main__": 
    args = parse_args() 
    socket_keyboard = SocketKeyBoard(ip=args.ip, port=args.port) 
    socket_keyboard.run() 

The 1.3X3 PI can run the human detection code

Because the action of the Street Fighter game is too intense, the real-time requirements for human bone detection are also very high, and the current market can be accurate, real-time, and cheap, on the horizon x3 pie, not to say much, directly engaged.

First of all, make sure you have a piece of x3 pie, as well as a camera (preferably usb, the line is a little longer, using a usb camera as a gesture recognition input tutorial see: Gesture recognition algorithm using a usb camera instead of a mipi camera).

Secondly, the Horizon Robot Development Platform software package (HHP) is installed, which is able to run “Gesture Recognition” in section 4.2: The Robot Platform User Manual - the Horizon Robot Platform User Manual 1.0 document

Finally, please download this python file to your x3 pie. Among them, the ACTION2KEY in line 11 must be consistent with the game controls in Chapter 1.1, otherwise the sent keys will not be recognized.

import rclpy 
import time 
import socket 
import math 
import json 
import argparse 
from rclpy.node import Node 
from ai_msgs.msg import PerceptionTargets 
from threading import Thread 
from collections import Counter 
 
#Action to key mapping 
ACTION2KEY = { 
    'COIN_P1': '5', 
    'COIN_P2': '6', 
    'P1_START': '1', 
    'P2_START': '2', 
    'P1_JPUNCH': 'L-CTRL', 
    'P2_JPUNCH': 'A', 
    'P1_RIGHT': 'RIGHT', 
    'P2_RIGHT': 'G', 
    'P1_LEFT': 'L', 
    'P2_LEFT': 'D', 
    'P1_DOWN': 'DOWN', 
    'P2_DOWN': 'F', 
    'P1_UP': 'UP', 
    'P2_UP': 'R', 
    'P1_SKICK': 'L-SHIFT', 
    'P2_SKICK': 'W', 
} 
""" 
hand gesture: 
0: Background,  
1: FingerHeart,  
2: ThumbUp,
3: Victory,
4: Mute, 
10: Palm,  
11: Okay, 
12: ThumbRight, 
13: ThumbLeft, 
14: Awesome, 
""" 
GESTURE_VICTORY = 3 
GESTURE_OKAY = 11 
GESTURE_AWESOME = 14 
 
class PlayerState(object): 
    """ 
    Player status 
    There are two players by default. 
    left_arm_line: state of left arm 
    right_arm_line: state of right arm 
    up_or_down: control up and down or squat jump 
    kick: kick 
    left_gesture: left hand gesture 
    right_gesture: right hand gesture 
    time_stamp: timestamp 
    """ 
 
    def __init__(self, right_arm_line=False, left_arm_line=False, move_direction=0, kick=False, right_gesture=0, 
                 left_gesture=0): 
        self.left_arm_line = left_arm_line 
        self.right_arm_line = right_arm_line 
        self.move_direction = move_direction 
        self.kick = kick 
        self.left_gesture = left_gesture 
        self.right_gesture = right_gesture 
        self.time_stamp = time.time() 
 
    def __str__(self): 
        return "\rarm " + str(self.left_arm_line) + " " + str(self.right_arm_line) + " move " + str( 
            self.move_direction) + " kick " + str(self.kick) + " gesture " + str(self.left_gesture) + " " + str( 
            self.right_gesture) + " " + str(self.time_stamp) 
 
 
class Player(object): 
    """ 
    This class defines players. 
    state_list The list stores a series of action states of players, which are obtained by matching the results of key point detection with the corresponding actions. 
    Method def get_ Action() filters the player's state_list to get the final action. 
    Filtering is to ensure the stability of output action. 
    """ 
 
    def __init__(self, name): 
        self.state_list = [] 
        self.id = 0 
        self.name = name 
 
    def get_action(self): 
        action_num = {} 
        buffer_time_step = 0.5 
        if len(self.state_list) < 15: 
            return [], {} 
        while self.state_list and time.time() - self.state_list[0].time_stamp > buffer_time_step: 
            self.state_list.pop(0) 
        if not self.state_list: 
            return [], {} 
 
        action_list = [] 
        select_tile = len(self.state_list) // int(buffer_time_step / 0.2) 
        right_arm_list = [state.right_arm_line for state in self.state_list] 
        left_arm_list = [state.left_arm_line for state in self.state_list] 
 
        if right_arm_list[-select_tile:].count(True) - select_tile // 5 > right_arm_list[ 
                                                                          -select_tile * 2:-select_tile].count( 
            True) or left_arm_list[-select_tile:].count(True) - select_tile // 5 > left_arm_list[ 
                                                                                   -select_tile * 2:-select_tile].count( 
            True): 
            action_list.append(self.name + '_JPUNCH') 
            action_num[self.name + '_JPUNCH'] = max(right_arm_list.count(True), left_arm_list.count(True)) 
 
        left_gesture_list = [state.left_gesture for state in self.state_list] 
        right_gesture_list = [state.right_gesture for state in self.state_list] 
        if left_gesture_list.count(GESTURE_VICTORY) > (len(left_gesture_list) / 3 + 1) or right_gesture_list.count( 
                GESTURE_VICTORY) > (len(right_gesture_list) / 3 + 1): 
            action_list.append('COIN_' + self.name) 
            action_num['COIN_' + self.name] = max(left_gesture_list.count(GESTURE_VICTORY), right_gesture_list.count( 
                GESTURE_VICTORY)) 
        elif left_gesture_list.count(GESTURE_OKAY) > (len(left_gesture_list) / 3 + 1) or right_gesture_list.count( 
                GESTURE_OKAY) > (len(right_gesture_list) / 3 + 1): 
            action_list.append(self.name + '_START') 
            action_num[self.name + '_START'] = max(left_gesture_list.count(GESTURE_OKAY), right_gesture_list.count( 
                GESTURE_OKAY)) 
 
        move_direction_list = [state.move_direction for state in self.state_list] 
 
        if move_direction_list.count(1) > len(move_direction_list) / 2: 
            action_list.append(self.name + '_DOWN') 
            action_num[self.name + '_DOWN'] = move_direction_list.count(1) 
        elif move_direction_list.count(2) > len(move_direction_list) / 2: 
            action_list.append(self.name + '_UP') 
            action_num[self.name + '_UP'] = move_direction_list.count(2) 
        elif move_direction_list.count(3) > len(move_direction_list) / 2: 
            action_list.append(self.name + '_RIGHT') 
            action_num[self.name + '_RIGHT'] = move_direction_list.count(3) 
        elif move_direction_list.count(4) > len(move_direction_list) / 2: 
            action_list.append(self.name + '_LEFT') 
            action_num[self.name + '_LEFT'] = move_direction_list.count(4) 
 
        kick_list = [state.kick for state in self.state_list] 
        if kick_list.count(True) > (len(kick_list) / 10 + 1): 
            action_list.append(self.name + '_SKICK') 
            action_num[self.name + '_SKICK'] = kick_list.count(True) 
        return action_list, action_num 
 
class Game(object): 
    """ 
    Instantiate players. 
    Give each player a unique ID. 
    To solve the track of key point detection_ For the problem that the player 
    cannot match after the ID changes, maintain a queue to store the ID. 
    """ 
 
    def __init__(self): 
        self.players = [Player("P1"), Player("P2")] 
        self.show_id_list = [] 
        self.action_buffer = set() 
        self.action_nums = {} 
 
    def push_state(self, hand_match_dict, body_parser_dict): 
        for track_id in body_parser_dict.keys(): 
            right_arm, left_arm, move_direction, kick = self.stat_detection(body_parser_dict[track_id]) 
            self.show_id_list.append(track_id) 
            while len(self.show_id_list) > 100: 
                self.show_id_list.pop(0) 
 
            match_idx = -1 
            for player_idx in range(2): 
                if self.players[player_idx].id == track_id: 
                    match_idx = player_idx 
 
            if match_idx == -1: 
                p0_sum = self.show_id_list.count(self.players[0].id) 
                p1_sum = self.show_id_list.count(self.players[1].id) 
                if p0_sum < 10: 
                    self.players[0].id = track_id 
                    match_idx = 0 
                elif p1_sum < 10: 
                    self.players[1].id = track_id 
                    match_idx = 1 
            if match_idx != -1: 
                right_gesture_id = 0 
                left_gesture_id = 0 
                if (track_id, 'left') in hand_match_dict: 
                    left_gesture_id = hand_match_dict[(track_id, 'left')] 
                if (track_id, 'right') in hand_match_dict: 
                    right_gesture_id = hand_match_dict[(track_id, 'right')] 
                self.players[match_idx].state_list.append( 
                    PlayerState(right_arm, left_arm, move_direction, kick, left_gesture_id, right_gesture_id)) 
        return self 
 
    @staticmethod 
    def stat_detection(points): 
 
        def is_line(a, b, c): 
            pow_len_a_b = (math.pow(a.x - b.x, 2) + math.pow(a.y - b.y, 2)) 
            pow_len_b_c = (math.pow(b.x - c.x, 2) + math.pow(b.y - c.y, 2)) 
            pow_len_a_c = (math.pow(a.x - c.x, 2) + math.pow(a.y - c.y, 2)) 
            cosC = -(pow_len_a_b + pow_len_b_c - pow_len_a_c) / (2 * math.sqrt(pow_len_a_b) * math.sqrt(pow_len_b_c)) 
            return cosC > 0.8 
 
        def is_y_similar(a, b, c): 
            x_a_c = abs(a.x - c.x) 
            if x_a_c < 0.1: 
                print("x is too small") 
                return False 
            y_a_c = abs(a.y - c.y) 
            x_y_ratio = y_a_c / x_a_c 
            return x_y_ratio < 0.4 
 
        def arm_status(): 
            is_right_line = is_line(points[10], points[8], points[6]) 
            is_right_y_similar = is_y_similar(points[10], points[8], points[6]) 
 
            is_left_line = is_line(points[9], points[7], points[5]) 
            is_left_y_similar = is_y_similar(points[9], points[7], points[5]) 
 
            return is_right_line and is_right_y_similar, is_left_line and is_left_y_similar 
 
        def move_direction(): 
            """ 
            0: stand 
            1: down 
            2: up 
            3: left 
            4: right 
 
            :return: 
            """ 
            # down 
            if abs(points[16].y - points[14].y) < abs(points[14].y - points[12].y) / 2 or abs( 
                    points[15].y - points[13].y) < abs(points[13].y - points[11].y) / 2: 
                return 1 
            # up 
            if points[8].y < points[0].y or points[7].y < points[0].y: 
                return 2 
 
            middle_x = (points[5].x + points[6].x + points[12].x + points[11].x) / 4 
            middle_y = (points[5].y + points[6].y + points[12].y + points[11].y) / 4 
            # right 
            if points[10].x < middle_x and points[9].x < middle_x and points[9].y > middle_y: 
                return 3 
            # left 
            if points[10].x > middle_x and points[9].x > middle_x and points[10].y > middle_y: 
                return 4 
            return 0 
 
        def kick(): 
            return (abs(points[16].y - points[13].y) < abs(points[15].y - points[14].y) and 
                    points[14].y < points[12].y) or abs(points[15].y - points[14].y) < abs( 
                points[15].y - points[13].y) and points[13].y < points[11].y 
 
        right_arm_line_, left_arm_line_ = arm_status() 
        move_direction_ = move_direction() 
        kick_ = kick() 
 
        return right_arm_line_, left_arm_line_, move_direction_, kick_ 
 
    def refresh_action(self): 
        for player in self.players: 
            action_list, action_num = player.get_action() 
            for action in action_list: 
                self.action_buffer.add(action) 
            self.action_nums = dict(Counter(self.action_nums) + Counter(action_num)) 
        return self 
 
    def get_action(self): 
        return self.action_buffer, self.action_nums 
 
    def clear_action(self): 
        self.action_buffer.clear() 
        self.action_nums = {} 
 
 
class socketer(object): 
    def __init__(self,ip,port): 
        self.ip = ip 
        self.port = port 
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
        self.connect() 
 
    def connect(self): 
        while True: 
            try: 
                self.socket.close() 
                self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
                self.socket.connect((self.ip, self.port)) 
                print('connect success') 
                return 
            except socket.error as e: 
                print('connect failed', e) 
                time.sleep(1) 
 
    def send(self, data): 
        try: 
            data = json.dumps(data) 
            print(data) 
            self.socket.send(bytes(data.encode('utf-8'))) 
        except socket.error: 
            self.connect() 
        except: 
            assert 0 
 
 
class MinimalSubscriber(Node): 
 
    def __init__(self, args): 
        super().__init__('minimal_subscriber') 
        self.subscription = self.create_subscription( 
            PerceptionTargets, 
            '/hobot_hand_gesture_detection', 
            self.listener_callback, 
            10) 
        self.subscription  # prevent unused variable warning 
        self.socketer = socketer(ip=args.ip, port=args.port) 
        self.game = Game() 
        self.auto_thread = Thread(target=self.auto_step, args=()) 
        self.auto_thread.start() 
 
    def auto_step(self): 
        need_hold_list = ['P1_RIGHT', 'P2_RIGHT', 'P1_LEFT', 'P2_LEFT',  'P1_DOWN', 'P2_DOWN'] #'P1_UP', 'P2_UP', 
        old_action_list=[] 
        old_action_num = {} 
        while True: 
            time.sleep(0.1) 
            action_list, action_num = self.game.get_action() 
            for act in old_action_list: 
                if act in action_list: 
                    action_list.remove(act) 
            old_action_list = action_list.copy() 
            key_list = [] 
            for act in action_list.copy(): 
                key_ = ACTION2KEY[act] 
                print(act, action_num) 
                if act in need_hold_list and act in old_action_num.keys() and (action_num[act] >= old_action_num[act]-20 and action_num[act] > 10): 
                    key_+=':hold0.3s' 
                key_list.append(key_) 
            if key_list: 
                self.socketer.send(key_list) 
            old_action_num = action_num.copy() 
            self.game.clear_action() 
 
    def hand_match(self, hand_parser_list, body_parser_dict): 
        def distance(x1, y1, x2, y2): 
            return math.sqrt(math.pow(x1 - x2, 2) + math.pow(y1 - y2, 2)) 
 
        hand_match_dict = {} 
        for hand_info in hand_parser_list: 
            min_distance = 10000 
            match_flag = (-1, 'left') 
            for track_id in body_parser_dict.keys(): 
                new_distance = distance(body_parser_dict[track_id][18].x, body_parser_dict[track_id][18].y, 
                                        hand_info[1][0], hand_info[1][1]) 
                if hand_info[1][1] > body_parser_dict[track_id][0].y: 
                    continue 
                if new_distance < min_distance and new_distance < 40: 
                    min_distance = new_distance 
                    match_flag = (track_id, 'right') 
                new_distance = distance(body_parser_dict[track_id][17].x, body_parser_dict[track_id][17].y, 
                                        hand_info[1][0], hand_info[1][1]) 
                if new_distance < min_distance and new_distance < 40: 
                    min_distance = new_distance 
                    match_flag = (track_id, 'left') 
            if match_flag[0] != -1: 
                hand_match_dict[match_flag] = hand_info[0] 
        return hand_match_dict 
 
    def get_body_hand_info(self, msg): 
        body_parser_dict = {} 
        hand_parser_list = [] 
        for targeter in msg.targets: 
            track_id = targeter.track_id 
            if len(targeter.points) == 0: 
                continue 
            point_object = targeter.points[0] 
            if point_object.type == 'body_kps': 
                body_points = point_object.point 
                body_points_confidence = point_object.confidence 
 
                if sum(body_points_confidence)/len(body_points_confidence) < 0.7: 
                    break 
                body_parser_dict[track_id] = body_points 
            if point_object.type == 'hand_kps': 
                average_x = 0.0 
                average_y = 0.0 
                hand_points = point_object.point 
                for p in hand_points: 
                    average_x += p.x 
                    average_y += p.y 
                average_x /= len(hand_points) 
                average_y /= len(hand_points) 
                if len(targeter.attributes) > 0 and targeter.attributes[0].type == 'gesture': 
                    gesture_id = targeter.attributes[0].value 
                    hand_parser_list.append((gesture_id, (average_x, average_y))) 
        hand_match_dict = self.hand_match(hand_parser_list, body_parser_dict) 
        return hand_match_dict, body_parser_dict 
 
    def listener_callback(self, msg): 
 
        # By monitoring the detected key points, the status information of the body and hands can be obtained. 
        #fps = msg.fps 
        #print(fps) 
        hand_match_dict, body_parser_dict = self.get_body_hand_info(msg) 
        self.game.push_state(hand_match_dict, body_parser_dict).refresh_action() 
 
        return None 
 
 
def main(args): 
    rclpy.init() 
    minimal_subscriber = MinimalSubscriber(args) 
    rclpy.spin(minimal_subscriber) 
 
    # Destroy the node explicitly 
    # (optional - otherwise it will be done automatically 
    # when the garbage collector destroys the node object) 
    minimal_subscriber.destroy_node() 
    rclpy.shutdown() 
 
def parse_args(): 
    description = "start web socket keyboard application" 
    parser = argparse.ArgumentParser(description=description) 
    parser.add_argument('--ip', help="the local ip number") 
    parser.add_argument('--port', help='set listen port ', default=8000) 
    args_ = parser.parse_args() 
    return args_ 
 
if __name__ == '__main__': 
    args = parse_args() 
    main(args) 

Two-use operation

There are three steps:

① On the Windows computer, run the python file downloaded in section 1.2 (ip query: enter ipconfig on the cmd command line to find your ip address) :

python3 your_python_file --ip your_ip 

② Start the game downloaded in chapter 1.1 (before starting, be sure to set the input method to English mode), the game start command is as follows:

mame -keyboardprovider win32 sfiii3nr1 

③ On the x3 pie, start the gesture detection example first, and then run the python script in section 1.3 on another terminal (note!! The ip address is the ip address of your Windows computer, as in step 1).

python3 your_python_file --ip your_remote_computer_ip