This project proposal envisions to accurately track body segment orientations, for a given arbitrary posture, using machine learning to estimate internal rotations of each body segment given a minimal set of points that represent joints and body extremities.The major goal of this project proposal is to develop a new algorithm for skeletal tracking with anatomically correct segment orientation by resorting on advanced machine learning techniques. We aim to augment current markerless motion capture used in exergames and rehabilitation. To validate the algorithm, two case scenarios will be considered: (i) rehabilitation for stroke recovery of patients with impaired upper limb movement; and (ii) exergaming with punching, jumping jacks, squatting exercises. Interested users such as physiotherapists, personal trainers, students and even patients, will be in the same room and interacting with the content. To this end, depth camera sensors will be used to capture users’ posture.