The “pencil down” date for Google Summer of Code 2015 has passed, and now it’s time to summarize what I have done during these months. It has been a very nice experience, mainly because I have had the possibility to work on a unique project, ERAS, by exploiting my previous experiences with Microsoft Kinect.
The title of my project was “Enhancement of Kinect integration in V-ERAS”, and you can read the full proposal here. In this post I will briefly summarize which were the goals of my project, and how I addressed them.
My project can be divided in four main stages:
- rewrite the existing body tracker in order to port it from C# to Python (which is the main language used in ERAS);
- implement a GUI to manage multiple Kinects connected to a single Windows machine;
- improve user’s navigation, by working on the user’s step estimation algorithm adopted in ERAS;
- integrate gesture recognition
Porting body tracker from Python to C#
The first step was to reimplement the body tracker by using Python language. After the V-ERAS-14 mission, where a non-working tracker written in C++ and based on OpenNI was used, the ERAS team made the decision to adopt Microsoft API, because they have proved to be more efficient in tracking skeletal joints. However, only C#, VB.NET and C++ programming languages were supported by Microsoft Kinect SDK v1.x. The only alternative out there to write a Python-based tracker was to use an open source project called PyKinect. It is basically a porting of the C++ capabilities of Microsoft API in Python, by strongly using ctypes module.
PyKinect have proved to be quite reliable and usable, although it was quite poorly documented. After some tests (some of which are now available as snippets on my BitBucket account, for everyone who wants to start with this Python solution for Kinect), I was able to implement a working tracker, which was able to recognize skeletal joints and send them to a Tango bus.
For who does not know it, Tango is the control system on which the whole ERAS software is based. It allows to implement several device server which can publish various kind of data on a bus that can be accessed by any other device server or generic software. In Python, the PyTango module allows to interact with Tango very easily, and it allowed me to implement a Tango device which publishes skeletal joints recognized by Kinect. I strongly recommend to take a look to this useful tutorial on how to implement a Tango device server.
This section of my source code is mainly part of tracker.py script.
GUI for managing multiple Kinects
With the previous C# code, I had implemented a GUI for managing multiple Kinect. It allowed to change Kinect tilt angle, and assign a Kinect to a specific Tango device. However, it did not display user’s images (depth, color and/or skeletal data), and this feature has revealed to be useful, mainly during setup of Motivity/Motigravity and at the beginning of user’s interactions.
After porting everything in Python, the C# GUI became no more usable, so I had to implement it in Python too. I decided to use the pgu library, because it was one of the solution compliant with pygame, a module used together with PyKinect to implement skeletal tracking. The resulting Python GUI includes the same capabilities of the previous one, with the addition of displaying depth and skeletal data of users. The following image shows how it looks with just one user:
The code related with GUI can be found in the gui.py script.
Improving users’ step estimation algorithm
User’s step estimation algorithm is one of the most significant part of ERAS software in terms of affecting user’s navigation in the virtual martian environment. Before GSoC 2015 started, this procedure was implemented as a Blender script, while now the idea is to provide user’s steps data from the body tracker module.
During V-ERAS-14 mission, user’s navigation was marked as one of the main issues from a user’s perspective. In order to improve this, I tried to change a little bit the algorithm, which is now implemented in tracker.py (if you are interested, look at the
User’s step estimation has been also used by Siddhant Shrivastava, another student that has worked for a ERAS-related project during this GSoC. He produced a very interesting video to demonstrate how user’s steps can be used to teleoperate a (virtual) rover in ROS:
One of the most interesting and challenging features that I have implemented during this GSoC was the gesture recognition capabilities. I focused on recognizing if user’s hands are open or closed. PyKinect, as well as Microsoft API, is not able to recognize this, but an extension made available by Microsoft and called Kinect Interactions includes some gesture recognition capabilities.
What I have done was to use the DLL file available with Microsoft Kinect Developer Toolkit 1.8, and convert C++ methods available in it to something usable in Python. To do this, I wrote some C++ code that simplify the porting, and relied on ctypes to use C++ methods in Python code. You can read something more in this post.
Until now, the system is capable to recognize if the hands are open or closed. Kinect Interactions should allow also to recognize a “press gesture”, but it seems to be more complicated than expected.
While the main goals of my project was the four ones described above, I also contributed with something not previously scheduled, but useful for future applications. I have implemented an algorithm to estimate user’s height, which can be retrieved as result of a Tango command (
get_height). You can see the related code here.
I have also helped the IMS with some user data collection and analysis, by writing some Python scripts used for aggregating data taken from Oculus Rift and Microsoft Kinect during training sessions for AMADEE (see this post for more details).
I hope to have exhaustively described all my contributions to the ERAS project during this GSoC 2015. In the future I think I will continue in collaborating with the Italian Mars Society, to improve Kinect integration a little bit more (we still need some full integration test to verify if everything is working correctly).
It was a great experience, and I hope to participate to the next GSoC too. But for the moment that’s all!