GSoC 2015

Enhancement of Kinect integration in V-ERAS: Final report

The “pencil down” date for Google Summer of Code 2015 has passed, and now it’s time to summarize what I have done during these months. It has been a very nice experience, mainly because I have had the possibility to work on a unique project, ERAS, by exploiting my previous experiences with Microsoft Kinect.

I introduced the project in this blog post, so if you want to have some overview of ERAS and what the Italian Mars Society (IMS) is doing, please read it or go to erasproject.org.

The title of my project was “Enhancement of Kinect integration in V-ERAS”, and you can read the full proposal here. In this post I will briefly summarize which were the goals of my project, and how I addressed them.

My project can be divided in four main stages:

  1. rewrite the existing body tracker in order to port it from C# to Python (which is the main language used in ERAS);
  2. implement a GUI to manage multiple Kinects connected to a single Windows machine;
  3. improve user’s navigation, by working on the user’s step estimation algorithm adopted in ERAS;
  4. integrate gesture recognition

Porting body tracker from Python to C#

The first step was to reimplement the body tracker by using Python language. After the V-ERAS-14 mission, where a non-working tracker written in C++ and based on OpenNI was used, the ERAS team made the decision to adopt Microsoft API, because they have proved to be more efficient in tracking skeletal joints. However, only C#, VB.NET and C++ programming languages were supported by Microsoft Kinect SDK v1.x. The only alternative out there to write a Python-based tracker was to use an open source project called PyKinect. It is basically a porting of the C++ capabilities of Microsoft API in Python, by strongly using ctypes module.

PyKinect have proved to be quite reliable and usable, although it was quite poorly documented. After some tests (some of which are now available as snippets on my BitBucket account, for everyone who wants to start with this Python solution for Kinect), I was able to implement a working tracker, which was able to recognize skeletal joints and send them to a Tango bus.

For who does not know it, Tango is the control system on which the whole ERAS software is based. It allows to implement several device server which can publish various kind of data on a bus that can be accessed by any other device server or generic software. In Python, the PyTango module allows to interact with Tango very easily, and it allowed me to implement a Tango device which publishes skeletal joints recognized by Kinect. I strongly recommend to take a look to this useful tutorial on how to implement a Tango device server.

This section of my source code is mainly part of tracker.py script.

GUI for managing multiple Kinects

With the previous C# code, I had implemented a GUI for managing multiple Kinect. It allowed to change Kinect tilt angle, and assign a Kinect to a specific Tango device. However, it did not display user’s images (depth, color and/or skeletal data), and this feature has revealed to be useful, mainly during setup of Motivity/Motigravity and at the beginning of user’s interactions.

After porting everything in Python, the C# GUI became no more usable, so I had to implement it in Python too. I decided to use the pgu library, because it was one of the solution compliant with pygame, a module used together with PyKinect to implement skeletal tracking. The resulting Python GUI includes the same capabilities of the previous one, with the addition of displaying depth and skeletal data of users. The following image shows how it looks with just one user:

GUI for managing multiple Kinects
GUI for managing multiple Kinects

The code related with GUI can be found in the gui.py script.

Improving users’ step estimation algorithm

User’s step estimation algorithm is one of the most significant part of ERAS software in terms of affecting user’s navigation in the virtual martian environment. Before GSoC 2015 started, this procedure was implemented as a Blender script, while now the idea is to provide user’s steps data from the body tracker module.

During V-ERAS-14 mission, user’s navigation was marked as one of the main issues from a user’s perspective. In order to improve this, I tried to change a little bit the algorithm, which is now implemented in tracker.py (if you are interested, look at the estimate_user_movements function).

User’s step estimation has been also used by Siddhant Shrivastava, another student that has worked for a ERAS-related project during this GSoC. He produced a very interesting video to demonstrate how user’s steps can be used to teleoperate a (virtual) rover in ROS:

Gesture recognition

One of the most interesting and challenging features that I have implemented during this GSoC was the gesture recognition capabilities. I focused on recognizing if user’s hands are open or closed. PyKinect, as well as Microsoft API, is not able to recognize this, but an extension made available by Microsoft and called Kinect Interactions includes some gesture recognition capabilities.

What I have done was to use the DLL file available with Microsoft Kinect Developer Toolkit 1.8, and convert C++ methods available in it to something usable in Python. To do this, I wrote some C++ code that simplify the porting, and relied on ctypes to use C++ methods in Python code. You can read something more in this post.

Until now, the system is capable to recognize if the hands are open or closed. Kinect Interactions should allow also to recognize a “press gesture”, but it seems to be more complicated than expected.

Press Gesture
The "pess gesture", as defined by Microsoft

The code related to gesture recognition is available here (C++) and here (Python).

Other contributions

While the main goals of my project was the four ones described above, I also contributed with something not previously scheduled, but useful for future applications. I have implemented an algorithm to estimate user’s height, which can be retrieved as result of a Tango command (get_height). You can see the related code here.

I have also helped the IMS with some user data collection and analysis, by writing some Python scripts used for aggregating data taken from Oculus Rift and Microsoft Kinect during training sessions for AMADEE (see this post for more details).

Conclusions

I hope to have exhaustively described all my contributions to the ERAS project during this GSoC 2015. In the future I think I will continue in collaborating with the Italian Mars Society, to improve Kinect integration a little bit more (we still need some full integration test to verify if everything is working correctly).

It was a great experience, and I hope to participate to the next GSoC too. But for the moment that’s all!

Ciao!

Advertisements
GSoC 2015

Enhancement of Kinect integration in V-ERAS: Sixth report

This is my sixth report on what I have done for my GSoC project. If you don’t know what it is about and want to find more information, please refer to this page and this blog post.

During the last two weeks, I have mainly worked on two topic: improving the gesture recognition module by fixing some bugs and adding a script to automatically use the last version of the module, and trying to debug the user’s step estimation algorithm by using ROS.

For what about the gesture recognition module, it is still able to detect if the hands are closed or open, but I have not implemented yet the press gesture (which seems to be easily doable anyway). What I have done was to make everything usable for a system manager, which should not focus on compiling and fixing dependencies, but just executing the software. I have written a script which check if the DLLs files are in the right folder and if they are up-to-date. If not, the script copied these files in the right location, so that there are no errors also after compiling the C++ module (which actually implements the gesture recognition, and it is used by ctypes in Python).

I have also tried to install ROS Indigo in my Ubuntu virtual machine, but at the end of the story I realized that trying to do it with Ubuntu 14.10 is not a good idea… Actually the ERAS project recommended to use Ubuntu 14.04, but I have used 14.10 until now, so I hoped to continue without reinstall everythin in another virtual machine. Unfortunately, now I know that it is necessary, and this is what I will do. After installing ROS, I will be able to use the Turtle simulator to easily debug the step estimation procedure (which actually seems to work quite well, as you can see from the good quality of this simulation video, made by Siddhant):

Now the end of GSoC is approaching, and what I hope to finalize is basically the implementation of press gesture recognition, a review of the body tracker documentation (which is quite acceptable actually) and the implementation of some unit tests.

I will update you on my work!

Ciao!

GSoC 2015

Enhancement of Kinect integration in V-ERAS: Fifth report

This is my fifth report on what I have done for my GSoC project. If you don’t know what it is about and want to find more information, please refer to this page and this blog post.

After finalizing the user step estimation (which is still under test by Siddhant, and probably will require some refinements), during the last week I have also helped Yuval with some scripts to help him in analyzing users’ data. What I have done is mainly to aggregate Kinect and Oculus Rift data, and outputting them in a single file. This has been made possible by using the timestamps related to every single line in the files, in order to synchronize data from different files.

I have not committed these files yet, because Franco has also worked on these stuff; so probably he will commit everything in a while, as soon as he can.

The second (and more important, interesting and compelling) task that I have just finished to implement (although it will need some minor improvements) is to implement hand gesture recognition. This feature is not included in PyKinect, but it ships with the Microsoft Kinect Developer Toolkit, as part of what they called Kinect Interactions. Because PyKinect is based on the C++ Microsoft Kinect API, I have decided to implement this feature in this language (so that I can use the API, rather than reimplementing everything from scratch), and then port it in Python by mean of ctypes.

I had never used ctypes before, and this implied a lot of hard work, but at the end of the story I figure out how to use this powerful technology. Here there are some links useful to everyone wants to start using this technology:

The whole C++ module is stored in this directory of ERAS repository, and its output is a .dll file named KinectGestureRecognizer.dll. This file needs to be placed in the same directory of tracker.py before executing the body tracker, as well as the KinectInteraction180_32.dll. The latter ships with the Developer Toolkit, and it can be found in C:\Program Files\Microsoft SDKs\Kinect\Developer Toolkit v1.8.0\bin.

Then, I have written a Python wrapper by using ctypes; you can see it at this link. I had also tried to use ctypesgen to automatically generate Python wrapper from header files, but it didn’t seem easy to use to me (mainly due to some issues with the Visual Studio C++ compiler).

I also had to change some settings in order to enable Kinect Interactions to work in the proper way, and it implied to edit also tracker.py and gui.py. For instance, I had to change the depth resolution used, which is now 640×480 pixels, while it was 320×240.

Another script involved in the last commit was visualTracker.py. This file is very useful for testing purposes, because it allows you to see an avatar of the user, moving in 3D space. After adding gesture recognition, I decided to improve the avatar by coloring the hand joints in red if the hand is closed.

I have also helped the IMS with their current mission AMADEE, by setting up one of their machine (the Windows one). This way we can verify if what we have developed during these months works fine, by testing it in a real context with several users.

That’s it for the moment. I will update you soon for what about my GSoC project!

Cheers!

GSoC 2015

Enhancement of Kinect integration in V-ERAS: Fourth report

This is my fourth report on what I have done for my GSoC project. If you don’t know what it is about and want to find more information, please refer to this page and this blog post.

During the past two weeks I have worked mainly on two issues: finalizing a first user’s step estimation algorithm, and supporting data analysis during and after the training session for AMADEE’15.

For what about the user’s step estimation, I have implemented this algorithm, which uses skeletal data got by Kinect to estimate user’s rotation and the walked distance every time a new skeletal frame is tracked. Then a Tango change-event on the moves attribute is fired, and any other Tango module can subscribe to this event in order to use this data and implement user’s navigation. This whole idea will be tested by using a module that Siddhant is writing, which will use estimated user’s movements to animate a rover on the (virtual) Mars surface.

I have also worked to support a training session for the AMADEE’15 mission, which has taken place in Innsbruck and was organized by the Austrian Mars Society. During this training session, the Italian Mars Society was there to test their V-ERAS system. What I did was, firstly, to configure two Windows 7 machines to be able to execute the new Python-based body tracker. For this purpose we used Team Viewer for remote control of PCs. After that, we noticed a strange issue, which did not allow to use the new body tracker, due to some strange Tango error (we are going to report this to the Tango community). To overcome this annoying and unexpected problem, the old body tracker (written in C# and still available in the ERAS repository) was used.

I have also written some scripts to support Yuval Brodsky and the other team members of IMS to evaluate the effects of virtual reality on the neurovestibular system. To do this, I wrote a first script to get the positions of head and torso skeletal joints from the body tracker, and a second script to convert this data in .xlsx format (to be used by Yuval in data analysis). This allowed me to learn how to use openpyxl, a very easy to use and powerful Python module for writing .xlsx files. To get a feel on it, take a look to this sample code:

from openpyxl import Workbook
wb = Workbook()

# grab the active worksheet
ws = wb.active

# Data can be assigned directly to cells
ws['A1'] = 42

# Rows can also be appended
ws.append([1, 2, 3])

# Python types will automatically be converted
import datetime
ws['A2'] = datetime.datetime.now()

# Save the file
wb.save("sample.xlsx")

The scripts I have written for data analysis are not yet on the repository (we are trying to improve them a little bit), and now I have to find a way to include, in the same .xlsx file generated from Kinect data, also data taken from Oculus Rift.

The next step then will be to include also some gesture recognition, in particular the possibility to identify if user’s hands are open or closed.

I will keep you updated with the next posts!

Ciao!

GSoC 2015

Enhancement of Kinect integration in V-ERAS: Mid-term summary

This is my third report on what I have done for my GSoC project. If you don’t know what it is about and want to find more information, please refer to this page and this blog post.

In this report, I will summarize what I have done until now, and also describe what I will do during the next weeks.

My project is about the enhancement of Kinect integration in V-ERAS, which was all based on C#, in order to use the official Microsoft API (SDK version: 1.8). However, the whole ERAS source code is mainly written in Python, so the first step was to port the C# body tracker in Python, by using PyKinect. This also required the rewrite of all the GUI (by using PGU).

Then, I have also integrated the height estimation of the user in the body tracker, by using skeletal information for calculating it. This has been implemented as a Tango command, so that it can be executed by any device connected to the Tango bus. This feature will be very useful to modulate the avatar size before starting simulation in V-ERAS.

I have also took a look to the webplotter module, which will be useful for the incoming AMADEE mission, to verify the effect of virtual reality interaction on user’s movements. What I have done is to edit the server.py script, which was not able to manage numpy arrays. These structures are used by PyTango for attributes defined as “SPECTRUM”; in order to correctly save user’s data in JSON, I had to add a custom JSON encoder (see this commit for more information).

What I am starting to do now is perhaps the most significant part of my project, which is the implementation of user’s step estimation. At the moment, this feature is integrated in the V-ERAS Blender repository, as a Python Blender script. The idea now is to change the architecture to be event-based: everytime a Kinect frame with skeletal data is read by the body tracker, it will calculate user’s movements in terms of body orientation and linear distance, and  I will push a new change event. This event will be read by a new module, that is being developed by Siddhant (another student which is participating to GSoC 2015 with IMS and PSF), to move a virtual rover (or any other humanoid avatar) according to user’s movements.

I have started to developing the event-based architecture, and what I will start to do in these days is to integrate the step estimation algorithm, starting from the one that is currently implemented in V-ERAS Blender. Then I will improve it, in particular for what about the linear distance estimation; the body orientation is quite well calculated with the current algortihm indeed, so although I will check its validity, hopefully it will be simply used as it is now.

The last stage of my project will be to implement gesture recognition, in particular the possibility to recognize if user’s hands are closed or not. In these days I had to implement this feature in C# for a project that I am developing for my PhD research. With Microsoft Kinect SDK 1.8, it is possible by using KinectInteraction, but I am still not sure about the feasibility of this feature with only PyKinect (which is a sort of binding of the C++ Microsoft API). I will discover more about this matter in the next weeks.

I will let you know every progress with the next updates!

Stay tunes!

GSoC 2015

Enhancement of Kinect integration in V-ERAS: Second report

This is my second report about what I have done for my GSoC project. If you don’t know what it is about and want to find more information, please refer to this page and this blog post.

The first problem I had to solve was to implement a valid head estimation algorithm. You can find the code of how I had implemented it at this link, while for the algorithm itself, I have also recently discussed it in an answer on StackOverflow.

After the height estimation (that we decided to implement as a Tango command), the next step was to update the Tango classes that I had added with the previous commits, in order to use the new Tango API. You can find more about this topic in this very useful documentation page on “High level server API”. This update allowed me to reduce the number of lines of code, and it is also much more simple to implement commands or events in the Tango server now. However I had some issues with data types and starting the server (which made me stuck for a bit). Thankfully I finally fix them in my last commit, yesterday evening.

I have also worked on the documentation. In particular, I have updated the Software Architecture Document (SAD) for the Body Tracker application, by adding the CLI section and updating the GUI part with some new features introduced together with the Python-based interface.

GUI for managing multiple Kinects
GUI for managing multiple Kinects

I have also removed a redundant document, named “Execution of body tracker”, that was about how to execute the old tracker (which was written in C# and is still available in the repository, but basically to be deprecated).

For more information about my project and the other ones supported by Italian Mars Society and Python Software Foundation, refers to the GSoC2015 page of ERAS website.

GSoC 2015

Enhancement of Kinect integration in V-ERAS: First report

This is the first report about what I have done in these weeks for my GSoC project. If you don’t know what it is about and want to find more information, please refer to this page and this blog post.

I started with some documentation, by updating the old one and adding a SAD document that describes my project. You can read it at this link, in the ERAS repository.

The main part of my work during these weeks was focused on porting in Python the existing C# code to track skeletal joints with Kinect. I did it by using PyKinect, and you can find the code in the ERAS repository, under the body_tracker module.

I had also the possibility to test my code by using some recordings of an user walking on a Motivity. These recordings were made by using Kinect Studio, and fortunately PyKinect has also revealed its ability to be linked to this tool. In this way, it is possible to simulate depth and color Kinect data by using previous recordings.

I had also to reorganize the two modules related to skeletal tracking, that were present in the repo:

  • body_tracker_ms, which included the C# implementation (based on Microsoft API) was removed, but the C# code is still there, under the old folder in the body_tracker module;
  • body_tracker is now the only tracker module, and it includes the Python files to track the skeletal joints (tracker folder), the old C# implementation of skeletal tracker (old folder), the only Blender-related file still included in this module (blender/blenderCoordinates.py) and the documentation. No more C++ files are there, because they used OpenNI, and now ERAS has moved to Microsoft API.

Also the old documentation (which was related to a not working version of the tracker) was removed.

For what concerns the Python code, I have written mainly two scripts: tracker.py and gui.py.
In tracker.py there are the classes related to Tango, and a Tracker class that can be used to manage a single tracker. The script can be executed as it is (mainly for test purposes), and it is also able to record tracked joints in a JSON file, and simulate the tracking by reading them from the latter JSON file (without the need of a Kinect plugged-in). In practice, tracker.py is not executed as a standalone script, but it is imported in gui.py, where the Tracker class is used.
The gui.py script is the one that displays the GUI to manage up to 4 Kinects simultaneously connected and working on a sinle PC. It uses the aforementioned Tracker class to instantiate up to four “trackers”, each of which takes data from a single Kinect.

The very last feature that I have been starting to implement in these days is the evaluation of user’s height, and the publishing of it in the Tango bus. The estimation is done by calculating distances between some joints. You can see some code at this link (although it is still in an early stage) but in order to understand it better, may be useful to take a look at the following image, which represents the positions of skeletal joints tracked by the Microsft Kinect SDK 1.x:

Position of skeletal joints in Microsoft Kinect SDK 1.x
Position of skeletal joints in Microsoft Kinect SDK 1.x

For more information about my project and the other ones supported by Italian Mars Society and Python Software Foundation, refers to the GSoC2015 page of ERAS website.