Issue about using Depth tracking in VISP_ROS

As we can see in my previous post "Object Tracking using ViSP in ROS - 1st demo", I used some ViSP packages for tracking our woodblock (object of interest) within ROS environment.

However, this tracking method only used RGB information to discover the object pose in reference to the camera link. This means that the Depth map provided by the same camera (Intel RealSense D435), did not contribute at all to the woodblock pose estimation.

For using ViSP as a ROS node for performing the tracking, there is a class called vpROSGrabber responsible for reading images out of ROS topics.

vpROSGrabber class does not have any methods ready to receive and read images of type 16UC1. The depth map published by the RealSense Camera Manager is a 16UC1 matrix, that cannot be directly subscribed by the ViSP node.

Because of this, I tried to convert the 16UC1 data, following this: 

and using the C library function memcpy().

I did get to expand the vpROSGrabber to receive such type of depth maps, but then the tracking result was clearly misplaced. I still didn't figure out why the tracker was continuously estimating a pose that obviously wasn't the right one.

This study was made in November/December of 2021 and by that time I didn't record any of these results. 

After this, and following the purpose of the project (that is way more related to the robot control than the environment perception), we choose to try to work with ViSP workspace out of ROS. This way, the challenge now is to send the important information that comes out of ViSP into ROS environment.