3D cameras play an important role in the modern industry. They help us solve computer vision tasks that can’t be completed with 2D data. Two years ago, OpenCV supported Orbbec 3D cameras of Astra series. Now here comes more good news. OpenCV supports its 3D UVC cameras!

Unlike working with previous Astra 3D cameras which requires OpenCV built with OpenNI2 SDK, no third party libraries are required to work with the new 3D UVC cameras of Astra+ series and Femto series. Plug in the 3D camera and the data from the depth sensor and color sensor or the RGBD streams can be read by using the cv::VideoCapture class. Calibration and alignment of the depth map and color image are also done without effort.

Below is an example of how easily OpenCV works with the Orbbec 3D UVC cameras.

#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>

using namespace cv;
int main()
{
    VideoCapture obsensorCapture(0, CAP_OBSENSOR);
    if(!obsensorCapture.isOpened()){
        std::cerr << "Failed to open obsensor capture! Index out of range or no response from device";
        return -1;
    }

    double fx = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_FX);
    double fy = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_FY);
    double cx = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_CX);
    double cy = obsensorCapture.get(CAP_PROP_OBSENSOR_INTRINSIC_CY);
    std::cout << "obsensor camera intrinsic params: fx=" << fx << ", fy=" << fy << ", cx=" << cx << ", cy=" << cy << std::endl;

    Mat image;
    Mat depthMap;
    Mat adjDepthMap;
    while (true)
    {
        // Grab the depth map:
        // obsensorCapture >> depthMap;

        // Another way to grab the depth map (and bgr image).
        if (obsensorCapture.grab())
        {
            // bgr image
            if (obsensorCapture.retrieve(image, CAP_OBSENSOR_BGR_IMAGE))
            {
                imshow("RGB", image);
            }

              // depth map
            if (obsensorCapture.retrieve(depthMap, CAP_OBSENSOR_DEPTH_MAP))
            {
                normalize(depthMap, adjDepthMap, 0, 255, NORM_MINMAX, CV_8UC1);
                applyColorMap(adjDepthMap, adjDepthMap, COLORMAP_JET);
                imshow("DEPTH", adjDepthMap);
            }

            // overlay depth map on bgr image
            static const float alpha = 0.6f;
            if (!image.empty() && !depthMap.empty())
            {
                normalize(depthMap, adjDepthMap, 0, 255, NORM_MINMAX, CV_8UC1);
                cv::resize(adjDepthMap, adjDepthMap, cv::Size(image.cols, image.rows));
                for (int i = 0; i < image.rows; i++)
                {
                    for (int j = 0; j < image.cols; j++)
                    {
                        cv::Vec3b& outRgb = image.at<cv::Vec3b>(i, j);
                        uint8_t depthValue = 255 - adjDepthMap.at<uint8_t>(i, j);
                        if (depthValue != 0 && depthValue != 255)
                        {
                            outRgb[0] = (uint8_t)(outRgb[0] * (1.0f - alpha) + depthValue * alpha);
                            outRgb[1] = (uint8_t)(outRgb[1] * (1.0f - alpha) + depthValue *  alpha);
                            outRgb[2] = (uint8_t)(outRgb[2] * (1.0f - alpha) + depthValue *  alpha);
                        }
                    }
                }
                imshow("DepthToColor", image);
            }
            image.release();
            depthMap.release();
        }

        if (pollKey() >= 0)
            break;
    }
    return 0;
}

ORBBEC, founded in 2013 and headquartered in Shenzhen, is a total solution provider of 3D perception technology, including sensor, software and algorithms. It is the fourth company that produces 3D sensors in mass production, right after Microsoft, Apple and Intel.

The company has launched multiple types of 3D sensors which have been widely used in industries such as mobile phones, smart retails, smart manufacturing, robotics and logistics. Nowadays Orbbec becomes one of the leading players in the field of 3D sensing technologies, and aims to provide “3D Eyes of Wisdom” for all terminals to understand the world!

Orbbec is one of the OpenCV development partners. It is of common interest for OpenCV and Orbbec to collaborate widely to serve the developers in the 3D computer vision fields.