First Flight: Intel RealSense D435 Depth Camera on Jetson TX2

 

This is part of a series of posts outlining the evolution of my GroundHog hexacopter into a multi-role UAV.  It is based on a Pixhawk flight controller with a Jetson TX2 companion computer.  It has now been fitted with an Intel RealSense D435 depthcam.

My last blog described how to adapt and install the librealsense library so that the TX2 can support the D435 depthcam.  This post gives the first impressions of the performance of the depth camera in actual flight.

The full video of the flight is available on my YouTube channel here.

Summary

The Intel RealSense D435 camera was tested in free flight for general depth performance and specific object detection. It performed well up to distances of approximately 20m, with useful information beyond. The alignment of RGB, IR and depth cameras mean that specific objects within 15m, even if moving, should be able to be detected and tracked.  However, it was found that the interpretation of open sky was problematic, which will require further investigation.

Why the D435 Depth Camera

I chose the D435 camera because it has properties suited to in-flight work:

  • Small, relatively light (compared to the ZR300)
  • Long range (quoted at around 15m but wanted to explore limits)
  • Wider field of view (91 degrees compared with 64 degrees D415)
  • Global shutter to work best with fast moving images (such as drone flyby above)

More generally, images for depth, IR and RGB are all aligned which should make for rapid and programmatically easier interrogation of information from the data streams.  For example we can identify above a UAV using an image detection algorithm on the TX2, then identify the distance away using the D435 depth stream.  Do this a couple of times and we can then calculate a trajectory for potential avoidance manoeuvres.

However, the D435 has one or two drawbacks.

  • No on-board IMU.  So it will probably need to be rigidly mounted to the airframe to produce a point cloud for occupancy mapping.  I would prefer something that can be directed.
  • Sensitivity to USB3 power supply.  The D435 drops frames if the power supply is not of sufficient quality.  The TX2 development board has only one USB3 port.  In the absence of fitting a powered (flying) hub, the D435 must be the only device plugged into the port, so the TX2 has to be set up headless.  The solution here will be to fit the Auvidea J120 carrier board for the TX2 which has two USB3 ports.

Viewing the Image Streams Intel Realsense Viewer

Once librealsense has been installed, various utilities are available in the folder /usr/local/bin, among them realsense-viewer.  From the command line, run:

..bin $ ./realsense-viewer

22-04-2018 06-54-32
Not shown, but hovering over the depth image generates a read-out of the depth over that pixel.

The utility allows viewing of the depth stream alongside the IR stream (as above).  The full RGB image can also be displayed.

Most importantly, the panel on the left allows for the camera to be configured, with multiple options for capture and post-processing.  This should help to get the most value from the camera itself, taking advantage of it’s on-board processing capabilities.  For example I set up capture at a resolution of 848×480 (optimal for the D435), then down-sampled the depth stream by a factor of two in post-processing to lighten the processing load on the Jetson.  A very useful paper on configuring the D series cameras is available here.

Configurations can also be saved to a JSON file, which will be important for me in later development as the camera is launched and configured from a ROS node.

The depth visualisation stream (left above) can also be configured.  It is possible to constrain the colour scheme and the depth range for the purposes of visualisation.  Here it is used with the Jet colouring scheme and the colours are automatically scaled across the full depth estimated at any given instant.  Note this means that ‘red’ in one image might correlate to a different depth in another image.

A legend of the depths at any moment is displayed on the right of the image and changes throughout the video.

22-04-2018 07-05-47

In this example, the colour range is spread over 16m.  However in free flight, it can be seen the displayed depth could be over 70m (which prompts a later discussion of accuracy, of course).

Flight Conditions

The day was bright, but overcast. It should be noted that as the D435 only uses stereo for depth perception, performance actually improves in bright conditions.

Flight testing was split into two phases:
1. Sensing near complex objects (trees) to test general depth performance.
2. Detection of other flying ojects (a second drone) to test more specific object detection.

General Depth Performance

Range

Visualisation parameters were set to automatically scale colouration to the fullest depth range detected. In open flight, this ranged to over 75m.

22-04-2018 10-10-28
The tree canopy within 8m lower left.  Blue at the very top due to the propellers.

It can be seen that up to about 20m, depth resolution was consistent and so useful for flight decision making. An example is shown below in which a gap between the trees allowed for successful navigation over the bridge.

22-04-2018 10-14-29

22-04-2018 10-22-15
The railing also becomes more distinct against the background.

Over an open area of ground the consistent banding of colours correlating to the receding ground again show good consistency up to around 20m, arguably more. Beyond this, the pattern of colours become more varied, reflecting (amongst other things) the greater errors involved. Documentation from Intel indicates that depth error is proportional to the square of the distance, so this is no surprise.

22-04-2018 10-26-08
Interestingly, the gradient of the contours in the depth image reveals the land falling away to the left.  This is not obvious from the IR stream on the right.

So it may be possible to produce a reasonable point-cloud up to 20m or so, but perhaps not much beyond. Nevertheless, useful information for immediate trajectory planning may be available at distances over 20m.

Object Detection

A second UAV of diameter 50cm (motor to motor) was flown in the vicinity of the GroundHog.

Initially it was flown towards the GroundHog to test the distance at which it could be detected from the background signal.  The images below indicate that the depth stream was able to discern the UAV at a distance of approximately 20m.

22-04-2018 10-37-43
UAV estimated 18m away here.
22-04-2018 11-17-06
Similar distance
22-04-2018 11-20-39
About 4m away here.  UAV arms become distinguishable.

The UAV can clearly be seen in the depth stream as it passes under the GroundHog.  The question of whether the UAV can be spotted by image detection algorithms at this distance will require further testing.

The UAV was then flown across the fixed position of the GroundHog to test detection at faster lateral components of velocity.  This revealed an issue with the D435 concerning the interpretation of sky.

22-04-2018 10-43-36
Whilst the UAV is observable, the clouded sky proves a problem with the current configuration.

The patchwork interpretation of depth from open sky obfuscated the appearance of the UAV as it traversed.  However, it should be borne in mind that direct visual interpretation of the depth stream is not representative of it’s use.  More normally, the depth stream will be interrogated to find the reported depth at a particular location on the image identified by the machine learning algrithms.  As long as this remains valid, it may not matter that the interpretation of sky surrounding that location is not valid.  This aspect will require further testing.

Other Notes

  • Prop interference. The intrusion of the props at the top of the images, especially in forward flight, is reflected in the depth stream as random blue areas. This is not a malfunction of the camera.
  • Repetitive patterns. Intel documentation indicates that some repetitive patterns such as in fencing, may cause errors in depth perception as a result of the stereo camera techniques deployed. This was not noticeable in these tests, however in some preliminary hand-held ‘garden’ testing, I believe a did find this. My car trailer has a mesh panel on the rear, and this seemed to appear as a ‘blue patch’ (and so much closer) at an actual distance of some 15m. In flight conditions, this alone might be mistakenly interpreted as another object (maybe a bird), so needs to be borne in mind when designing image detection algorithms.

The GroundHog

Mission:  To evolve into a multi-role UAV with high endurance and multiple redundancies.

Current Configuration:

  • Tarot 680 Pro hexacopter airframe, with 3d printed additions
  • Pixhawk running Arducopter
  • Jetson TX2 as companion computer
  • Intel RealSense D435 depth camera
  • Multistar Elite 3508 KV268 motors
  • 13 x 4 carbon fibre props
  • Power from 8S, 8000mAh Lipos
  • 12V and 5V switched supplies
  • Servo camera gimbal
  • TeraRanger One rangefinder
  • Ublox GPS
  • And yes, those are Lezyne bike lights!

 

 

 

Advertisements

Author: Mike Isted

A multi-disciplinary technologist. Embedded software engineer.