ros image processing correlation

1. image_transport

ros specifies a variety of basic data structures for transmission between node s. Image is also a commonly used data, image_transport package is used to process image data transmission. It is a framework that only provides the most basic raw image data. If you need to reduce the bandwidth during transmission, you also need to compress the format, which is completed by various plug-ins integrated under the framework.

image_transport will publish sensor_msgs/Image Other compressed formats are provided by the plug-in.

The original image format "sensor_msgs/Image" in ROS is as follows:

                        # This message contains an uncompressed image
                        # (0, 0) is at top-left corner of image
Header header           # std_msgs/Header

uint32 height           # image height, that is, number of rows
uint32 width            # image width, that is, number of columns

                        # The legal values for encoding are in file src/image_encodings.cpp
                        # If you want to standardize a new string format, join
                        # and send an email proposing a new encoding.
string encoding         # Encoding of pixels -- channel meaning, ordering, size
                        # taken from the list of strings in include/sensor_msgs/image_encodings.h

uint8 is_bigendian      # is this data bigendian?
uint32 step             # Full row length in bytes
uint8[] data            # actual matrix data, size is (step * rows)

Where header is the standard structure:

                        # Standard metadata for higher-level stamped data types.
                        # This is generally used to communicate timestamped data
                        # in a particular coordinate frame.
                        # sequence ID: consecutively increasing ID
uint32 seq

                        #Two-integer timestamp that is expressed as:
                        # * stamp.sec: seconds (stamp_secs) since epoch (in Python the variable is called 'secs')
                        # * stamp.nsec: nanoseconds since stamp_secs (in Python the variable is called 'nsecs')
                        # time-handling sugar is provided by the client library
time stamp

                        #Frame this data is associated with
string frame_id

A specific example:

root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw --noarr
  seq: 293              # Current frame sequence number
  stamp:                # Gets the timestamp of the image
    secs: 1636868180
    nsecs:  53476744
  frame_id: "camera"    # frame_id
height: 480             # Number of rows
width: 640              # Number of columns
encoding: "bgr8"        # The compression format of pixels, the number of channels is 3, the order is BGR, and the depth is 24. According to this, you can know the meaning of the value in the data array
is_bigendian: 0         # Big end sort
step: 1920              # Step size, number of bytes in a row, width (640) x number of pixel bytes (3) = 1920bytes. The image has 480 lines in total, that is, 1920x480=921600 bytes, which is also the length of the lower data array
data: "<array type: uint8, length: 921600>"

Examples of use
Do not use image_ Subscription publication for transport:

// Do not communicate images this way!
#include <ros/ros.h>

void imageCallback(const sensor_msgs::ImageConstPtr& msg)
  // ...

ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe("in_image_topic", 1, imageCallback);
ros::Publisher pub = nh.advertise<sensor_msgs::Image>("out_image_topic", 1);

Using image_ Subscription publication for transport:

// Use the image_transport classes instead.
#include <ros/ros.h>
#include <image_transport/image_transport.h>

void imageCallback(const sensor_msgs::ImageConstPtr& msg)
  // ...

ros::NodeHandle nh;
image_transport::ImageTransport it(nh);  // It is equivalent to encapsulating the original nh once
image_transport::Subscriber sub = it.subscribe("in_image_base_topic", 1, imageCallback);
image_transport::Publisher pub = it.advertise("out_image_base_topic", 1);

What to offer:



image_transport itself only publishes sensors on the basic topic: < base topic >_ Images in the original format of msgs / image, if there are other plug-ins, will be automatically published in other topics under the basic topic: < base topic > / < transport name >, for example, the basic topic is "/ cv_camera",
image_transport will publish topic in the original format of "/ cv_camera/image_raw" and the plug-in compressed_image_transport will publish images in compressed format in "/ cv_camera/image_raw/compressed".

After you subscribe to a basic topic, you will automatically subscribe to other related topics.


image_transport itself does not publish parameters, but the plug-in will publish various parameters, such as frame rate, compression level, etc. it depends on the plug-in itself.
Publishers should use dynamic parameter mechanism to make parameters easier to be used by subscribers.
The parameter format is generally: < base topic > / < transport name > / < parameter name >
For example: / cv_camera/image_raw/compressed/parameter_descriptions


$ rosrun image_transport republish [in_transport] in:=<in_base_topic> [out_transport] out:=<out_base_topic>

For example, decompress the topic image in theora format and republish it to another topic. It can be written as follows:

$ rosrun image_transport republish theora in:=camera/image raw out:=camera/image_decompressed

If a node only publishes the original image format, you can use image_transport re publishes topic s in multiple formats.

Command line tools

$ rosrun image_transport list_transports

You can view all current package s to see whether there are available image processing plug-ins and their status.

The following plug-in introduction
Using image_ A node command provided by transport: list_transports, you can view the current topic s and corresponding plug-ins:

root@suntus:~# rosrun image_transport list_transports
Declared transports:
image_transport/ffmpeg (*): Not available. Try 'catkin_make --pkg ffmpeg_image_transport'.

 - Provided by package: compressed_image_transport
 - Publisher:
      This plugin publishes a CompressedImage using either JPEG or PNG compression.

 - Subscriber:
      This plugin decompresses a CompressedImage topic.

 - Provided by package: compressed_depth_image_transport
 - Publisher:
      This plugin publishes a compressed depth images using PNG compression.

 - Subscriber:
      This plugin decodes a compressed depth images.

*** Plugins are not built. ***
 - Provided by package: ffmpeg_image_transport
 - Publisher:
      This plugin encodes frame into ffmpeg compressed packets

 - Subscriber:
      This plugin decodes frame from ffmpeg compressed packets

 - Provided by package: image_transport
 - Publisher:
      This is the default publisher. It publishes the Image as-is on the base topic.

 - Subscriber:
      This is the default pass-through subscriber for topics of type sensor_msgs/Image.

 - Provided by package: theora_image_transport
 - Publisher:
      This plugin publishes a video packet stream encoded using Theora.

 - Subscriber:
      This plugin decodes a video packet stream encoded using Theora.

Publish and subscribe to images in JPEG or PNG compressed format, and support changing image quality at run time.

Compression format:

                    # This message contains a compressed image
Header header       # std_msgs/Header
string format       # Specifies the format of the data
                    #   Acceptable values:
                    #     jpeg, png
uint8[] data        # Compressed image buffer

An example:

root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw/compressed --noarr
  seq: 0
    secs: 1636880329
    nsecs: 751462995
  frame_id: "camera"
format: "bgr8; jpeg compressed bgr8"
data: "<array type: uint8, length: 43548>"

What to offer:

Publish plug-ins

Publish topic:

<base_topic>/compressed (sensor_msgs/CompressedImage)


  • <base_topic>/compressed/format (string, default: jpeg) Compression format to use, "jpeg" or "png".
  • <base_topic>/compressed/jpeg_quality (int, default: 80) JPEG quality percentile, in the range [1, 100]. Lower values trade image quality for space savings.
  • <base_topic>/compressed/png_level (int, default: 9) PNG compression level, in the range [1, 9]. Higher values trade computation time for space savings.

Subscription plug-in

Subscribe to topic:
<base_topic>/compressed (sensor_msgs/CompressedImage)

Provide depth image, which is mainly used in 3D field and image format with distance.

Publish as video stream
Video stream data in theora video coding format
theora is a video coding format, similar to H264
The corresponding sound coding technology is vorbis
The corresponding file encapsulation format is ogg

Publish topic
<base topic>/theora (theora_image_transport/Packet)


  • <base topic>/theora/optimize_for (int, default: Quality (1))
    Controls whether to use constant bitrate (CBR) encoding, aiming for ~theora/target_bitrate; or variable bitrate (VBR) encoding, aiming for ~theora/quality. Values are Bitrate (0) and Quality (1).
  • <base topic>/theora/target_bitrate (int, default: 800000)
    Target bitrate. Used if optimize_for is set to Bitrate.
  • <base topic>/theora/quality (int, default: 31)
    Encoding quality level, in the range [0, 63]. Used if optimize_for is set to Quality.
  • <base topic>/theora/keyframe_frequency (int, default: 64)
    Maximum distance between key frames. If set to 1, every frame is a keyframe.

Subscribe to topic:
<base topic>/theora (theora_image_transport/Packet)

A node: Ogg is provided_ Saver, you can save the package file in. ogv format and play it with other players.

Use ffmpe to convert images to h264 or h265 format, and support nvidia GPU hardware acceleration.

Convert between ROS Image messages and OpenCV images formats.

ros image format is based on sensor_msgs/Image and sensor_ Msgs / compressedimage. The format used in opencv is cv::Mat matrix. It needs to be converted before it can be used in OpenCV.

The pixel encoding formats supported in opencv are:

cv_bridge sometimes performs necessary pixel format conversion. The following string can be used to represent the target format:

  • mono8: CV_8UC1, grayscale image
  • mono16: CV_16UC1, 16-bit grayscale image
  • bgr8: CV_8UC3, color image with blue-green-red color order
  • rgb8: CV_8UC3, color image with red-green-blue color order
  • bgra8: CV_8UC4, BGR color image with an alpha channel
  • rgba8: CV_8UC4, RGB color image with an alpha channel

Among them, mono8 and bgr8 are common formats in opencv.


Initialization and stop
ros::init(argc, argv, "my_node_name");
ros::ok() is used to check whether the node is shut down


// Timer to periodically execute callback.
ros::Timer ros::NodeHandle::createTimer(ros::Duration period, <callback>, bool oneshot = false);
ros::Timer timer = nh.createTimer(ros::Duration(0.1), timerCallback); // Create timer

// Callback signature:
void callback(const ros::TimerEvent&);
// Where ros::TimerEvent:
//      ros::Time last_expected -- ideally, the time when the previous callback should be executed
//      ros::Time last_real -- the actual execution time of the previous callback
//      ros::Time current_expected -- ideally, the time when the current callback should be executed
//      ros::Time current_real -- in fact, the execution time of the current callback
//      ros::WallTime profile.last_duration -- the time taken for the execution of the previous callback


Single thread

// Single thread call
ros::init(argc, argv, "my_node");
ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe(...);

// Self implementation spin()
#include <ros/callback_queue.h>
ros::NodeHandle n;
while (ros::ok())

// Another form of single threaded call
ros::Rate r(10); // 10 hz
while (should_continue)
  ... do some work, publish some messages, etc. ...

// Implement spinOnce() yourself
#include <ros/callback_queue.h>


// Synchronous spin, blocking
ros::MultiThreadedSpinner spinner(4); // Use 4 threads
spinner.spin(); // spin() will not return until the node has been shutdown

// Asynchronous spin
ros::AsyncSpinner spinner(4); // Use 4 threads
ros::waitForShutdown(); // There will be a jam here

Call queue

// ros has a global default queue to which all callbacks will hang

// Create your own call queue, which has two levels:
#include <ros/callback_queue.h>
ros::CallbackQueue my_queue;

// Level 1: each subscribe(), advertise(), advertiseService()
// Use the Options structure to pass in your own queue

// Level 2: each NodeHandle is more commonly used. ros::spin() does not automatically trigger these callbacks, but needs to be triggered manually
ros::NodeHandle nh;

// Call all callbacks at once

// Call one callback at a time

Tags: ROS media

Posted on Sat, 04 Dec 2021 14:09:01 -0500 by stndrdsnz