1.5.6. TCP over WAN with Discovery Server¶

Background
Overview
Prerequisites
Set up the docker networks
XML configuration files
- Server side
- Client side
Create the Docker compose file
Run the example
Clean workspace
TCP over real WAN with Discovery Server

1.5.6.1. Background¶

In this tutorial, it is explained how to deploy a TCP communication over WAN using Discovery Server as discovery mechanism. The following diagram describes de main idea of the deployment tutorial.

../../../../../_images/ds_wan_tcp_simple.svg

TCP communication over WAN with Discovery Server¶

1.5.6.1.1. TCP: Transmission Control Protocol¶

The Transmission Control Protocol is a Internet Protocol which provides mainly reliability in the communication process. As it is connection-oriented, this protocol has several features that ensures the delivery, order and error check of the packages. Despite the fact that the latency is higher than other Internet Protocols such as UDP, its use has several advantages in particular scenarios where reliability has greater importance than the latency cost.

1.5.6.1.2. WAN: Wide Area Network¶

A Wide Area Network is a telecommunication network extended over a large geographic area. It usually involves a large number of nodes and redundancy, to ensure the reliability of the network. The Internet could be considered as a WAN itself.

1.5.6.1.3. Discovery Server¶

The Discovery Server is a Fast DDS enabled feature that procures an alternative discovery mechanism to the default ROS 2 discovery mechanism, Simple Discovery Protocol (SDP), which is served by the DDS implementations according to the DDS specification. Whereas SDP (right figure) provides automatic out-of-the-box discovery by leveraging multicast, the ROS 2 Discovery Server (left figure) provides a centralized hub for managing discovery which drastically reduces network bandwidth utilization when compared to SDP, since the nodes, publishers, and subscribers, only discover those remote ROS 2 entities with which they need to communicate with, as opposed to the SDP model where everyone knows each other.

A server is a context to which the clients (and maybe other servers) send their discovery information. The role of the server is to re-distribute the clients (and servers) discovery information to their known clients and servers.

A client is a context that connects to one or more servers from which it receives only the discovery information they require to establish communication with matching endpoints.

1.5.6.2. Overview¶

This tutorial will use ROS 2 demo_nodes_cpp talker and listener applications to establish the communication between clients through the server. Each node would be deployed in a docker container, in different networks.

The discovery server, would be deployed also in its own docker container, but it will be part of two networks: the WAN and the same network as the talker node. This setup allows the discovery server to perform routing tasks as a regular router does in a LAN (having a private IP which would be in the talker LAN IP, and a public IP, which would be in the WAN IP). Additionally, the same routing element is required in the listener LAN. A router container is included as the intermediary between the WAN and the listener node network.

Within these defined scenario, the following diagram describes the network setup for deploying the simulation.

../../../../../_images/ds_wan_tcp_complex.svg

Network setup example to simulate TCP communication over WAN with Discovery Server¶

The expected behavior is that both talker and listener are able to connect to the discovery server, discover each other, and send and receive (respectively) the HelloWorld example messages over the WAN.

Important

All the communication, including discovery phase, would be performed using TCP. See the Fast DDS discovery phases documentation for further information.

Note

Docker performs a network configuration to isolate each of the docker networks. This tutorial would update some of the iptables configuration set automatically by docker in order to simulate properly the WAN scenario.

1.5.6.3. Prerequisites¶

First of all, make sure that Vulcanexus jazzy is installed. The docker installation is required for this tutorial (see Docker installation), together with the eProsima Vulcanexus docker image. The Vulcanexus jazzy image can be downloaded by running:

docker pull eprosima/vulcanexus:jazzy

In addition, docker compose is used to simplify the example deployment, and iptables is required to update the network configuration. Follow docker compose installation guide, and install the iptables dependency by running:

sudo apt install iptables

Note

It is highly recommended to complete the Configuring Fast-DDS QoS via XML profiles tutorial to learn how to configure ROS 2 via XML configuration files.

1.5.6.4. Set up the docker networks¶

Set the specific interface pools by running the following commands.

docker network create --subnet=10.1.0.0/16 talker_net
docker network create --subnet=10.2.0.0/16 listener_net
docker network create --subnet=10.3.0.0/16 wan_net

The three networks would have been created, all of them isolated from the others. In order to enable the communication between networks, it is mandatory to update the system iptables.

sudo iptables -I DOCKER-ISOLATION-STAGE-2 -i talker_net -o listener_net -j ACCEPT
sudo iptables -I DOCKER-ISOLATION-STAGE-2 -o talker_net -i listener_net -j ACCEPT
sudo iptables -I DOCKER-ISOLATION-STAGE-2 -i talker_net -o wan_net -j ACCEPT
sudo iptables -I DOCKER-ISOLATION-STAGE-2 -o talker_net -i wan_net -j ACCEPT
sudo iptables -I DOCKER-ISOLATION-STAGE-2 -i listener_net -o wan_net -j ACCEPT
sudo iptables -I DOCKER-ISOLATION-STAGE-2 -o listener_net -i wan_net -j ACCEPT

This set of commands is enabling both input (-i) and output (-o) tables to allow both sides communication between the three networks.

1.5.6.5. XML configuration files¶

It is mandatory to set the discovery server and the client nodes with the following configuration options in order to enable the TCP communication, regardless of whether the deployment is over Docker networks or WAN.

Both XML configuration files will be later used in the docker compose instructions to perform the containers deployment.

1.5.6.5.1. Server side¶

The following XML configuration describes the discovery server TCP configuration required. Create a workspace to run the tutorial, and include an XML configuration file named server_configuration.xml with the below code.

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles" >

    <!-- TCP transport descriptor -->
    <transport_descriptors>
        <transport_descriptor>
            <!-- Transport descriptor identifier -->
            <transport_id>TCP_ds_transport</transport_id>

            <!-- TCP transport -->
            <type>TCPv4</type>

            <!-- Discovery server listening (physical) port -->
            <listening_ports>
                <port>10111</port>
            </listening_ports>

            <!-- Discovery server WAN address -->
            <wan_addr>10.1.1.1</wan_addr>
        </transport_descriptor>
    </transport_descriptors>

    <!-- Participant profile -->
    <participant profile_name="TCP_discovery_server_profile" is_default_profile="true">
        <rtps>
            <!-- Use declared TCP transport descriptor -->
            <userTransports>
                <transport_id>TCP_ds_transport</transport_id>
            </userTransports>

            <!-- Do not use default builtin transports -->
            <useBuiltinTransports>false</useBuiltinTransports>

            <!-- Set server's GUID prefix -->
            <prefix>44.53.00.5f.45.50.52.4f.53.49.4d.41</prefix>

            <builtin>
                <!-- Discovery server configuration -->
                <discovery_config>
                    <!-- Node kind: SERVER -->
                    <discoveryProtocol>SERVER</discoveryProtocol>
                </discovery_config>

                 <!-- Set server's listening locator for discovery phase -->
                <metatrafficUnicastLocatorList>
                    <locator>
                        <!-- TCP Discovery server listening locator -->
                        <tcpv4>
                            <address>10.1.1.1</address>
                            <port>10111</port>
                            <physical_port>10111</physical_port>
                        </tcpv4>
                    </locator>
                </metatrafficUnicastLocatorList>
            </builtin>
        </rtps>
    </participant>
</profiles>

Note that in the discovery configuration section, the profile is described as server, with a specific GUID prefix, and listening locator. This listening locator is configured with the IP 10.1.1.1, which belongs to the talker node LAN, and its physical port is included in the transport descriptor as a known listening port. This, along setting the wan address with the previously mentioned IP address, and setting the transport descriptor and locator types as TCPv4, is crucial to ensure the TCP communication.

1.5.6.5.2. Client side¶

Include the following XML configuration in the workspace, and name the file as talker_configuration.xml. This former configuration describes the talker node configuration for the discovery phase, and transport layer.

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles" >

    <!-- TCP transport descriptor -->
    <transport_descriptors>
        <transport_descriptor>
            <!-- Transport descriptor identifier -->
            <transport_id>TCP_talker_transport</transport_id>

            <!-- TCP transport -->
            <type>TCPv4</type>

            <!-- Talker listening (physical) port -->
            <listening_ports>
                <port>10102</port>
            </listening_ports>

            <!-- Talker WAN address -->
            <wan_addr>10.1.0.2</wan_addr>
        </transport_descriptor>
    </transport_descriptors>

    <!-- Participant profile -->
    <participant profile_name="TCP_talker_profile" is_default_profile="true">
        <rtps>
            <!-- Use declared TCP transport descriptor -->
            <userTransports>
                <transport_id>TCP_talker_transport</transport_id>
            </userTransports>

            <!-- Do not use default builtin transports -->
            <useBuiltinTransports>false</useBuiltinTransports>

            <builtin>
                <!-- Discovery phase configuration -->
                <discovery_config>

                    <!-- Node kind: CLIENT -->
                    <discoveryProtocol>CLIENT</discoveryProtocol>

                    <!-- Discovery Server configuration -->
                    <discoveryServersList>
                        <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
                            <metatrafficUnicastLocatorList>
                                <!-- Set server's locator for discovery phase -->
                                <locator>
                                    <tcpv4>
                                        <address>10.1.1.1</address>
                                        <port>10111</port>
                                        <physical_port>10111</physical_port>
                                    </tcpv4>
                                </locator>
                            </metatrafficUnicastLocatorList>
                        </RemoteServer>
                    </discoveryServersList>
                </discovery_config>
            </builtin>
        </rtps>
    </participant>
</profiles>

It involves setting the profile as client in the discovery configuration section, and adding the discovery server GUID prefix and listening locator. The talker TCP transport descriptor must be configured with the wan address and physical port described. Note that the listening port configured must be different from the set in the discovery server.

Then, also include the following listener XML configuration in the workspace, and name the file as listener_configuration.xml.

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles" >

    <!-- TCP transport descriptor -->
    <transport_descriptors>
        <transport_descriptor>
            <!-- Transport descriptor identifier -->
            <transport_id>TCP_listener_transport</transport_id>

            <!-- TCP transport -->
            <type>TCPv4</type>

            <!-- Listener listening (physical) port -->
            <listening_ports>
                <port>10202</port>
            </listening_ports>

            <!-- Listener WAN address -->
            <wan_addr>10.2.0.2</wan_addr>
        </transport_descriptor>
    </transport_descriptors>

    <!-- Participant profile -->
    <participant profile_name="TCP_listener_profile" is_default_profile="true">
        <rtps>
            <!-- Use declared TCP transport descriptor -->
            <userTransports>
                <transport_id>TCP_listener_transport</transport_id>
            </userTransports>

            <!-- Do not use default builtin transports -->
            <useBuiltinTransports>false</useBuiltinTransports>

            <builtin>
                <!-- Discovery phase configuration -->
                <discovery_config>

                    <!-- Node kind: CLIENT -->
                    <discoveryProtocol>CLIENT</discoveryProtocol>

                    <!-- Discovery Server configuration -->
                    <discoveryServersList>
                        <RemoteServer prefix="44.53.00.5f.45.50.52.4f.53.49.4d.41">
                            <metatrafficUnicastLocatorList>
                                <!-- Set server's locator for discovery phase -->
                                <locator>
                                    <tcpv4>
                                        <address>10.1.1.1</address>
                                        <port>10111</port>
                                        <physical_port>10111</physical_port>
                                    </tcpv4>
                                </locator>
                            </metatrafficUnicastLocatorList>
                        </RemoteServer>
                    </discoveryServersList>
                </discovery_config>
            </builtin>
        </rtps>
    </participant>
</profiles>

Note that the listener discovery server configuration is exactly the same as talker discovery server configuration, but the WAN IP address and the listening port set in the transport descriptor configuration are different according to each LAN.

1.5.6.6. Create the Docker compose file¶

Once the XML configuration files have been included in the workspace, create a new file named Dockerfile and paste the following code. It contains the required commands to assemble a docker image based on Vulcanexus jazzy. That includes some dependencies, and the recently created XML configuration files.

FROM eprosima/vulcanexus:jazzy

RUN apt update && DEBIAN_FRONTEND=noninteractive apt install -y ros-jazzy-demo-nodes-cpp net-tools iptables

COPY listener_configuration.xml server_configuration.xml talker_configuration.xml ./

Finally, the compose.yml is where all the containers and their configuration are described:

fast_dds_discovery_server: the container is included in both created talker_net and wan_net. The IP addresses has been manually set to 10.1.1.1 in the talker_net, and 10.3.1.1 in the wan_net. This container’s default gateway is redirected to the router container. The iptables has been configured to redirect any traffic from any network and interface.
ros_listener: the container is included in the created listener_net. The IP address has been manually set to 10.2.0.2, and the environment variable FASTRTPS_DEFAULT_PROFILES_FILE is set with the XML configuration listener_configuration.xml. This container’s default gateway is redirected to the router container.
ros_talker: the container is included in the created talker_net. The IP address has been manually set to 10.1.0.2, and the environment variable FASTRTPS_DEFAULT_PROFILES_FILE is set with the XML configuration talker_configuration.xml. This container’s default gateway is redirected to the discovery server, which would behave as a router too.
router: the container is included in both listener_net and wan_net networks. The IP addresses has been manually set to 10.2.1.1 in the listener_net, and 10.3.2.1 in the wan_net. This container’s default gateway is redirected to the discovery server. The iptables has been configured to redirect traffic from any network and interface.

Please, include the following compose.yml file in the workspace.

version: "3"

networks:
  listener_net:
    name: listener_net
    external: true
  talker_net:
    name: talker_net
    external: true
  wan_net:
    name: wan_net
    external: true

services:
  fast_dds_discovery_server:
    build:
      context: .
    command: sh -c "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE &&
      iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE &&
      iptables -A FORWARD -i eth0 -j ACCEPT &&
      iptables -A FORWARD -i eth1 -j ACCEPT &&
      iptables -A INPUT -i eth0 -j ACCEPT &&
      iptables -A INPUT -i eth1 -j ACCEPT &&
      iptables -A OUTPUT -j ACCEPT &&
      route del default &&
      route add default gw 10.3.2.1 eth1 &&
      fastdds discovery -x server_configuration.xml"
    privileged: true
    cap_add:
      - ALL
    volumes:
      - /tmp/.X11-unix:/tmp/.X11-unix:rw
    environment:
      - DISPLAY
    sysctls:
      - net.ipv4.conf.all.forwarding=1
      - net.ipv4.ip_forward=1

    networks:
      talker_net:
        ipv4_address: 10.1.1.1
      wan_net:
        ipv4_address: 10.3.1.1

  ros_listener:
    build:
      context: .
    command: >-
      sh -c "route del default &&
      route add default gw 10.2.1.1 eth0 &&
      ros2 run demo_nodes_cpp listener"
    cap_add:
      - ALL
    privileged: true
    volumes:
      - /tmp/.X11-unix:/tmp/.X11-unix:rw
    environment:
      DISPLAY: ":1"
      FASTRTPS_DEFAULT_PROFILES_FILE: "listener_configuration.xml"
    networks:
      listener_net:
        ipv4_address: 10.2.0.2

  ros_talker:
    build:
      context: .
    command: >-
      sh -c "route del default &&
      route add default gw 10.1.1.1 eth0 &&
      ros2 run demo_nodes_cpp talker"
    privileged: true
    volumes:
      - /tmp/.X11-unix:/tmp/.X11-unix:rw
    environment:
      DISPLAY: ":1"
      FASTRTPS_DEFAULT_PROFILES_FILE: "talker_configuration.xml"
    cap_add:
      - ALL
    networks:
      talker_net:
        ipv4_address: 10.1.0.2

  router:
    build:
      context: .
    command: sh -c "iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE &&
      iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE &&
      iptables -A FORWARD -i eth0 -j ACCEPT &&
      iptables -A FORWARD -i eth1 -j ACCEPT &&
      iptables -A INPUT -i eth0 -j ACCEPT &&
      iptables -A INPUT -i eth1 -j ACCEPT &&
      iptables -A OUTPUT -j ACCEPT &&
      route del default &&
      route add default gw 10.3.1.1 eth1 &&
      sleep infinity"
    privileged: true
    cap_add:
      - ALL
    volumes:
      - /tmp/.X11-unix:/tmp/.X11-unix:rw
    environment:
      - DISPLAY
    sysctls:
      - net.ipv4.conf.all.forwarding=1
      - net.ipv4.ip_forward=1

    networks:
      listener_net:
        ipv4_address: 10.2.1.1
      wan_net:
        ipv4_address: 10.3.2.1

1.5.6.7. Run the example¶

Make sure that the workspace has been set with all the previous files, and the docker networks and iptables have been set too.

├ <workspace>
·  ├ compose.yml
·  ├ Dockerfile
·  ├ listener_configuration.xml
·  ├ server_configuration.xml
·  └ talker_configuration.xml

As explained in the Overview, the expected behavior is that both talker and listener are able to connect to the discovery server, discover each other, and send and receive (respectively) the HelloWorld example messages over the WAN simulation using TCP.

Run the example:

cd <workspace>
docker compose -f compose.yml up --build

1.5.6.8. Clean workspace¶

Stop the example by pressing Ctrl + C, and stop the containers by running:

docker compose -f compose.yml down

The docker networks, containers and images can be removed using the docker prune argument, but using the rm argument plus the identifiers would remove only the ones created for this tutorial:

docker network rm listener_net talker_net wan_net
docker container rm <workspace_name>_fast_dds_discovery_server_1 <workspace_name>_router_1 <workspace_name>_ros_listener_1 <workspace_name>_ros_talker_1
docker image rm <workspace_name>_fast_dds_discovery_server:latest <workspace_name>_router:latest <workspace_name>_ros_listener:latest <workspace_name>_ros_talker:latest

Note

The iptables configuration would be removed automatically each time the system gets rebooted.

1.5.6.9. TCP over real WAN with Discovery Server¶

The configuration of the docker networks and the iptables has been performed to simulate the WAN scenario locally. The requirements to achieve TCP communication over WAN with Discovery Server in a real deployment are launching the three elements of the communication (talker, listener and discovery server) with their corresponding XML configurations applied, and setting the proper firewall or router configuration rules.

../../../../../_images/ds_wan_tcp_scenarios.svg

Possible scenarios of TCP communication over WAN with Discovery Server¶

Regardless of whether the discovery server belongs, or not, to one of the clients’ LANs, it is necessary to configure one port forwarding rule for the discovery server, and another rule per every pair of clients communicating over the WAN, on either one of the sides.

If possible, deploy the different elements of the tutorial in different docker containers over WAN, following one of the possibilities displayed in the image above. In order to address the port forwarding configuration, see the Configure transversal NAT on the network router section from WAN communication over TCP Fast DDS Router tutorial for further information.

# run the server docker image with the XML configuration
docker run -td --name discovery_server --net host eprosima/vulcanexus:jazzy
docker cp <workspace>/server_configuration.xml discovery_server:/server_configuration.xml
docker exec -it discovery_server bash

# run the discovery server
source /opt/vulcanexus/jazzy/setup.bash
fastdds discovery -x server_configuration.xml

# run the listener docker image with the XML configuration
docker run -td --name ros_listener --net host eprosima/vulcanexus:jazzy
docker cp <workspace>/listener_configuration.xml ros_listener:/listener_configuration.xml
docker exec -it ros_listener bash

# run the listener client
source /opt/ros/jazzy/setup.bash
export FASTRTPS_DEFAULT_PROFILES_FILE="listener_configuration.xml"
ros2 run demo_nodes_cpp listener

# run the talker docker image with the XML configuration
docker run -td --name ros_talker --net host eprosima/vulcanexus:jazzy
docker cp <workspace>/talker_configuration.xml ros_talker:/talker_configuration.xml
docker exec -it ros_talker bash

# run the talker client
source /opt/ros/jazzy/setup.bash
export FASTRTPS_DEFAULT_PROFILES_FILE="talker_configuration.xml"
ros2 run demo_nodes_cpp talker

Make sure that the discovery server public IP has been properly set in all sections of the XML configuration files (in both listener_configuration.xml, server_configuration.xml and talker_configuration.xml). The host public IP address can be obtained by running:

wget -qO- http://ipecho.net/plain | xargs echo

It is also required to set the WAN address for each client in the XML transport descriptor configuration. If any of them are hosted in the same LAN as the discovery server, then make sure that the XML transport descriptor listening ports configured for each of them are different. Also, take into account that the WAN IP address would be the same for both contexts (client and server).