Bringing Generative AI to Robotics at ROSCon in Denmark

In the historic city of Odense, Denmark, which serves as a center for automation, a remarkable announcement unfolded at the ROSCon event. NVIDIA, in collaboration with several partners within its robotics ecosystem, unveiled a series of groundbreaking generative AI tools, simulation capabilities, and perception workflows tailored for developers working with the Robot Operating System (ROS). These innovations aim to enhance the capabilities of robots, enabling them to understand their environment better, communicate more naturally with humans, and make autonomous decisions.

Generative AI Revolutionizes the ROS Community

A significant highlight from the conference was the introduction of new generative AI nodes and workflows specifically designed for ROS developers who are deploying their applications on the NVIDIA Jetson platform. Generative AI is a cutting-edge technology that allows robots to perceive and comprehend the context surrounding them, enhancing their interaction with humans and improving their decision-making processes.

One of the standout tools introduced is ReMEmbR, which is built on ROS 2. This tool leverages generative AI to enhance the reasoning and actions of robots. It integrates large language models (LLMs), vision language models (VLMs), and retrieval-augmented generation to help robots build long-term semantic memories. This allows them to navigate and interact with their environments more effectively.

Moreover, the speech recognition capabilities are powered by the WhisperTRT ROS 2 node. This node utilizes NVIDIA TensorRT to optimize OpenAI’s Whisper model, ensuring low-latency inference on NVIDIA Jetson platforms. The result is a more responsive interaction between humans and robots.

The ROS 2 robots with voice control project employs the NVIDIA Riva ASR-TTS service, enabling robots to understand and respond to verbal commands. Additionally, the NASA Jet Propulsion Laboratory showcased an AI-powered agent named ROSA running on their Nebula-SPOT robot and the NVIDIA Nova Carter robot within NVIDIA Isaac Sim.

At the event, Canonical demonstrated NanoOWL, a zero-shot object detection model operating on the NVIDIA Jetson Orin Nano. This model enables robots to identify an extensive array of objects in real-time, without relying on pre-defined categories.

Developers eager to explore these advancements can start with the ROS 2 Nodes for Generative AI, which introduce NVIDIA Jetson-optimized LLMs and VLMs to elevate robotic capabilities.

Improving ROS Workflows with a ‘Sim-First’ Approach

Simulation plays a critical role in safely testing and validating AI-enabled robots before they are deployed in the real world. NVIDIA Isaac Sim, a robotics simulation platform developed on OpenUSD, provides a virtual environment for ROS developers to test their robots by seamlessly connecting them to their ROS packages. For those new to the field, a Beginner’s Guide to ROS 2 Workflows With Isaac Sim is now available. This guide details the complete workflow for robot simulation and testing.

Foxglove, a participant in the NVIDIA Inception program for startups, showcased an integration that aids developers in visualizing and debugging simulation data in real-time. This integration is built on Isaac Sim using Foxglove’s custom extension.

Exciting Developments in Isaac ROS 3.2

NVIDIA Isaac ROS is built on the open-source ROS 2 software framework. It offers a collection of accelerated computing packages and AI models for robotics development. The anticipated 3.2 release brings enhancements to robot perception, manipulation, and environment mapping.

Notable improvements include advancements to the NVIDIA Isaac Manipulator, with new reference workflows that integrate FoundationPose and cuMotion. These enhancements accelerate the development of pick-and-place and object-following pipelines in robotics.

In addition, the NVIDIA Isaac Perceptor incorporates a new visual SLAM reference workflow, enhanced multi-camera detection, and 3D reconstruction. These features boost an autonomous mobile robot’s (AMR) environmental awareness and performance in dynamic settings such as warehouses.

Robotics Partners Embrace NVIDIA Isaac

Robotics companies are increasingly integrating NVIDIA Isaac’s accelerated libraries and AI models into their platforms.

Universal Robots, part of Teradyne Robotics, unveiled a new AI Accelerator toolkit to foster the development of AI-powered collaborative robot (cobot) applications.

Miso Robotics is leveraging Isaac ROS to enhance its AI-powered robotic french fry-making Flippy Fry Station, driving improvements in efficiency and accuracy in food service automation.

Wheel.me is collaborating with RGo Robotics and NVIDIA to create a production-ready AMR using Isaac Perceptor.

Main Street Autonomy is utilizing Isaac Perceptor to streamline sensor calibration.

Orbbec announced its Perceptor Developer Kit, providing an out-of-the-box AMR solution for Isaac Perceptor.

LIPS Corporation introduced a multi-camera perception devkit to improve AMR navigation.

Canonical showcased a fully certified Ubuntu environment for ROS developers, offering long-term support directly from the box.

Engaging with the Community at ROSCon

ROS community members and partners, including Canonical, Ekumen, Foxglove, Intrinsic, Open Navigation, Siemens, and Teradyne Robotics, gathered in Denmark to present workshops, talks, booth demos, and interactive sessions. Highlights included:

• A “Nav2 User Meetup” Birds of a Feather session with Steve Macenski from Open Navigation LLC
• A session on “ROS in Large-Scale Factory Automation” with Michael Gentner from BMW AG and Carsten Braunroth from Siemens AG
• “Integrating AI in Robot Manipulation Workflows” Birds of a Feather session with Kalyan Vadrevu from NVIDIA
• “Accelerating Robot Learning at Scale in Simulation” Birds of a Feather session with Markus Wuensch from NVIDIA
• A session on “On Use of Nav2 Docking” with Open Navigation’s Macenski

  Moreover, Teradyne Robotics and NVIDIA co-hosted a lunch and evening reception on Oct. 22 in Odense, Denmark.

  The Open Source Robotics Foundation (OSRF) organizes ROSCon. NVIDIA actively supports Open Robotics, the organization managing OSRF and its initiatives.

  For the latest information and updates, visit the ROSCon page.

For more Information, Refer to this article.