Evaluating the Performance of LLMs on Technical Language Processing tasks

In this paper we present the results of an evaluation study of the perfor-mance of LLMs on Technical Language Processing tasks. Humans are often confronted with tasks in which they have to gather information from dispar-ate sources and require making sense of large bodies of text. These tasks can be significantly complex for humans and often require deep study including rereading portions of a text. Towards simplifying the task of gathering in-formation we evaluated LLMs with chat interfaces for their ability to provide answers to standard questions that a human can be expected to answer based on their reading of a body of text. The body of text under study is Title 47 of the United States Code of Federal Regulations (CFR) which describes regula-tions for commercial telecommunications as governed by the Federal Com-munications Commission (FCC). This has been a body of text of interest be-cause our larger research concerns the issue of making sense of information related to Wireless Spectrum Governance and usage in an automated manner to support Dynamic Spectrum Access. The information concerning this wireless spectrum domain is found in many disparate sources, with Title 47 of the CFR being just one of many. Using a range of LLMs and providing the required CFR text as context we were able to quantify the performance of those LLMs on the specific task of answering the questions below.

Virtualization of Electromagnetic Operations (VEMO)

Today's operations in the spectrum occur across many disparate and unique devices. For example, a military or commercial maritime platform can have dozens of apertures used for various functions. Utilizing software-defined radios and dynamic analog front ends, we propose that future systems involving multiple applications for RF, virtualize operations to gain performance efficiencies. The concept of virtualization of electromagnetic operations (VEMO) is considered a means to enable flexible use of hardware and spectrum resources to achieve diverse mission requirements in military and commercial settings. This will result in more efficient use of resources, including size/weight/power and integration of objectives across systems that employ the electromagnetic spectrum.

Sparser Sparse Roadmaps

We present methods for offline generation of sparse roadmap spanners that result in graphs 79% smaller than existing approaches while returning solutions of equivalent path quality. Our method uses a hybrid approach to sampling that combines traditional graph discretization with random sampling. We present techniques that optimize the graph for the L1-norm metric function commonly used in joint-based robotic planning, purposefully choosing a $t$-stretch factor based on the geometry of the space, and removing redundant edges that do not contribute to the graph quality. A high-quality pre-processed sparse roadmap is then available for re-use across many different planning scenarios using standard repair and re-plan methods. Pre-computing the roadmap offline results in more deterministic solutions, reduces the memory requirements by affording complex rejection criteria, and increases the speed of planning in high-dimensional spaces allowing more complex problems to be solved such as multi-modal task planning. Our method is validated through simulated benchmarks against the SPARS2 algorithm. The source code is freely available online as an open source extension to OMPL.

Experience-Based Planning with Sparse Roadmap Spanners

We present an experienced-based planning framework called Thunder that learns to reduce computation time required to solve high-dimensional planning problems in varying environments. The approach is especially suited for large configuration spaces that include many invariant constraints, such as those found with whole body humanoid motion planning. Experiences are generated using probabilistic sampling and stored in a sparse roadmap spanner (SPARS), which provides asymptotically near-optimal coverage of the configuration space, making storing, retrieving, and repairing past experiences very efficient with respect to memory and time. The Thunder framework improves upon past experience-based planners by storing experiences in a graph rather than in individual paths, eliminating redundant information, providing more opportunities for path reuse, and providing a theoretical limit to the size of the experience graph. These properties also lead to improved handling of dynamically changing environments, reasoning about optimal paths, and reducing query resolution time. The approach is demonstrated on a 30 degrees of freedom humanoid robot and compared with the Lightning framework, an experience-based planner that uses individual paths to store past experiences. In environments with variable obstacles and stability constraints, experiments show that Thunder is on average an order of magnitude faster than Lightning and planning from scratch. Thunder also uses 98.8% less memory to store its experiences after 10,000 trials when compared to Lightning. Our framework is implemented and freely available in the Open Motion Planning Library.

Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study

Developing robot agnostic software frameworks involves synthesizing the disparate fields of robotic theory and software engineering while simultaneously accounting for a large variability in hardware designs and control paradigms. As the capabilities of robotic software frameworks increase, the setup difficulty and learning curve for new users also increase. If the entry barriers for configuring and using the software on robots is too high, even the most powerful of frameworks are useless. A growing need exists in robotic software engineering to aid users in getting started with, and customizing, the software framework as necessary for particular robotic applications. In this paper a case study is presented for the best practices found for lowering the barrier of entry in the MoveIt! framework, an open-source tool for mobile manipulation in ROS, that allows users to 1) quickly get basic motion planning functionality with minimal initial setup, 2) automate its configuration and optimization, and 3) easily customize its components. A graphical interface that assists the user in configuring MoveIt! is the cornerstone of our approach, coupled with the use of an existing standardized robot model for input, automatically generated robot-specific configuration files, and a plugin-based architecture for extensibility. These best practices are summarized into a set of barrier to entry design principles applicable to other robotic software. The approaches for lowering the entry barrier are evaluated by usage statistics, a user survey, and compared against our design objectives for their effectiveness to users.