Towards LLM Agents for Model-Based Engineering: A Case in Transformation Selection
This program is tentative and subject to change.
In Model-Based Engineering (MBE), practitioners frequently face the challenge of selecting appropriate tools from a large number of options. This requires both deep domain-specific knowledge and technical expertise. LLM-based agents are software components that depend on Large Language Models (LLMs) to autonomously select and apply software tools to perform specific tasks. Although LLMs have already been applied to support various MBE activities, considering LLM-based agents to autonomously assist users of MBE tools remains underexplored. This is particularly challenging in industrial MBE environments where only medium-size on-premise LLMs can be used due to company policies related to security or data privacy (for instance). To investigate the potential of LLM-based agents for MBE, we start with model-to-model transformation as a core MBE technique. Currently, off-the-shelf agents such as Microsoft Copilot can invoke a transformation engine (e.g. ATL) when the task is explicitly described. However, these agents struggle to select the correct transformation when they only have limited contextual information, especially when coupled with medium-size LLMs. To overcome this, we propose an approach based on complementary elements. First, we build a model transformation server and an LLM agent with dedicated tools for each transformation available on the server. Second, to enable the agent to efficiently select transformations, we rely on a tool retrieval technique based on a tool relevance score computed by an LLM. We evaluate our LLM agent on a model-to-model transformation dataset we also contribute to the community. Our comparative study shows that the newly proposed LLM agent responds more accurately to user instructions.
This program is tentative and subject to change.
Tue 7 OctDisplayed time zone: Eastern Time (US & Canada) change
08:30 - 10:00 | |||
08:30 5mDay opening | Day Opening SAM Conference | ||
08:35 28mTalk | Mitigating Hallucinations in SysML v2 Generation Using LLMs and a Tri-Layered Knowledge Graph Reasoning Framework SAM Conference Richard Qualis Florida Institute of Technology | ||
09:03 28mTalk | Towards LLM Agents for Model-Based Engineering: A Case in Transformation Selection SAM Conference Zakaria Hachm IMT Atlantique, LS2N (UMR CNRS 6004), Théo Le Calvar IMT Atlantique, LS2N (UMR CNRS 6004), Hugo Bruneliere IMT Atlantique, LS2N (UMR CNRS 6004), Massimo Tisi IMT Atlantique, LS2N (UMR CNRS 6004) | ||
09:31 29mTalk | Automated AADL Architecture Modeling : Leveraging Large Language Models for Safety-Critical Software SAM Conference Yaxin Zou Nanjing University of Aeronautics and Astronautics, Zhibin Yang Nanjing University of Aeronautics and Astronautics, Hao Liu Nanjing University of Aeronautics and Astronautics, Jiawei Liang Nanjing University of Aeronautics and Astronautics, Zonghua Gu Hofstra University, Yong Zhou Nanjing University of Aeronautics and Astronautics |