Assessing LLMs Capabilities for Industrial Knowledge Management: Performance Analysis of Embedded Knowledge and Retrieval Augmented Generation Approach

Recent advancements in Generative Artificial Intelligence (GenAI), driven by the development of Large Language Models (LLMs), have created opportunities for innovative solutions across multiple sectors, including manufacturing. However, deploying LLMs in industrial settings presents significant challenges due to their general-purpose nature, often resulting in inaccurate or irrelevant responses when applied to domain-specific tasks. This paper explores the integration of Retrieval-Augmented Generation (RAG) to improve the performance of LLMs in industrial environments. The study investigates the effectiveness of five LLMs (GPT-4o, Mixtral-8 × 22B-Instruct-v0.1, Llama-3.3-70B-Instruct, DeepSeek-V3, Qwen2.5-72B-Instruct) in processing industrial technical documentation. The models’ performance was evaluated with and without RAG augmentation, using a dataset of 100 verified frequently asked questions (FAQs) from an industrial 3D printer manual. The evaluation employed three approaches: expert-based evaluation, embedding-based evaluation, and LLM-based comparative assessment. Additionally, the study examines whether the results from the embedding-based and LLM-based evaluations align with expert judgments. Findings indicate that RAG significantly enhances the accuracy and relevance of LLM-generated responses, demonstrating its potential as a scalable and flexible approach for adapting LLMs to specialized manufacturing contexts. However, expert-based evaluation remains essential in assessing LLM performance, as automated evaluation methods alone may not fully capture domain-specific nuances. This research contributes to a deeper understanding of LLM implementation in industrial applications.