Background: This study presents a locally deployed large language model (LLM)-powered group chat system integrating multi-level conversation summarization, private AI interaction, and context-aware retrieval through retrieval-augmented generation (RAG). The system addresses current limitations in AI-assisted messaging platforms, which typically depend on cloud services and lack features such as privacy, adaptive summarization, and contextual recall.
Methods: The framework was implemented on a self-contained MERN architecture and powered by LLaMA 3.1 (8B) via the Ollama framework. Through a command-based interface (@aria), users can trigger summarization, retrieval, or private queries directly within group chats. The methodology combines prompt engineering, FAISS-based vector retrieval, and socket-level message routing to ensure contextual grounding and low-latency response.
Results: Evaluation showed average response times of 69.09 ms (TTFT) and 6,076.47 ms (TTFR) for private queries, 8,489.02 ms for summarization, and 215.5 ms (TTFT) and 691.33 ms (TTFR) for RAG. Summarization quality improved with granularity (ROUGE-1 = 0.285–0.434), and human evaluations confirmed balanced accuracy and readability. The RAG feature achieved 90% factual correctness in contextual queries.
Conclusions: These findings demonstrate that localized LLM frameworks can deliver intelligent, privacy-preserving, and context-aware assistance comparable to cloud-based systems, forming a foundation for scalable, multimodal, and crossplatform conversational AI.
Keywords: Large Language Model (LLM); Retrieval-Augmented Generation; Multi-level Summarization; Privacy-Preserving Chat System; Context-Aware Conversational AI
|