National Kaohsiung University of Hospitality and Tourism / Asst. Prof. Meng-Jun Hsu
Pain Points Solved
Service quality and complaint-handling capability in hospitality rely heavily on staff training, but traditional drills consume instructor time, are hard to repeat and cannot quantify soft skills such as tone, empathy and facial expression; consistency of service standards across branches and across borders is even harder to maintain. DrTrainius replaces live role-players with AI-simulated virtual customers covering front-desk check-in, housekeeping, restaurant ordering and a wide range of high-difficulty complaint scenarios (complaints, refunds, upgrade requests). It fuses three emotion-recognition engines: (1) Large Language Model for semantic analysis, (2) voice-print emotion analysis of tone and speaking rate to assess calmness and empathy, and (3) YOLO-based facial emotion detection of the trainee’s micro-expressions - producing quantifiable service-standard compliance scores and coaching suggestions in real time. The platform ingests employee manuals and service SOPs into a RAG knowledge base, becoming a 24x7 immersive training base that cuts training cost by over 60% and tracks each employee’s growth across professional knowledge, emotion management and complaint handling, enabling systematic standardised service training for the hospitality industry.
Technology Introduction
The system supports 40+ languages, voice interaction (Web Speech API), a 3D Digital Twin, and dual emotion-recognition engines (voice-print emotion analysis and YOLO-based facial emotion detection) for immersive situational training. Virtual customers can automatically switch roles (regular guests, VIPs, difficult customers, international travellers) for complaint-handling drills; each conversation is automatically scored against the uploaded employee manual and service SOP and exported as a PDF training report. The back office integrates Firebase Auth, multi-model management (Google Gemini + local Ollama models), Model Mix, data management, SaaS tiers (Trial / Starter / Pro / Enterprise), venue management, staff scoreboards, training logs, an analytics dashboard, voucher redemption and super-admin account management - a unified service-standard training solution from single-venue operators to hotel-chain groups.
Figure 1. Smart Dialog Main UI (Digital Twin + Voice)
Figure 2. Scenario Dialog Training Center
Figure 3. Local Model Self-Training Forge (Distill→SFT→Ollama)
Figure 4. Global Settings (Dual AI Engines + TTS)
Application Examples
The system has been applied to multiple training and education scenarios, including:
1. Tourist hotels: multilingual front-desk reception training and high-intensity complaint-handling simulations, with voice-print and facial-emotion recognition tracking staff emotional control.
2. Restaurant chains: systematic complaint-handling drills that build service standards and shorten new-hire onboarding.
3. Housekeeping / butler service: VIP service scripts and abnormal-incident handling drills.
4. Language learners: self-paced foreign-language speaking practice (40+ languages).
5. Hospitality and tourism departments: a simulation platform for practical student training and service-standard assessment.
Related Links
None
Industry-Academia / Tech Transfer Partner
None
Honors and Awards
None
Technical Contact
Administrative Assistant Hsin-Yi, Chu
Research and Development Office
National Kaohsiung University of Hospitality and Tourism
Tel: +886 7-8060505 ext. 16102
Email: hsichu@mail.nkuht.edu.tw