Automating and Improving Customer Support with Open-Source LLMs 🤗

This project demonstrates a practical application of Natural Language Processing (NLP) and Large Language Models (LLMs) to enhance customer support operations. It involves performing sentiment analysis on customer reviews, generating empathetic AI-driven responses to negative feedback, and extracting actionable insights from the review data.

The primary tools used are the Hugging Face transformers library for sentiment analysis and the hugchat library (an unofficial API) for interacting with chat models hosted on Hugging Face Chat.

Project Highlights

• Sentiment Analysis: Classifies customer reviews (from a simulated Booking.com dataset) into positive or negative sentiments using a pre-trained transformer model.
• Empathetic Response Generation: Leverages an LLM via hugchat to draft personalized and empathetic replies to negative customer reviews.
• Insight Extraction: Uses the LLM to summarize common themes and pain points from a collection of negative reviews, providing valuable feedback for service improvement.
• Prompt Engineering: Showcases techniques for crafting effective prompts to guide LLM behavior for specific tasks.
• Practical Application: Provides a blueprint for automating aspects of customer interaction and feedback analysis.

Technologies & Libraries Used

• Python 3
• Hugging Face transformers: For state-of-the-art sentiment analysis.
• hugchat: An unofficial Python wrapper to interact with Hugging Face Chat models.
• pandas: For data manipulation and analysis.
• BeautifulSoup4: For parsing HTML content (from a pre-scraped review file).
• scikit-learn: For model evaluation metrics (classification report).
• NumPy: For numerical operations.
• Google Colab: For an interactive, cloud-based development environment.

Setup and Usage

1. Clone the Repository (Optional):

   git clone https://github.com/codebywiam/sentiment-analysis-huggingface.git
   cd sentiment-analysis-huggingface

2. Open in Google Colab: Click the "Open In Colab" badge above, or manually upload the customer_support_llm_automation.ipynb notebook to Google Colab.

3. Hugging Face Credentials for hugchat: This project uses hugchat which requires your Hugging Face account credentials.

   • Ensure you have a Hugging Face account: huggingface.co/join
   • In the Colab notebook, you'll need to set up secrets:
     1. Click the Key icon (Secrets) on the left sidebar in Colab.
     2. Add two new secrets:
        • HF_EMAIL: Your Hugging Face account email.
        • HF_PASS: Your Hugging Face account password.
     3. Ensure the "Notebook access" toggle is enabled for both secrets.
   • Security Note: Never hardcode your credentials directly in the notebook if you plan to share it. Colab secrets provide a safer way.

4. Run the Notebook Cells:

   • Manually upload HTML.txt to the content folder in Google Colab . Execute the cells in the notebook sequentially. The notebook will:
   • Install necessary dependencies.
   • Load and process customer review data from a provided HTML.txt file.
   • Perform sentiment analysis.
   • (If hugchat login is successful) Generate empathetic responses to negative reviews.
   • (If hugchat login is successful) Extract insights from reviews.

Project Structure

• Customer_Support_LLM_Automation.ipynb: The main Jupyter/Colab notebook containing all the code and explanations.
• HTML.txt: (Downloaded by the notebook) A file containing pre-scraped HTML of customer reviews.
• processed_hotel_reviews.csv: (Generated by the notebook) CSV file with parsed reviews and their ground truth sentiment.
• negative_reviews_with_replies.csv: (Generated by the notebook, if hugchat runs successfully) CSV file containing negative reviews and their AI-generated replies.
• README.md: This file.

Results & Outputs

Sentiment Analysis

Used the distilbert-base-uncased-finetuned-sst-2-english model for classifying customer reviews.

Examples:

• "Great hotel, in a great location..." → POSITIVE (Score: 0.9999)

• "Very poor reception experience today." → NEGATIVE (Score: 0.9998)

• "The hotel was mediocre at best..." → NEGATIVE (Score: 0.9978)

Dataset Summary:

• Total Reviews Analyzed: 1,422

• Positive Reviews: 717

• Negative Reviews: 705

Model Performance:

Accuracy: 86.22%

Classification Report:
              precision    recall  f1-score   support
Negative (0)       0.89      0.83      0.86       705
Positive (1)       0.84      0.90      0.87       717

AI-Generated Empathetic Responses

Example Negative Review:

"The door was open to the room when I got there.. the key keep deactivating.. someone pulled the smoke alarm."

AI Response:

"I am so sorry to hear that your stay with us was not up to our usual standards... I would like to assure you that we take incidents like this very seriously and will be investigating the matter further..."

Other AI-Generated Responses:

Review Snippet	AI Response Summary
"no refrigerator..."	Acknowledged inconvenience, took note of suggestion
"hallways were musty..."	Apologized and promised facility improvements
"no microwave, shared bathroom..."	Addressed lack of amenities and room design concerns
"spa was closed..."	Noted disappointment and feedback for reopening
"shower not working..."	Committed to maintenance and fixing the issue

All responses were compiled and saved to negative_reviews_with_replies.csv.

Insight Extraction

Using 20 sampled negative reviews, the LLM extracted the following Top 5 Areas for Improvement:

1. Room Maintenance & Cleanliness : Common complaints about dirty bathrooms, broken furniture, and malfunctioning amenities.

2. Front Desk & Staff Service : Reports of rudeness and unhelpful interactions suggest a need for better training.

3. Room Quality & Amenities : Issues with TVs, shower doors, and lack of refrigerators point to an amenities upgrade.

4. Smoking Policy & Room Allocation : Concerns about being assigned smoking rooms require better room assignment practices.

5. Communication & Information Sharing : Guests lacked access to clear hotel and local area information.

These insights are specific, measurable, and actionable—ideal for improving guest satisfaction and operational quality.

Conclusion

This project demonstrates how open-source LLMs can be used effectively for:

• Automating sentiment analysis,

• Crafting empathetic responses to customer concerns,

• Extracting data-driven insights for strategic decision-making in hospitality or similar domains.

By combining NLP pipelines with prompt engineering and LLMs, businesses can scale their customer support operations while maintaining a high degree of personalization and professionalism.

Potential Improvements & Future Work

• Fine-tune Sentiment Model: Fine-tune the sentiment analysis model on a larger, domain-specific dataset for improved accuracy.
• Explore Different LLMs: Experiment with various open-source LLMs (e.g., via Hugging Face Inference API, or locally run models if resources permit) for response generation and insight extraction.
• Advanced Prompting: Implement more sophisticated prompt engineering techniques (e.g., few-shot, chain-of-thought).
• Scalable Deployment: For production, replace hugchat with more robust solutions like official Hugging Face Inference Endpoints, self-hosted models, or commercial LLM APIs.
• Interactive UI: Develop a simple web application (e.g., using Streamlit or Gradio) for easier interaction.
• Comprehensive Evaluation: Define and implement more rigorous metrics for evaluating the quality of generated LLM responses (e.g., ROUGE, BLEU for summarization-like tasks, or human evaluation).

Disclaimer

• The hugchat library is an unofficial tool and its functionality may change or break if Hugging Face modifies its chat interface or login mechanisms.
• LLM outputs can sometimes be unpredictable or contain biases. Always review and curate AI-generated content before using it in a live customer-facing scenario.
• This project is for educational and demonstration purposes.

License

This project is licensed under the MIT License. See the LICENSE file for details.