ShopNest Global Support Ticket Analysis System

01 · Business Problem

The Problem

ShopNest Global is a large-scale e-commerce platform operating in 30+ countries with 50M+ active customers and 2,000+ human support agents working 24/7 across the US, Europe, India, and Southeast Asia, processing 200,000+ orders/day. Every order carries risk of delivery delay, payment failure, wrong item, or return — and each generates a support ticket. Real tickets are messy: some overloaded with emotional detail, some shorthand-heavy, some a single sparse sentence. Agents spend 2–3 minutes per ticket just decoding the issue before resolution work starts, and peak periods spike volume from ~5,000/day to ~15,000/day.

The Solution

A 4-stage n8n workflow powered by GPT-4.1 that summarizes raw tickets into clean, factual summaries, drafts a professional and policy-grounded customer response with a clear next step, and evaluates both outputs automatically with LLM-as-a-Judge scoring — giving the business measurable, auditable quality signal before a broader rollout decision.

02 · By the Numbers

Tickets Processed

Pipeline Stages

LLM-Judge Rubrics

9.82

Avg Summary Score /10

9.28

Avg Response Score /10

60/60

Submission Grade

03 · Embedded Business Policies

Policy	Terms
Refund & Return	Full refund or replacement within 10 days; refunds processed in 5–7 business days; COD refunds via bank transfer
Delivery Delay	>3 days late → Rs 100 voucher; >7 days late → may qualify for full refund
Wrong Item	Free replacement or refund; pickup arranged within 2 business days; photo evidence may be requested
Payment Failure	Failed transactions usually reversed in 3–5 business days; ask customer for transaction reference if needed

04 · Sample Ticket Walkthrough — Ticket #1

Raw Ticket (support_ticket_desc)

"I cannot believe the level of service I have received. I have been a loyal ShopNest customer since 2020… I spent over three hours today trying to reach your support team… Order SNX-4421 for a Bosch dishwasher. Delivered wrong model. Want replacement."

↓ Stage 1 · Ticket Summarization

Summary

"The customer received the wrong model of Bosch dishwasher for order SNX-4421 and is requesting a replacement. They experienced significant difficulty reaching support through multiple channels, including long hold times and unhelpful responses."

↓ Stage 2 · Summary Evaluation — Accuracy 9/10 · Clarity 10/10 · Completeness 9/10

↓ Stage 3 · Response Generation

Generated Response

"Thank you for reaching out and sharing your experience. I sincerely apologize for the frustration… we will arrange a free pickup of the incorrect dishwasher within 2 business days and process a replacement for the correct model. Could you please reply with a photo of the received item and its packaging?"

↓ Stage 4 · Response Evaluation — Professionalism 9/10 · Empathy 9/10 · Clarity 9/10 · Policy Compliance 8/10 · Actionability 9/10

Pipeline

▶

Manual
Trigger

Load CSV

→

Ticket
Summarization

GPT-4.1, temp 0.2

→

Summary
Evaluation

LLM-as-Judge

→

Response
Generation

GPT-4.1, temp 0.4

→

Response
Evaluation

LLM-as-Judge

→

💾

Export
CSV

output_file.csv

Stage 1

Ticket Summarization

GPT-4.1 · temp 0.2 · max 150 tokens

System Prompt Rules

Summarize in 2–3 concise sentences. Clearly identify the main issue. Remove unnecessary emotional language. Preserve important facts. Mention order/payment/delivery/refund issue explicitly. Maintain factual accuracy. Do not hallucinate.

Input / Output

Input: raw support_ticket_desc field. Output: clean 2–3 sentence summary, stored as Summarization.

Stage 2

Summary Evaluation (LLM-as-a-Judge)

GPT-4.1 · temp 0.2 · max 150 tokens

Rubric

Accuracy — does the summary correctly reflect the original ticket? Clarity — is it easy to understand and professional? Completeness — does it include all major issues, requests, and context? Each scored /10, plus short feedback. Penalizes hallucinations and missing detail.

Judge Input

Original ticket text + the Stage 1 summary, so the judge can compare directly. Output format constrained to plain text (no markdown).

Stage 3

Response Generation

GPT-4.1 · temp 0.4 · max 250 tokens

System Prompt Rules

Be empathetic and professional. Acknowledge the customer's issue. Provide clear next steps. Maintain a calm, respectful tone. Keep it concise and actionable. Do not make unsupported promises.

Policies Embedded in Prompt

Refund & Return, Delivery Delay, Wrong Item, and Payment Failure policies are hard-coded directly into the system prompt (see Overview tab) so every generated response is grounded in the same rules, not model guesswork.

Stage 4

Response Evaluation (LLM-as-a-Judge)

GPT-4.1 · temp 0.1 · max 180 tokens

Rubric

Professionalism, Empathy, Clarity, Policy Compliance (does it correctly follow the stated policies?), Actionability (clear, useful next steps). Each scored /10, plus short feedback. Penalizes unsupported promises and vague actions.

Why a Lower Temperature

temp 0.1 for the judge (vs. 0.2–0.4 for the generation stages) — evaluation needs to be consistent and objective run to run, not creative.

Design Notes

Why n8n?

No-code orchestration made the 4-stage pipeline visually debuggable node-by-node, and the Set node pattern (Prepare Final Output) made it trivial to assemble all 6 output fields — ticket id, description, summary, summary eval, response, response eval — into one row per ticket before CSV export.

Why Two Separate Judge Calls?

Summarization and response generation are different tasks with different failure modes (factual drift vs. policy compliance), so each gets its own rubric and its own LLM-as-a-Judge call rather than one combined evaluation — keeping each score interpretable and actionable on its own.

How to Import into n8n

01 · Download

Click "Download .json" above to get the workflow file.

02 · Import

In n8n: Menu → Workflows → Import from File. Select the downloaded JSON.

03 · Credentials

Connect your OpenAI API key on all 4 GPT-4.1 nodes.

04 · File Paths

Update the Load Support Ticket Dataset and Export Final Results node paths to your environment, then run.

Summary Evaluation — 30/30 Tickets (/10)

Accuracy

9.83

Clarity

10.00

Completeness

9.63

Overall average: 9.82/10 · 20 of 30 tickets scored a perfect 10/10/10

Response Evaluation — 30/30 Tickets (/10)

Professionalism

9.73

Empathy

9.00

Clarity

9.37

Policy Compliance

9.23

Actionability

9.07

Overall average: 9.28/10

Key Finding

Policy Compliance was the lowest and most variable metric (min 7/10). The judge repeatedly flagged responses that promised a specific outcome — a full refund, a Rs 100 voucher — without the ticket containing enough information to confirm eligibility against the stated policy. This is the clearest lever for prompt iteration: constrain Response Generation to confirm eligibility criteria before offering compensation, rather than offering it proactively.

Sample Scored Tickets

#	Issue	Summary Eval	Response Eval
1	Wrong dishwasher model delivered, replacement requested	9 / 10 / 9	9 / 9 / 9 / 8 / 9
3	"not working. please help." — near-zero detail	10 / 10 / 9	10 / 9 / 9 / 10 / 9
10	14-day non-delivery, conflicting agent explanations, BBB escalation threat	10 / 10 / 10	10 / 9 / 10 / 10 / 10
14	Missing laptop charger, heavy typos, urgent for work	10 / 10 / 10	9 / 9 / 10 / 10 / 10
21	Stock-shown-as-available cancellation, refund past SLA, billing escalation	10 / 10 / 10	10 / 9 / 9 / 10 / 9

Summary Eval columns: Accuracy / Clarity / Completeness. Response Eval columns: Professionalism / Empathy / Clarity / Policy Compliance / Actionability. Full 30-ticket results in data/output_file.csv.