ShopNest Global AI Support Copilot

01 · Business Problem

The Problem

ShopNest Global is a large-scale e-commerce platform operating in 30+ countries with 50M+ active customers and 2,000+ human support agents working 24/7 across the US, Europe, India, and Southeast Asia, processing 200,000+ orders/day. Every order carries risk of delivery delay, payment failure, wrong item, or return — and each generates a support ticket. Real tickets are messy: some overloaded with emotional detail, some shorthand-heavy, some a single sparse sentence. Agents spend 2–3 minutes per ticket just decoding the issue before resolution work starts, and peak periods spike volume from ~5,000/day to ~15,000/day.

The Solution

A 4-stage n8n workflow powered by GPT-4.1 that summarizes raw tickets into clean, factual summaries, drafts a professional and policy-grounded customer response with a clear next step, and evaluates both outputs automatically with LLM-as-a-Judge scoring — giving the business measurable, auditable quality signal before a broader rollout decision.

02 · By the Numbers

Tickets Processed

Pipeline Stages

LLM-Judge Rubrics

9.82

Avg Summary Score /10

9.28

Avg Response Score /10

60/60

Submission Grade

03 · Embedded Business Policies

Policy	Terms
Refund & Return	Full refund or replacement within 10 days; refunds processed in 5–7 business days; COD refunds via bank transfer
Delivery Delay	>3 days late → Rs 100 voucher; >7 days late → may qualify for full refund
Wrong Item	Free replacement or refund; pickup arranged within 2 business days; photo evidence may be requested
Payment Failure	Failed transactions usually reversed in 3–5 business days; ask customer for transaction reference if needed

04 · Sample Ticket Walkthrough — Ticket #1

Raw Ticket (support_ticket_desc)

"I cannot believe the level of service I have received. I have been a loyal ShopNest customer since 2020… I spent over three hours today trying to reach your support team… Order SNX-4421 for a Bosch dishwasher. Delivered wrong model. Want replacement."

↓ Stage 1 · Ticket Summarization

Summary

"The customer received the wrong model of Bosch dishwasher for order SNX-4421 and is requesting a replacement. They experienced significant difficulty reaching support through multiple channels, including long hold times and unhelpful responses."

↓ Stage 2 · Summary Evaluation — Accuracy 9/10 · Clarity 10/10 · Completeness 9/10

↓ Stage 3 · Response Generation

Generated Response

"Thank you for reaching out and sharing your experience. I sincerely apologize for the frustration… we will arrange a free pickup of the incorrect dishwasher within 2 business days and process a replacement for the correct model. Could you please reply with a photo of the received item and its packaging?"

↓ Stage 4 · Response Evaluation — Professionalism 9/10 · Empathy 9/10 · Clarity 9/10 · Policy Compliance 8/10 · Actionability 9/10

01 · Product Thesis & North Star

Product Thesis

Build a governed agent copilot, not an autonomous service bot. The copilot identifies intent, assembles permissioned order and policy evidence, proposes an allowed next action, drafts a cited response, and routes uncertain or high-risk cases to a person. Agents retain final authority during the pilot.

North Star KPI

Verified First-Contact Resolution Rate: the percentage of eligible contacts resolved in one interaction, with no policy violation, correction, reopen, or repeat contact within seven days. Pilot target: ≥12% relative improvement.

≥12%

Verified FCR Lift

−20%

Median Handle Time

≥95%

Grounded Responses

≥4.3

CSAT / 5

<1%

Severe Error Rate

≤8s

p95 Draft Latency

02 · Problem Framing, Assumptions & Scope

Primary User & Segment

Frontline support agents serving high-value returning customers with post-purchase issues. This behavioral segment combines high support load, high relationship value, reusable order history, and a strong opportunity to improve the North Star.

High-value returningPost-purchaseChat + email

Business Alignment

Increase correct first-contact resolution while lowering cost per resolved contact. Assumptions: authenticated customer context; permissioned read access to orders, CRM history, and approved policy; live chat and email first; market-specific policy metadata is available.

In scope	Out of scope for pilot
Intent and entity extraction; order/policy retrieval; evidence panel; response draft; allowed-action recommendation; risk routing	Autonomous refunds, credits, cancellations, fraud appeals, legal/safety cases, voice automation, public social replies, or foundation-model training

03 · Problem Selection & Focused Experience

Candidate problem	Impact	AI fit	Decision
Agents search several systems for the right order and policy context	High	High with retrieval and citations	Selected
Agents rewrite similar responses	Medium	High	Supporting capability
Policy exceptions require judgment	High	Medium / high risk	Human escalation
Carrier status must be current	Medium	Low	Deterministic API

Understand

Intent + entities

→

Retrieve

Order + policy evidence

→

Assist

Draft + allowed action

→

Govern

Approve or escalate

04 · AI, Model & Adaptation Choices

Capability	Approach	Rationale
Authentication, order status, refund limits, eligibility, action execution	Rules + APIs	Exact, auditable, and fail-closed
Intent, entity extraction, summarization	Fast model / LLM	Customer language is varied and ambiguous
Policy and precedent discovery	Hybrid RAG	Semantic retrieval plus market/version metadata filters
Draft and explanation	Grounded LLM	Natural language generation only after facts and allowed actions are fixed
Material exception or action	Human approval	Financial and relationship judgment remains accountable

Prompting

Define role, workflow, professional tone, citation requirements, structured output, and abstention behavior.

RAG

Required for dynamic policies, order facts, approved macros, and factual grounding.

Fine-tuning

Defer until stable error patterns and enough expert-reviewed examples exist.

05 · RAG & Data Strategy

Retrieval Design

Sources: policy CMS, order and logistics APIs, permissioned CRM history, approved response macros, and exception directory.

Chunking: one policy clause or decision unit per chunk, retaining title, market, product, effective date, version, and authority.

Retrieval: metadata filtering + keyword/vector search + reranking. No material claim or action without a valid citation or deterministic fact.

Golden Dataset

Versioned conversations with retrieved evidence, approved intent, entities, allowed action, escalation decision, and ideal response. Cover top intents, regions, languages, ambiguity, conflicting policies, missing context, and rare high-risk cases. Support QA and policy owners adjudicate labels; the test split remains frozen.

Precision-first stance: retrieve broadly, answer narrowly. An agent can recover an omitted suggestion; a confidently wrong promise may create financial loss or breach trust. Missing, weak, or conflicting evidence triggers abstention or escalation.

06 · Evaluation & Human-in-the-Loop

Stage	Measures	Gate
Offline deterministic	Intent macro-F1, entity F1, retrieval recall@k, citation precision, action-rule accuracy	100% critical-action rules; ≥95% citation precision
Offline human	Correctness, evidence support, relevance, completeness, tone, safe escalation	≥4.5/5 correctness; severe errors <1%
Shadow	Compare draft and action with expert resolution	No customer exposure; review every high-risk disagreement
Controlled A/B	Verified FCR, handle time, CSAT, reopen, escalation, edit distance, adoption	North Star improves without safety or CSAT regression

Where Humans Intervene

Agents review every pilot draft. Mandatory escalation covers fraud, safety, legal threats, vulnerable customers, policy conflicts, missing evidence, unknown intent, and actions above the agent's monetary authority.

Feedback Loop

Capture helpful/not-helpful reasons, edit distance, rejected actions, escalations, reopens, and repeat contacts. Weekly QA turns confirmed failures into regression cases before any prompt, retrieval, or model release.

07 · Target Architecture, Operations & Safety

ExperienceAgent workspaceConversation, customer and order context, evidence, draft, actions, escalation

OrchestrationIntent → tools → retrieval → guardrails → generationModel routing, authority checks, citation contract, and risk-based routing

KnowledgeLive APIs + governed policy indexOrders, logistics, CRM, policy CMS, approved macros, versions, permissions

AssuranceObservability, evaluation, privacy, and reliabilityLatency/cost, groundedness, drift, RBAC, redaction, audit logs, caching, timeouts, fallback

Operational Plan

Parallelize order and policy retrieval, stream drafts, route simple tasks to smaller models, cap context, cache versioned policy results, autoscale for traffic spikes, and fall back to deterministic facts and standard search if generation is unavailable.

Safety Controls

Citation-required claims, post-generation verification, prompt-injection isolation, least-privilege tools, PII masking, professional-tone moderation, parity tests by language/region/tier, and allowlisted actions validated server-side.

08 · Go / No-Go Framework & Trade-offs

Dimension	Launch / expand if	Stop / roll back if
Value	Verified FCR improves ≥12% and handle time improves ≥20%	No significant FCR gain or CSAT drops >0.1
Quality	Grounded-response rate ≥95%	Severe factual or policy error rate ≥1%
Safety	100% critical actions pass authorization tests	Unauthorized action, material privacy incident, or missed critical escalation
Reliability	p95 ≤8 seconds and availability ≥99.5%	Persistent failure without safe fallback
Economics	Cost per resolved eligible contact is below manual baseline	Model and tool costs erase measurable benefit

Accuracy vs Latency

Accept modest latency for verified evidence; parallelize to remain under eight seconds.

Cost vs Quality

Route by complexity instead of using the strongest model for every step.

Automation vs Control

Recommend and draft; humans approve material actions during the pilot.

Recommendation: launch a limited, agent-facing pilot for high-value returning customers and the highest-volume post-purchase intents. Start in shadow mode and expand only after all customer, quality, safety, reliability, and economic gates pass.

Pipeline

▶

Manual
Trigger

Load CSV

→

Ticket
Summarization

GPT-4.1, temp 0.2

→

Summary
Evaluation

LLM-as-Judge

→

Response
Generation

GPT-4.1, temp 0.4

→

Response
Evaluation

LLM-as-Judge

→

💾

Export
CSV

output_file.csv

Stage 1

Ticket Summarization

GPT-4.1 · temp 0.2 · max 150 tokens

System Prompt Rules

Summarize in 2–3 concise sentences. Clearly identify the main issue. Remove unnecessary emotional language. Preserve important facts. Mention order/payment/delivery/refund issue explicitly. Maintain factual accuracy. Do not hallucinate.

Input / Output

Input: raw support_ticket_desc field. Output: clean 2–3 sentence summary, stored as Summarization.

Stage 2

Summary Evaluation (LLM-as-a-Judge)

GPT-4.1 · temp 0.2 · max 150 tokens

Rubric

Accuracy — does the summary correctly reflect the original ticket? Clarity — is it easy to understand and professional? Completeness — does it include all major issues, requests, and context? Each scored /10, plus short feedback. Penalizes hallucinations and missing detail.

Judge Input

Original ticket text + the Stage 1 summary, so the judge can compare directly. Output format constrained to plain text (no markdown).

Stage 3

Response Generation

GPT-4.1 · temp 0.4 · max 250 tokens

System Prompt Rules

Be empathetic and professional. Acknowledge the customer's issue. Provide clear next steps. Maintain a calm, respectful tone. Keep it concise and actionable. Do not make unsupported promises.

Policies Embedded in Prompt

Refund & Return, Delivery Delay, Wrong Item, and Payment Failure policies are hard-coded directly into the system prompt (see Overview tab) so every generated response is grounded in the same rules, not model guesswork.

Stage 4

Response Evaluation (LLM-as-a-Judge)

GPT-4.1 · temp 0.1 · max 180 tokens

Rubric

Professionalism, Empathy, Clarity, Policy Compliance (does it correctly follow the stated policies?), Actionability (clear, useful next steps). Each scored /10, plus short feedback. Penalizes unsupported promises and vague actions.

Why a Lower Temperature

temp 0.1 for the judge (vs. 0.2–0.4 for the generation stages) — evaluation needs to be consistent and objective run to run, not creative.

Design Notes

Why n8n?

No-code orchestration made the 4-stage pipeline visually debuggable node-by-node, and the Set node pattern (Prepare Final Output) made it trivial to assemble all 6 output fields — ticket id, description, summary, summary eval, response, response eval — into one row per ticket before CSV export.

Why Two Separate Judge Calls?

Summarization and response generation are different tasks with different failure modes (factual drift vs. policy compliance), so each gets its own rubric and its own LLM-as-a-Judge call rather than one combined evaluation — keeping each score interpretable and actionable on its own.

How to Import into n8n

01 · Download

Click "Download .json" above to get the workflow file.

02 · Import

In n8n: Menu → Workflows → Import from File. Select the downloaded JSON.

03 · Credentials

Connect your OpenAI API key on all 4 GPT-4.1 nodes.

04 · File Paths

Update the Load Support Ticket Dataset and Export Final Results node paths to your environment, then run.

Summary Evaluation — 30/30 Tickets (/10)

Accuracy

9.83

Clarity

10.00

Completeness

9.63

Overall average: 9.82/10 · 20 of 30 tickets scored a perfect 10/10/10

Response Evaluation — 30/30 Tickets (/10)

Professionalism

9.73

Empathy

9.00

Clarity

9.37

Policy Compliance

9.23

Actionability

9.07

Overall average: 9.28/10

Key Finding

Policy Compliance was the lowest and most variable metric (min 7/10). The judge repeatedly flagged responses that promised a specific outcome — a full refund, a Rs 100 voucher — without the ticket containing enough information to confirm eligibility against the stated policy. This is the clearest lever for prompt iteration: constrain Response Generation to confirm eligibility criteria before offering compensation, rather than offering it proactively.

Sample Scored Tickets

#	Issue	Summary Eval	Response Eval
1	Wrong dishwasher model delivered, replacement requested	9 / 10 / 9	9 / 9 / 9 / 8 / 9
3	"not working. please help." — near-zero detail	10 / 10 / 9	10 / 9 / 9 / 10 / 9
10	14-day non-delivery, conflicting agent explanations, BBB escalation threat	10 / 10 / 10	10 / 9 / 10 / 10 / 10
14	Missing laptop charger, heavy typos, urgent for work	10 / 10 / 10	9 / 9 / 10 / 10 / 10
21	Stock-shown-as-available cancellation, refund past SLA, billing escalation	10 / 10 / 10	10 / 9 / 9 / 10 / 9

Summary Eval columns: Accuracy / Clarity / Completeness. Response Eval columns: Professionalism / Empathy / Clarity / Policy Compliance / Actionability. Full 30-ticket results in data/output_file.csv.