GPT-4o-mini — Fine-Tuning

01 · Business Problem

The Problem

When a base model receives vague feedback like "This app is garbage", it tends to invent technical details — fabricated system names, made-up reproduction steps, imaginary root causes. This is hallucination: the model generates a confident, professional-looking engineering ticket from nothing, which is exactly the wrong behavior to ship to an engineering team.

The Solution

A 2-agent n8n pipeline — Feedback Classifier → Engineering Insight Writer — fine-tuned via the OpenAI fine-tuning API on gpt-4o-mini-2024-07-18. The training data (10 detailed + 5 corrected-vague examples per agent) directly teaches the "insufficient data" response: flag vague feedback as UNCLEAR and recommend reaching out to the user instead of fabricating a ticket. Before/after behavior is measured with DeepEval using the larger gpt-4o as judge.

02 · By the Numbers

Fine-Tuned Models

Training Epochs

Examples / Model

10+5

Detailed / Corrected-Vague

8/8

Test Cases Passed

100%

Pass Rate (Before & After)

03 · Sample Case — F1 (Detailed Feedback)

"The checkout page keeps freezing when I try to apply a discount code on my iPhone 14. Happens every time. I've lost 3 orders this week because of it. Using Safari, latest iOS." — App Store review, 1 star

Run	Ticket type	Severity / priority	Note
Baseline	BUG	P1-CRITICAL	Assumed blast radius extends to broader revenue/user-satisfaction impact beyond the 3 reported orders
Post-fine-tuning	BUG	P2-HIGH	Still assumed Production, React, and blocking operations — unsupported technical detail not in the original feedback

Both runs correctly identify the bug and produce a useful ticket, but both introduce unsupported technical assumptions — the core groundedness weakness fine-tuning did not fully resolve (see Evaluation Results).

Pipeline

💬

Chat
Trigger

n8n

→

Feedback
Classifier

fb-classifier FT

→

Engineering
Insight Writer

eng-insight FT

→

💾

Structured
Ticket

output

Stage 1

OpenAI Fine-Tuning Jobs

gpt-4o-mini-2024-07-18 · 3 epochs · run in parallel

Two Independent Jobs

15 JSONL examples each (10 detailed + 5 corrected-vague), uploaded and launched by scripts/fine_tune_engineering.py. Both jobs run in parallel on OpenAI's infrastructure and are polled until complete.

Resulting Model IDs

ft:gpt-4o-mini-2024-07-18:...:fb-classifier:E5v3KgdP
ft:gpt-4o-mini-2024-07-18:...:eng-insight:E5v1LQ0i — pasted directly into the Classifier LLM / Insight LLM nodes in n8n.

Stage 2

Why a Separate, Larger Judge Model

gpt-4o evaluates gpt-4o-mini's outputs

DeepEval, 4 Metrics

Engineering Groundedness, Engineering Usefulness, Vague Feedback Handling, Answer Relevancy — each a G-Eval custom metric with a 0.50 pass threshold, judged by gpt-4o rather than the mini model being evaluated, to avoid the judge sharing the same blind spots as the model under test.

Design Notes

Why 10 Detailed + 5 Corrected-Vague?

The 5 corrected-vague examples are the key to fixing hallucination — they explicitly teach the model what to do when it doesn't have enough information, rather than letting it default to inventing a plausible-sounding ticket.

Why the Baseline Already Passed 100%?

The course guide anticipated a ~25–50% baseline pass rate; this run scored 100% before any fine-tuning. Pass/fail at a 0.50 threshold is a coarse signal — the comparison that actually matters is the per-metric score movement (see Evaluation Results).

How to Import into n8n

01 · Download

Click "Download .json" above to get the workflow file.

02 · Import

In n8n: Menu → Workflows → Import from File. Select the downloaded JSON.

03 · Fine-tuned models

Paste your own ft:gpt-4o-mini-... model IDs into the Classifier LLM and Insight LLM nodes.

Baseline vs. Post-Fine-Tuning (DeepEval, judge = gpt-4o)

Metric	Baseline	Post-fine-tuning	Change
Overall test-case pass rate	100% (8/8)	100% (8/8)	Unchanged
Detailed (F1–F4) Groundedness	0.83	0.74	−0.09
Detailed (F1–F4) Usefulness	0.93	0.92	~unchanged
Detailed (F1–F4) Answer Relevancy	0.93	0.96	+0.03
Vague (F5–F8) Feedback Handling	0.99	0.99	Unchanged (excellent)
Vague (F5–F8) Answer Relevancy	0.86	0.85	~unchanged

The baseline already passed 100% of test cases, so pass rate alone doesn't show fine-tuning's effect — the per-metric scores tell the real story.

Key Finding

Mixed result, not an across-the-board improvement. Fine-tuning was effective at teaching conservative handling of vague feedback — F5–F8 consistently avoided inventing systems, platforms, and reproduction steps, and vague-feedback handling stayed excellent at 0.99. But Engineering Groundedness regressed from 0.83 to 0.74 on detailed cases: F1 assumed Production/React/blocking operations, F2 turned customer-reported latency into a "confirmed" regression, F3 declared the entire onboarding flow broken from one report, and F4 treated a suspicious-login alert as a confirmed compromise. The fix isn't more training on the same data — it's negative examples that explicitly reject unsupported implementation detail, plus stricter schema constraints (one consistent `UNKNOWN` value, hypotheses clearly labeled as such).

Vague-Feedback Case Results (F5–F8)

Case	Feedback	Post-fine-tuning behavior	Result
F5	"This app is garbage..."	No invented systems, platforms, or reproduction steps	PASS
F6	"Terrible..."	No invented systems, platforms, or reproduction steps	PASS
F7	"Meh. It's fine I guess."	Correct engineering decision, though wording was unnecessarily harsh	PASS
F8	"Fix your stuff."	Clean NO_ACTION_NEEDED / insufficient-data decision	PASS

All 4 vague cases passed both before and after (group average: 0.99 handling, 0.85 relevancy — see table above). Detailed cases F1–F4 also passed all metrics both before and after, despite the groundedness score decline. Full per-case DeepEval reasoning is in the repo's notes/ and results/ folders.

Recommended Next Iteration

1. Add negative training examples that explicitly reject unsupported implementation details and incident conclusions.
2. Require uncertain fields to use one consistent value such as UNKNOWN.
3. Distinguish customer-reported metrics from independently verified facts.
4. Require investigation ideas to be labeled as hypotheses, not conclusions.
5. Standardize ticket type, severity, and priority enums across cases.
6. Add professional-tone examples for vague or dismissive user feedback.

Full source: github.com/vppuri-vjra/gpt-fine-tuning