Llama 3.1 8B — Fine-Tuning

01 · Business Problem

The Problem

When a base model (meta-llama/Llama-3.1-8B-Instruct) receives vague feedback like "This app is garbage", it tends to invent technical details — fabricated system names, made-up reproduction steps, imaginary root causes. This is hallucination: the model generates a confident, professional-looking engineering ticket from nothing, which is exactly the wrong behavior to ship to an engineering team.

The Solution

A 2-agent n8n pipeline — Feedback Classifier → Engineering Insight Writer — run entirely on a local Llama 3.1 8B Instruct model (Apple Silicon, MLX-LM, 4-bit). Two small LoRA adapters (~40 MB each, 0.131% of parameters trainable) are fine-tuned to teach each agent the correct behavior on vague input: flag it as UNCLEAR / AFFECTED_SYSTEM: UNKNOWN instead of fabricating a ticket. Before/after behavior is measured with DeepEval using gpt-4o as judge.

02 · By the Numbers

LoRA Adapters

0.131%

Params Trainable

Examples / Adapter

~40MB

Adapter Size

8/8

Test Cases Run

75%

Pass Rate (Before & After)

03 · Sample Case — F1 (Detailed Feedback)

"The checkout page keeps freezing when I try to apply a discount code on my iPhone 14. Happens every time. I've lost 3 orders this week because of it. Using Safari, latest iOS." — App Store review, 1 star

Field	Baseline Llama result	Expected result
Category / ticket type	PERFORMANCE_ISSUE	BUG_REPORT / BUG
Priority	P2-HIGH	P1-CRITICAL

The base model extracted system, platform, and reproduction hints correctly, but treated a deterministic functional freeze as a performance issue and understated the priority despite 3 lost orders — a clear fine-tuning target.

04 · n8n Execution Evidence (Baseline)

All 8 baseline cases (F1–F8) completed successfully end-to-end: Chat Trigger → Feedback Classifier → Classifier LLM → Engineering Insight Writer → Insight LLM. Full screenshot set in the repo's assets/ folder.

Pipeline

💬

Chat
Trigger

n8n

→

Feedback
Classifier

classifier-n8n LoRA

→

Engineering
Insight Writer

insight LoRA

→

💾

Structured
Ticket

output

Stage 1

Base Model & Local Serving

MLX-LM · mlx-community/Llama-3.1-8B-Instruct-4bit · FastAPI compatibility adapter

4-bit MLX Build

~4.2 GB, ~8.03B parameters, chosen because it represents the instructor's Llama 3.1 8B Instruct model in a format that runs natively on Apple Silicon.

n8n Compatibility Adapter

A local FastAPI service exposes two model aliases (classifier-llama, insight-llama) routing to two MLX inference services on ports 8006/8007, relaying streamed Chat Completions so n8n's LLM nodes work unmodified. ngrok tunnels it to hosted n8n.

Stage 2

LoRA Fine-Tuning

MLX-LM LoRA · rank 8, scale 20 · 16 trainable layers

Two Adapters, One Base Model

10.486M of 8.03B parameters trainable (0.131%). 15 training examples each, batch size 1, max sequence length 1,024, learning rate 1e-5, prompt loss masked so learning focuses on the assistant answer.

Classifier — 60 iterations

Validation loss 1.413 → 0.013, training loss 0.085. Retrained a second time on the exact n8n system prompt after the first version misclassified live input (see lesson below).

Insight Writer — 60 + 30 iterations

Validation loss 2.396 → 0.083 → 0.042. The 30-iteration continuation was added because the first sanity check stopped before the priority section.

Key Lesson — Training Prompts Must Match Inference Prompts

The first classifier adapter memorized the 15 training examples perfectly (using the instructor's shorter system prompt) but misclassified live n8n input as USABILITY_COMPLAINT instead of UNCLEAR, because n8n sends a longer system prompt than the training data used. Retraining every example against the exact n8n system prompt fixed it. Low training loss alone does not guarantee correct production behavior if training and inference instructions diverge.

How to Import into n8n

01 · Download

Click "Download .json" above to get the workflow file.

02 · Import

In n8n: Menu → Workflows → Import from File. Select the downloaded JSON.

03 · Local model

Both LLM nodes point at a local Llama endpoint via ngrok — swap in the compatibility adapter's public URL.

Baseline vs. Post-Fine-Tuning (DeepEval, judge = gpt-4o)

Metric	Baseline	Post-fine-tuning	Change
Overall test-case pass rate	75% (6/8)	75% (6/8)	No change
Detailed (F1–F4) Groundedness	0.79	0.75	−0.04
Detailed (F1–F4) Usefulness	0.95	0.86	−0.09
Detailed (F1–F4) Answer Relevancy	0.94	0.93	−0.01
Vague (F5–F8) Feedback Handling	0.56	0.57	+0.01
Vague (F5–F8) Answer Relevancy	0.68	0.76	+0.08

GPT-4o is an LLM judge, so small score differences between runs shouldn't be treated as statistically significant without repeated evaluations.

Key Finding

Fine-tuning did not raise the overall pass rate. With only 15 training examples per adapter, the model learned the specific examples shown rather than the general rule. F8 — the exact pattern in the training data — went from failing (score 0.22) to passing strongly (1.00, correctly using INVESTIGATION_NEEDED / P4-LOW and keeping system/platform unknown). But F5 and F6, different negative phrasings the training set didn't cover, went from passing to failing — the fine-tuned model over-generalized toward P1-CRITICAL and invented reproduction context on those. This is a classic small-dataset generalization gap, not a broken approach — the fix is more diverse vague-negative training examples, not more iterations on the same 15.

Vague-Feedback Case Results (F5–F8)

Case	Feedback	Baseline	Post-fine-tuning
F5	"This app is garbage..."	PASS	FAIL (0.21) — used P1-CRITICAL, invented context
F6	"Terrible..."	PASS	FAIL — invented checkout context, P1-CRITICAL
F7	"Meh. It's fine I guess."	FAIL (0.34)	PASS (0.81) — correct P4-LOW
F8	"Fix your stuff."	FAIL (0.22) — asked for feedback again	PASS (1.00) — exemplary INVESTIGATION_NEEDED / P4-LOW

Detailed cases F1–F4 passed all metrics both before and after. Full per-metric scores and DeepEval reasoning in the repo's notes/ and results/ folders.

Evaluation Limitation

F1–F8 are present in the instructor-supplied training data, so this comparison measures how well the model learned known examples — not whether it generalizes to unseen feedback. A stronger follow-up would add an F9–F16 holdout set that is never included in training. Full source: github.com/vppuri-vjra/llama-fine-tuning