Project 02 / Independent ML experimentation · DIY JEV
Language Model Evaluation & Calibration
Evaluating Qwen3-0.6B on CommonsenseQA: LoRA fine-tuning improved accuracy, then temperature scaling brought confidence closer to observed performance without changing the predicted answers.
Average confidence Accuracy
600 held-out questions. Fine-tuning improved accuracy; temperature scaling improved calibration while accuracy stayed at 65.17%.
| Metric | Original | Fine-tuned | Calibrated |
|---|---|---|---|
| Accuracy | 49.33% | 65.17% | 65.17% |
| Average confidence | 92.00% | 82.60% | 65.07% |
| NLL | 3.7584 | 1.1344 | 0.9160 |
| ECE | 42.94% | 17.43% | 3.91% |
CommonsenseQA training split: 8,341 training / 400 validation / 400 calibration / 600 held out. Temperature fitted only on the calibration partition.
The question
A confident answer is not necessarily a correct answer. In DIY JEV, I independently investigated prediction quality, confidence estimation, and decision-making using Qwen3-0.6B on five-choice CommonsenseQA questions. The original model was highly overconfident: 92.00% average confidence at 49.33% accuracy.
Evaluation method
The final experiment shuffled the 9,741-question CommonsenseQA training split with seed 42, then used 8,341 questions for training, 400 for validation, 400 for calibration, and 600 for held-out evaluation. These are internal partitions of the training split, not an independent external benchmark.
I evaluated the original model, the selected LoRA adapter, and the same adapter after temperature scaling on the same 600 held-out questions. The temperature was fitted only on the separate calibration partition. Earlier experiments on the official validation split are separate development runs and are not mixed into this comparison.
What changed
LoRA fine-tuning raised accuracy from 49.33% to 65.17% and reduced average confidence from 92.00% to 82.60%. Applying temperature scaling to the fine-tuned model then reduced average confidence to 65.07%, while accuracy stayed at 65.17%.
Negative log-likelihood fell from 3.7584 for the original model to 1.1344 after fine-tuning and 0.9160 after calibration. Ten-bin expected calibration error fell from 42.94% to 17.43% to 3.91%. Lower NLL indicates better probability assignments to correct answers; lower ECE indicates a smaller binned confidence–accuracy gap.
Scaling logits by a positive temperature changes probabilities without changing the highest-scoring answer. Fine-tuning improved prediction accuracy in this experiment; temperature scaling improved calibration. Average confidence alone is not a complete calibration test.
Implementation
The Python workflow uses PyTorch, Hugging Face Transformers and Datasets, and PEFT. LoRA adapters target the attention q_proj and v_proj modules with rank 8, alpha 16, and dropout 0.05. Training uses AdamW, a learning-rate schedule, gradient accumulation, validation checks, and early stopping.
The inference pipeline scores answer tokens A–E, applies the fitted temperature (approximately 2.0905), and can abstain below a confidence threshold. At a 70% threshold, it answered 260 of 600 questions: 43.33% coverage with 82.69% accuracy on answered questions. That selective accuracy reflects abstention, not an improvement in overall model accuracy.
Limits and learning
The results describe a specific model, prompt, dataset partition, and adapter. Calibration may not transfer to unfamiliar questions or a different distribution, and abstention reduces coverage. This is an independent experiment using an existing model, rather than foundation-model training or a production AI system.
The work helped me distinguish prediction quality, confidence estimation, and evaluation design—and use separate training, validation, calibration, and held-out partitions to assess each step.