Instruction Tuning

Status: public · Confidence: medium (0.82) · Basis: verified_sources

## TL;DR

Instruction tuning turns a pretrained language model into a model that is better at following natural-language tasks. For AI programming agents, this is one of the reasons a model can interpret task descriptions, constraints, and tool instructions.

## Core Explanation

Instruction tuning is not the same thing as every alignment method. In the source-mapped slice here, it means supervised fine-tuning or task fine-tuning on instruction-shaped examples, sometimes followed by preference or reinforcement learning stages.

## Source-Mapped Facts

- FLAN fine-tuned a pretrained language model on many tasks expressed through natural-language instructions. ([source](https://arxiv.org/abs/2109.01652))
- The FLAN ablations report that the number of fine-tuning datasets, model scale, and natural-language instructions are key factors in instruction-tuning success. ([source](https://arxiv.org/abs/2109.01652))
- Scaling Instruction-Finetuned Language Models studies instruction finetuning by scaling task count, model size, and chain-of-thought data. ([source](https://arxiv.org/abs/2210.11416))
- InstructGPT collected labeler-written demonstrations and used them to fine-tune GPT-3 with supervised learning before RLHF. ([source](https://arxiv.org/abs/2203.02155))

## Further Reading

- [Finetuned Language Models Are Zero-Shot Learners](https://arxiv.org/abs/2109.01652)
- [Scaling Instruction-Finetuned Language Models](https://arxiv.org/abs/2210.11416)
- [Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/abs/2203.02155)