Supervised fine tuning (SFT) In this approach, the language model is fine-tuned on a labeled dataset specific to the target domain or task. The model learns to generate outputs similar to the labeled examples in the dataset. This approach can be useful to fine-tune a language model for legal document summarization, where you have a dataset of legal documents and their corresponding summaries.
Reinforcement Learning from Human Feedback (RLHF) is a technique involves providing human feedback (e.g., ratings, comparisons, or corrections) to the language model during the fine-tuning process. The model is trained to generate outputs that align with the human feedback, effectively shaping its behavior according to human preferences.
RLHF can be particularly useful when you want to imbue the language model with specific traits, such as factuality, safety, or ethical behavior, for example when training a language model for customer service applications, where it needs to provide helpful, polite, and factual responses.
Denoising Pre-training Objective (DPO): DPO is a self-supervised pre-training technique that aims to improve the language model’s understanding of the target domain by training it on corrupted input data from that domain. The model learns to reconstruct the original, uncorrupted input, thereby capturing domain-specific patterns and knowledge, for example a model can be trained on corrupted scientific paper abstracts to improve its performance on tasks related to scientific literature, such as question answering or text summarization.
Orthogonal Random Projection Objective (ORPO): ORPO is a self-supervised training technique that aims to enhance the language model’s ability to understand and generate text from the target domain. It involves projecting the input text into a random subspace and training the model to predict the original input from the projected representation. For example, a general-purpose language model to the financial domain by training it with ORPO on a corpus of financial reports and news articles. model alignment techniques
These techniques can be used individually or in combination, depending on the specific requirements and constraints of the domain adaptation task.