交互式网站设计,企业网站管理系统项目文档,免费网站建设知识,如何影响网站排名1 指令微调数据集形式“花样”太多
大家有没有分析过 prompt对模型训练或者推理的影响#xff1f;之前推理的时候#xff0c;发现不加训练的时候prompt#xff0c;直接输入模型性能会变差的#xff0c;这个倒是可以理解。假如不加prompt直接训练#xff0c;是不是测试的时…1 指令微调数据集形式“花样”太多
大家有没有分析过 prompt对模型训练或者推理的影响之前推理的时候发现不加训练的时候prompt直接输入模型性能会变差的这个倒是可以理解。假如不加prompt直接训练是不是测试的时候不加prompt也可以还有一个就是多轮prompt和单轮prompt怎么构造的问题好多模型训练方式不统一 包括指令数据形式有所不同选择困难症又来了。。 先说一些观点假如我们在微调一个大模型单次实验微调所用的指令微调数据集应该选取“质量高、多样性”,在训练资源充足的情况可以加入数量更多长度更大的数据集。可以基于多个质量比较高的数据做一份格式统一的多样性数据用来做sft一次性微调完比较好多次微调效果可能会折扣。或者有继续微调比较合适的方案也可以不损失之前模型的效果或者损失比较小目前可以尝试Lora或者Qlora的方式微调底座模型然后将训练好的Lora权重合并到原始模型这样可以减轻多次微调对模型的影响。
2 常见指令微调模板
通过观测一些排行榜靠前和主流指令微调数据集笔者总结一些常见的指令微调的Prompt
常见的是stanford_alpaca中模板
PROMPT_DICT {prompt_input: (Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:),prompt_no_input: (Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:),
}Llama2中的模板
instruction [INST] SYS\nYou are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you dont know the answer to a question, please dont share false information.\n/SYS\n\n{} [/INST]
Linly-AI中模板
### Instruction:{prompt.strip()} ### Response:OpenLLM 排行榜top1的NousResearch
和alpaca模板差不多
### Instruction:
prompt### Response:
leave a newline blank for model to respond### Instruction:
prompt### Input:
additional context### Response:
leave a newline blank for model to respond
Yayi模板 https://huggingface.co/wenge-research/yayi-7b-llama2 prompt 你是谁
formatted_prompt f|System|:
You are a helpful, respectful and honest assistant named YaYi developed by Beijing Wenge Technology Co.,Ltd. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you dont know the answer to a question, please dont share false information.|Human|:
{prompt}|YaYi|:StableBeluga2的模板
### System:
This is a system prompt, please behave and help the user.### User:
Your prompt here### Assistant:
The output of Stable Beluga 2比如
system_prompt ### System:\nYou are Stable Beluga, an AI that follows instructions extremely well. Help as much as you can. Remember, be safe, and dont do anything illegal.\n\nmessage Write me a poem please
prompt f{system_prompt}### User: {message}\n\n### Assistant:\nGuanaco数据集常用模板
### Human: {prompt}
### Assistant:prompt Introduce yourself
formatted_prompt (fA chat between a curious human and an artificial intelligence assistant.fThe assistant gives helpful, detailed, and polite answers to the users questions.\nf### Human: {prompt} ### Assistant:
)3 多轮对话输入和输出构造
参考yangjianxin1/Firefly项目和LinkSoul-AI/Chinese-Llama-2-7b项目一般采用的方式是 在计算loss时我们通过mask的方式input部分的loss不参与参数更新只有“target”部分的loss参与参数更新。 这种方式充分利用了模型并行计算的优势训练更加高效且多轮对话中的每个target部分都参与了训练训练更充分。 否则就需要把一个n轮对话拆分成n条数据且只计算最后一个target的loss大大降低了训练效率。 具体实现方式1
# https://github.com/LinkSoul-AI/Chinese-Llama-2-7b/blob/main/train.py
def tokenize(item, tokenizer):roles {human: user, gpt: assistant}input_ids []labels []if instruction in item and len(item[instruction]) 0:system item[instruction]else:system dummy_message[system]system B_SYS system E_SYS# add system before the first content in conversationsitem[conversations][0][value] system item[conversations][0][value]for i, turn in enumerate(item[conversations]):role turn[from]content turn[value]content content.strip()if role human:content f{B_INST} {content} {E_INST} content_ids tokenizer.encode(content)labels [IGNORE_TOKEN_ID] * (len(content_ids))else:# assert role gptcontent f{content} content_ids tokenizer.encode(content, add_special_tokensFalse) [tokenizer.eos_token_id] # add_special_tokensFalse remove bos token, and add eos at the endlabels content_idsinput_ids content_idsinput_ids input_ids[:tokenizer.model_max_length]labels labels[:tokenizer.model_max_length]trunc_id last_index(labels, IGNORE_TOKEN_ID) 1input_ids input_ids[:trunc_id]labels labels[:trunc_id]if len(labels) 0:return tokenize(dummy_message, tokenizer)input_ids safe_ids(input_ids, tokenizer.vocab_size, tokenizer.pad_token_id)labels safe_ids(labels, tokenizer.vocab_size, IGNORE_TOKEN_ID)return input_ids, labels具体实现方式1
# https://github.com/yangjianxin1/Firefly/blob/master/component/dataset.py
class SFTDataset(Dataset):def __init__(self, file, tokenizer, max_seq_length):self.tokenizer tokenizerself.bos_token_id tokenizer.bos_token_idself.eos_token_id tokenizer.eos_token_idself.eos_token tokenizer.eos_tokenself.bos_token tokenizer.bos_tokenself.max_seq_length max_seq_lengthlogger.info(Loading data: {}.format(file))with open(file, r, encodingutf8) as f:data_list f.readlines()logger.info(there are {} data in dataset.format(len(data_list)))self.data_list data_listdef __len__(self):return len(self.data_list)def __getitem__(self, index):# 每条数据格式为: sinput1/starget1/sinput2/starget2/s...data self.data_list[index]data json.loads(data)conversation data[conversation]# 收集多轮对话utterances []for x in conversation:utterances.append(x[human])utterances.append(x[assistant])utterances_ids self.tokenizer(utterances, add_special_tokensFalse).input_ids# 模型的输入格式为sinput1/starget1/sinput2/starget2/s...input_ids [self.bos_token_id]target_mask [0] # 用于对input进行mask只计算target部分的lossfor i, utterances_id in enumerate(utterances_ids):input_ids (utterances_id [self.eos_token_id])if i % 2 0:target_mask [0] * (len(utterances_id) 1)else:target_mask [1] * (len(utterances_id) 1)assert len(input_ids) len(target_mask)# 对长度进行截断input_ids input_ids[:self.max_seq_length]target_mask target_mask[:self.max_seq_length]attention_mask [1] * len(input_ids)assert len(input_ids) len(target_mask) len(attention_mask)inputs {input_ids: input_ids,attention_mask: attention_mask,target_mask: target_mask}return inputs核心代码就是通过IGNORE_INDEX(-100)遮蔽掉input对应的目标输出即可。
4 如何高效率微调大模型
如何短时间、高效率的训练出实际效果不错、综合能力比较强的大模型呢从指令微调数据集处理工作上个人认为可以从以下方式进行 1 事先准备多种高质量的指令微调数据集每个数据集尽量保持差异性。那高质量如何定义呢我们可以从一些效果不错的模型收集它们训练使用的指令数据集 2笔者在实验过程中发现加入多伦对话的数据有助于提升模型生成能力如果仅用单轮对话或者单轮指令训练出的模型生成长度可能偏短。 3另外通过实验发现如果模型微调的时候使用模板那么推理的时候应该也使用模板否则效果会影响直观上就是生成效果不理想生成比较短甚至“驴唇不对马嘴”训练使用了英文模板推理的时候未使用提示模板的情况下会出现中英文混杂现象。
