Training Details |
Data Sources: | WhiteRabbitNeo/WRN-Chapter-1, WhiteRabbitNeo/WRN-Chapter-2, LDJnr/Capybara, teknium/openhermes, teknium/GPTeacher-General-Instruct, Weyaxi/sci-datasets, TIGER-Lab/MathInstruct, hiyouga/glaive-function-calling-v2-sharegpt, glaiveai/glaive-code-assistant, m-a-p/CodeFeedback-Filtered-Instruction, m-a-p/Code-Feedback, migtissera/Synthia-v1.3, abacusai/SystemChat, jondurbin/airoboros-3.2, vicgalle/alpaca-gpt4, garage-bAInd/Open-Platypus, trollek/Mouse-Diffusion-Instruct, trollek/Self-Rewarding-Mouse |
|
Data Volume: | filtered for token count between 2k and 8k |
|
Methodology: | Layerwise insertion of data following the Llama Pro method with additional techniques like BAdam, DoRA, QLoRA. |
|
Context Length: | |
Model Architecture: | MistralForCausalLM with 34 MistralDecoderLayer |
|
|