Model Type | text generation, auto-regressive |
|
Use Cases |
Areas: | |
Applications: | Natural Language Generation |
|
Primary Use Cases: | Assistant-like chat, Tailored natural language processing tasks |
|
Limitations: | Not intended for use in violation of laws or non-English languages without compliance. |
|
Considerations: | Ensure compliance with the Acceptable Use Policy and further fine-tune for language variations. |
|
|
Additional Notes | Developers are encouraged to contribute to the community with feedback and improvements using Meta's GitHub resources. |
|
Supported Languages | English (Advanced proficiency) |
|
Training Details |
Data Sources: | A new mix of publicly available online data, Publicly available instruction datasets, Human-annotated examples |
|
Data Volume: | Public pretraining data with 15 trillion+ tokens and over 10 million human-annotated examples |
|
Methodology: | Pretrained and instruction tuned. Utilizes supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) |
|
Context Length: | |
Training Time: | Llama 3 used a cumulative 7.7M GPU hours of computation |
|
Hardware Used: | Meta's Research SuperCluster, Third-party cloud compute, Hardware type H100-80GB |
|
Model Architecture: | Optimized transformer architecture with supervised fine-tuning and reinforcement learning with human feedback. Uses Grouped-Query Attention (GQA) for improved inference scalability. |
|
|
Safety Evaluation |
Methodologies: | Red teaming, Adversarial evaluations, Safety mitigations |
|
Findings: | Model helps in ensuring safety through thoughtful design, but residual risks might remain. |
|
Risk Categories: | Cybersecurity, Child Safety |
|
Ethical Considerations: | Adheres to Responsible AI development principles. Emphasis on open approach, responsible deployment, and essential safety tools. |
|
|
Responsible Ai Considerations |
Fairness: | Open approach ensures inclusivity with mechanisms to address problematic outputs when identified. |
|
Transparency: | Uses detailed Responsible AI practices and safety toolkits available for the community. |
|
Accountability: | Meta is committed to addressing and mitigating risks with improved public safety measures. |
|
Mitigation Strategies: | Tools like Meta Llama Guard 2 and Code Shield provide safety layers on top of model outputs. |
|
|
Input Output |
Input Format: | |
Accepted Modalities: | |
Output Format: | |
Performance Tips: | Optimal for instruction-tuned chat and dynamic text generation contexts. |
|
|
Release Notes |
Version: | |
Date: | |
Notes: | First release of the Llama 3 model with substantial advancements in transformer efficiency and safety features. |
|
|
|