Model Type | auto-regressive language model, transformer architecture |
|
Use Cases |
Areas: | research on large language models |
|
Applications: | question answering, natural language understanding, reading comprehension |
|
Primary Use Cases: | understanding capabilities and limitations of current language models, developing improvement techniques |
|
Limitations: | not intended for downstream applications without risk evaluation |
|
Considerations: | Model can produce harmful content or incorrect information. |
|
|
Supported Languages | en (>20 languages including English, but primarily trained on English) |
|
Training Details |
Data Sources: | CCNet (67%), C4 (15%), GitHub (4.5%), Wikipedia (4.5%), Books (4.5%), ArXiv (2.5%), Stack Exchange (2%) |
|
|
Responsible Ai Considerations |
Fairness: | The model might exhibit biases related to gender, religion, race, sexual orientation, age, nationality, disability, and physical appearance, reflecting biases from the training data. |
|
Transparency: | The model's performance and evaluation metrics are detailed in published resources. |
|
Accountability: | Meta AI's FAIR team is responsible for the model's outputs. |
|
Mitigation Strategies: | Data filtering was performed using a Kneser-Ney language model and a fastText linear classifier to remove offensive content. |
|
|
Release Notes |
Version: | |
Date: | |
Notes: | Initial release of the model. |
|
|
|