BLOOM

A group of more than 1000 AI researchers have developed a massive language model larger than GPT-3. This model is free for everyone to use and comes with all the resources and documentation necessary to understand how it works. The project started in March, but the work behind preparing datasets spanned months before that. BLOOM consists of 384 graphic cards with 80 gigabytes each running on Jean Zay supercomputer in France. It has 176 billion parameters, 70 layers, 112 attention heads per layer, hidden dimensionality of 14336 and 2048 token sequence length. In addition to this, it uses ALiBi positional embeddings and GeLU activation function while being trained on various languages such as English, Spanish and Italian along with programming codes. Big Science is an open collaboration initiated by HuggingFace Inc., GENCI (Grand Équipement National de Calcul Intensif) & IDRIS (Institut du Développement et des Ressources en Informatique Scientifique). The research workshop brings together industry professionals from different disciplines such as AI, NLP social science ethics etc., who are interested in exploring public policy implications of their findings.