team, GPT-Neo is a codebase for creating transformer-based language models. It allows users to create and scale up their models in a similar style to GPT2 and GPT3, using the mesh-tensorflow library for model and data parallelism. With this tool, you can easily increase your model’s size to full GPT3 sizes (and even beyond!), making it easier than ever before to experiment with powerful language understanding algorithms.