Alternatives to GPT-J

With T5, we propose re-framing all NLP into a unified format where the input and the output are always text strings. This is in contrast to BERT models which can only output a class label, or a span from the input. Our text-totext framework allows us use the same model and loss function on any NLP task. This includes machine translation, document summary, question answering and classification tasks. We can also apply T5 to regression by training it to predict a string representation of a numeric value instead of the actual number.

Pythia combines interpretationability analysis and scaling law to understand how knowledge evolves and develops during training with autoregressive transformators.

PygmalionAI, a community of open-source projects based upon EleutherAI’s GPT-J 6B models and Meta’s LLaMA model, was founded in 2009. Pygmalion AI is designed for roleplaying and chatting. The 7B variant of the Pygmalion AI is currently actively supported. It is based on Meta AI’s LLaMA AI model. Pygmalion's chat capabilities are superior to larger language models that require much more resources. Our curated datasets of high-quality data on roleplaying ensure that your bot is the best RP partner. The model weights as well as the code used to train the model are both open-source. You can modify/re-distribute them for any purpose you like. Pygmalion and other language models run on GPUs because they require fast memory and massive processing to produce coherent text at a reasonable speed.

StableLM: Stability AI language models StableLM builds upon our experience with open-sourcing previous language models in collaboration with EleutherAI. This nonprofit research hub. These models include GPTJ, GPTNeoX and the Pythia Suite, which were all trained on The Pile dataset. Cerebras GPT and Dolly-2 are two recent open-source models that continue to build upon these efforts. StableLM was trained on a new dataset that is three times bigger than The Pile and contains 1.5 trillion tokens. We will provide more details about the dataset at a later date. StableLM's richness allows it to perform well in conversational and coding challenges, despite the small size of its dataset (3-7 billion parameters, compared to GPT-3's 175 billion). The development of Stable LM 3B broadens the range of applications that are viable on the edge or on home PCs. This means that individuals and companies can now develop cutting-edge technologies with strong conversational capabilities – like creative writing assistance – while keeping costs low and performance high.

Production-ready AI models that are fast and accurate. High-availability inference API that leverages the most advanced NVIDIA GPUs. We have selected the most popular open-source natural language processing models (NLP) and deployed them for the community. You can fine-tune your models (including GPT-J) or upload your custom models. Then, deploy them to production. Upload your AI models, including GPT-J, to your dashboard and immediately use them in production.

Dolly is an inexpensive LLM that demonstrates a surprising amount of the capabilities of ChatGPT. Whereas the work from the Alpaca team showed that state-of-the-art models could be coaxed into high quality instruction-following behavior, we find that even years-old open source models with much earlier architectures exhibit striking behaviors when fine tuned on a small corpus of instruction training data. Dolly uses an open source model with 6 billion parameters from EleutherAI, which is modified to include new capabilities like brainstorming and text creation that were not present in the original.

A model parallel autoregressive transformator implementation on GPUs based on the DeepSpeed Library. This repository contains EleutherAI’s library for training large language models on GPUs. Our current framework is based upon NVIDIA's Megatron Language Model, and has been enhanced with techniques from DeepSpeed, as well as some novel improvements. This repo is intended to be a central and accessible place for techniques to train large-scale autoregressive models and to accelerate research into large scale training.

The expansion of large language model has led to significant advancements in natural language processing, understanding and generation. This study introduces a new system that uses Ascend 910 AI processing units and the MindSpore framework in order to train a language with over one trillion parameters, 1.085T specifically, called PanGu-Sigma. This model, which builds on the foundation laid down by PanGu-alpha transforms the traditional dense Transformer model into a sparse model using a concept called Random Routed Experts. The model was trained efficiently on a dataset consisting of 329 billion tokens, using a technique known as Expert Computation and Storage Separation. This led to a 6.3 fold increase in training performance via heterogeneous computer. The experiments show that PanGu-Sigma is a new standard for zero-shot learning in various downstream Chinese NLP tasks.

VideoPoet, a simple modeling technique, can convert any large language model or autoregressive model into a high quality video generator. It is composed of a few components. The autoregressive model learns from video, image, text, and audio modalities in order to predict the next audio or video token in the sequence. The LLM training framework introduces a mixture of multimodal generative objectives, including text to video, text to image, image-to video, video frame continuation and inpainting/outpainting, styled video, and video-to audio. Moreover, these tasks can be combined to provide additional zero-shot capabilities. This simple recipe shows how language models can edit and synthesize videos with a high level of temporal consistency.

The next generation of the large language model. This release includes modelweights and starting code to pretrained and fine tuned Llama languages models, ranging from 7B-70B parameters. Llama 1 models have a context length of 2 trillion tokens. Llama 2 models have a context length double that of Llama 1. The fine-tuned Llama 2 models have been trained using over 1,000,000 human annotations. Llama 2, a new open-source language model, outperforms many other open-source language models in external benchmarks. These include tests of reasoning, coding and proficiency, as well as knowledge tests. Llama 2 has been pre-trained using publicly available online data sources. Llama-2 chat, a fine-tuned version of the model, is based on publicly available instruction datasets, and more than 1 million human annotations. We have a wide range of supporters in the world who are committed to our open approach for today's AI. These companies have provided early feedback and have expressed excitement to build with Llama 2

We are developing large language models that are efficient for India's diverse cultural diversity and enabling GenAI applications with bespoke enterprise models. We are building a platform for enterprise-grade apps that allows you to develop and evaluate them. We believe that open-source can accelerate AI innovation. We will be contributing open-source datasets and models, and leading efforts for large data curation projects in the public-good space. We are a dynamic team of AI experts, combining expertise in research, product design, engineering and business operations. Our diverse backgrounds are united by a commitment to excellence in science, and creating societal impact. We create an environment in which tackling complex tech problems is not only a job but a passion.

Jurassic-2 is the latest generation AI21 Studio foundation models. It's a game changer in the field AI, with new capabilities and top-tier quality. We're also releasing task-specific APIs with superior reading and writing capabilities. AI21 Studio's focus is to help businesses and developers leverage reading and writing AI in order to build real-world, tangible products. The release of Task-Specific and Jurassic-2 APIs marks two significant milestones. They will enable you to bring generative AI into production. Jurassic-2 (or J2, as we like to call it) is the next generation of our foundation models with significant improvements in quality and new capabilities including zero-shot instruction-following, reduced latency, and multi-language support. Task-specific APIs offer developers industry-leading APIs for performing specialized reading and/or writing tasks.

GPT-5 is OpenAI's Generative Pretrained Transformer. It is a large-language model (LLM), which is still in development. LLMs have been trained to work with massive amounts of text and can generate realistic and coherent texts, translate languages, create different types of creative content and answer your question in a way that is informative. It's still not available to the public. OpenAI has not announced a release schedule, but some believe it could launch in 2024. It's expected that GPT-5 will be even more powerful. GPT-4 has already proven to be impressive. It is capable of writing creative content, translating languages and generating text of human-quality. GPT-5 will be expected to improve these abilities, with improved reasoning, factual accuracy and ability to follow directions.

Qwen LLM is a family of large-language models (LLMs), developed by Damo Academy, an Alibaba Cloud subsidiary. These models are trained using a large dataset of text and codes, allowing them the ability to understand and generate text that is human-like, translate languages, create different types of creative content and answer your question in an informative manner. Here are some of the key features of Qwen LLMs. Variety of sizes: Qwen's series includes sizes ranging from 1.8 billion parameters to 72 billion, offering options that meet different needs and performance levels. Open source: Certain versions of Qwen have open-source code, which is available to anyone for use and modification. Qwen is multilingual and can translate multiple languages including English, Chinese and Japanese. Qwen models are capable of a wide range of tasks, including text summarization and code generation, as well as generation and translation.

StarCoderBase and StarCoder are Large Language Models (Code LLMs), trained on permissively-licensed data from GitHub. This includes data from 80+ programming language, Git commits and issues, Jupyter Notebooks, and Git commits. We trained a 15B-parameter model for 1 trillion tokens, similar to LLaMA. We refined the StarCoderBase for 35B Python tokens. The result is a new model we call StarCoder. StarCoderBase is a model that outperforms other open Code LLMs in popular programming benchmarks. It also matches or exceeds closed models like code-cushman001 from OpenAI, the original Codex model which powered early versions GitHub Copilot. StarCoder models are able to process more input with a context length over 8,000 tokens than any other open LLM. This allows for a variety of interesting applications. By prompting the StarCoder model with a series dialogues, we allowed them to act like a technical assistant.

GPT-4 (Generative Pretrained Transformer 4) a large-scale, unsupervised language model that is yet to be released. GPT-4, which is the successor of GPT-3, is part of the GPT -n series of natural-language processing models. It was trained using a dataset of 45TB text to produce text generation and understanding abilities that are human-like. GPT-4 is not dependent on additional training data, unlike other NLP models. It can generate text and answer questions using its own context. GPT-4 has been demonstrated to be capable of performing a wide range of tasks without any task-specific training data, such as translation, summarization and sentiment analysis.

PanGu-a was developed under MindSpore, and trained on 2048 Ascend AI processors. The MindSpore Auto-parallel parallelism strategy was implemented to scale the training task efficiently to 2048 processors. This includes data parallelism as well as op-level parallelism. We pretrain PanGu-a with 1.1TB of high-quality Chinese data collected from a variety of domains in order to enhance its generalization ability. We test the generation abilities of PanGua in different scenarios, including text summarizations, question answering, dialog generation, etc. We also investigate the effects of model scaling on the few shot performances across a wide range of Chinese NLP task. The experimental results show that PanGu-a is superior in performing different tasks with zero-shot or few-shot settings.

PaLM 2 is Google's next-generation large language model, which builds on Google’s research and development in machine learning. It excels in advanced reasoning tasks including code and mathematics, classification and question-answering, translation and multilingual competency, and natural-language generation better than previous state-of the-art LLMs including PaLM. It is able to accomplish these tasks due to the way it has been built - combining compute-optimal scale, an improved dataset mix, and model architecture improvement. PaLM 2 is based on Google's approach for building and deploying AI responsibly. It was rigorously evaluated for its potential biases and harms, as well as its capabilities and downstream applications in research and product applications. It is being used to power generative AI tools and features at Google like Bard, the PaLM API, and other state-ofthe-art models like Sec-PaLM and Med-PaLM 2.

Gemma is the family of lightweight open models that are built using the same research and technology as the Gemini models. Gemma was developed by Google DeepMind, along with other teams within Google. The name is derived from the Latin gemma meaning "precious stones". We're also releasing new tools to encourage developer innovation, encourage collaboration, and guide responsible use of Gemma model. Gemma models are based on the same infrastructure and technical components as Gemini, Google's largest and most powerful AI model. Gemma 2B, 7B and other open models can achieve the best performance possible for their size. Gemma models can run directly on a desktop or laptop computer for developers. Gemma is able to surpass much larger models in key benchmarks, while adhering our rigorous standards of safe and responsible outputs.

Databricks has created an open, general purpose LLM called DBRX. DBRX is the new benchmark for open LLMs. It also provides open communities and enterprises that are building their own LLMs capabilities that were previously only available through closed model APIs. According to our measurements, DBRX surpasses GPT 3.5 and is competitive with Gemini 1.0 Pro. It is a code model that is more capable than specialized models such as CodeLLaMA 70B, and it also has the strength of a general-purpose LLM. This state-of the-art quality is accompanied by marked improvements in both training and inference performances. DBRX is the most efficient open model thanks to its finely-grained architecture of mixtures of experts (MoE). Inference is 2x faster than LLaMA2-70B and DBRX has about 40% less parameters in total and active count compared to Grok-1.

Megatron-Turing Natural Language Generation Model (MT-NLG) is the largest and most powerful monolithic English language model. It has 530 billion parameters. This 105-layer transformer-based MTNLG improves on the previous state-of-the art models in zero, one, and few shot settings. It is unmatched in its accuracy across a wide range of natural language tasks, including Completion prediction and Reading comprehension. NVIDIA has announced an Early Access Program for its managed API service in MT-NLG Mode. This program will allow customers to experiment with, employ and apply a large language models on downstream language tasks.

XLNet, a new unsupervised language representation method, is based on a novel generalized Permutation Language Modeling Objective. XLNet uses Transformer-XL as its backbone model. This model is excellent for language tasks that require long context. Overall, XLNet achieves state of the art (SOTA) results in various downstream language tasks, including question answering, natural languages inference, sentiment analysis and document ranking.

Aya is an open-source, state-of-the art, massively multilingual large language research model (LLM), which covers 101 different languages. This is more than twice the number of languages that are covered by open-source models. Aya helps researchers unlock LLMs' powerful potential for dozens of cultures and languages that are largely ignored by the most advanced models available today. We open-source both the Aya Model, as well as the most comprehensive multilingual instruction dataset with 513 million words covering 114 different languages. This data collection contains rare annotations by native and fluent speakers from around the world. This ensures that AI technology is able to effectively serve a global audience who have had limited access up until now.

Qwen-7B, also known as Qwen-7B, is the 7B-parameter variant of the large language models series Qwen. Tongyi Qianwen, proposed by Alibaba Cloud. Qwen-7B, a Transformer-based language model, is pretrained using a large volume data, such as web texts, books, code, etc. Qwen-7B is also used to train Qwen-7B Chat, an AI assistant that uses large models and alignment techniques. The Qwen-7B features include: Pre-trained with high quality data. We have pretrained Qwen-7B using a large-scale, high-quality dataset that we constructed ourselves. The dataset contains over 2.2 trillion tokens. The dataset contains plain texts and codes and covers a wide range domains including general domain data as well as professional domain data. Strong performance. We outperform our competitors in a series benchmark datasets that evaluate natural language understanding, mathematics and coding. And more.

GPT-3.5 is the next evolution to GPT 3 large language model, OpenAI. GPT-3.5 models are able to understand and generate natural languages. There are four main models available with different power levels that can be used for different tasks. The main GPT-3.5 models can be used with the text completion endpoint. There are models that can be used with other endpoints. Davinci is the most versatile model family. It can perform all tasks that other models can do, often with less instruction. Davinci is the best choice for applications that require a deep understanding of the content. This includes summarizations for specific audiences and creative content generation. These higher capabilities mean that Davinci is more expensive per API call and takes longer to process than other models.

Instruction-following models such as GPT-3.5 (text-DaVinci-003), ChatGPT, Claude, and Bing Chat have become increasingly powerful. These models are now used by many users, and some even for work. However, despite their widespread deployment, instruction-following models still have many deficiencies: they can generate false information, propagate social stereotypes, and produce toxic language. It is vital that the academic community engages in order to make maximum progress towards addressing these pressing issues. Unfortunately, doing research on instruction-following models in academia has been difficult, as there is no easily accessible model that comes close in capabilities to closed-source models such as OpenAI's text-DaVinci-003. We are releasing our findings about an instruction-following language model, dubbed Alpaca, which is fine-tuned from Meta's LLaMA 7B model.

The training of state-of-the art language models is extremely difficult. They require large compute budgets, complex distributed computing techniques and deep ML knowledge. Few organizations are able to train large language models from scratch. The number of organizations that do not open source their results is increasing, even though they have the expertise and resources to do so. We at Cerebras believe in open access to the latest models. Cerebras is proud to announce that Cerebras GPT, a family GPT models with 111 million to thirteen billion parameters, has been released to the open-source community. These models are trained using the Chinchilla Formula and provide the highest accuracy within a given computing budget. Cerebras GPT has faster training times and lower training costs. It also consumes less power than any other publicly available model.

You can use advanced language models and coding to solve a variety of problems. To build cutting-edge applications, leverage large-scale, generative AI models that have deep understandings of code and language to allow for new reasoning and comprehension. These coding and language models can be applied to a variety use cases, including writing assistance, code generation, reasoning over data, and code generation. Access enterprise-grade Azure security and detect and mitigate harmful use. Access generative models that have been pretrained with trillions upon trillions of words. You can use them to create new scenarios, including code, reasoning, inferencing and comprehension. A simple REST API allows you to customize generative models with labeled information for your particular scenario. To improve the accuracy of your outputs, fine-tune the hyperparameters of your model. You can use the API's few-shot learning capability for more relevant results and to provide examples.

Code Llama, a large-language model (LLM), can generate code using text prompts. Code Llama, the most advanced publicly available LLM for code tasks, has the potential to improve workflows for developers and reduce the barrier for those learning to code. Code Llama can be used to improve productivity and educate programmers to create more robust, well documented software. Code Llama, a state-of the-art LLM, is capable of generating both code, and natural languages about code, based on both code and natural-language prompts. Code Llama can be used for free in research and commercial purposes. Code Llama is a new model that is built on Llama 2. It is available in 3 models: Code Llama is the foundational model of code; Codel Llama is a Python-specific language. Code Llama-Instruct is a finely tuned natural language instruction interpreter.

ChatGPT is an OpenAI language model. It can generate human-like responses to a variety prompts, and has been trained on a wide range of internet texts. ChatGPT can be used to perform natural language processing tasks such as conversation, question answering, and text generation. ChatGPT is a pretrained language model that uses deep-learning algorithms to generate text. It was trained using large amounts of text data. This allows it to respond to a wide variety of prompts with human-like ease. It has a transformer architecture that has been proven to be efficient in many NLP tasks. ChatGPT can generate text in addition to answering questions, text classification and language translation. This allows developers to create powerful NLP applications that can do specific tasks more accurately. ChatGPT can also process code and generate it.

Vicuna-13B, an open-source chatbot, is trained by fine-tuning LLaMA using user-shared conversations from ShareGPT. Vicuna-13B's preliminary evaluation using GPT-4, as a judge, shows that it achieves a quality of more than 90%* for OpenAI ChatGPT or Google Bard and outperforms other models such as LLaMA or Stanford Alpaca. Vicuna-13B costs around $300 to train. The online demo and the code, along with weights, are available to non-commercial users.

Introducing DeepSeek LLM - an advanced language model with 67 billion parameters. It was trained from scratch using a massive dataset of 2 trillion tokens, both in English and Chinese. To encourage research, we made DeepSeek LLM 67B Base and DeepSeek LLM 67B Chat available as open source to the research community.

Samsung Gauss, a new AI-model developed by Samsung Electronics, is a powerful AI tool. It is a large-language model (LLM) which has been trained using a massive dataset. Samsung Gauss can generate text, translate different languages, create creative content and answer questions in a helpful way. Samsung Gauss, which is still in development, has already mastered many tasks, including Follow instructions and complete requests with care. Answering questions in an informative and comprehensive way, even when they are open-ended, challenging or strange. Creating different creative text formats such as poems, code, musical pieces, emails, letters, etc. Here are some examples to show what Samsung Gauss is capable of: Translation: Samsung Gauss is able to translate text between many languages, including English and German, as well as Spanish, Chinese, Japanese and Korean. Coding: Samsung Gauss can generate code.

Stability AI, in collaboration with its CarperAI Lab, is proud to announce FreeWilly1 (and its successor FreeWilly2), two powerful, new Large Language Models. Both models show exceptional reasoning abilities across a variety of benchmarks. FreeWilly1 leverages the original LLaMA 65B foundation model and was carefully fine-tuned with a new synthetically-generated dataset using Supervised Fine-Tune (SFT) in standard Alpaca format. FreeWilly2 uses the LLaMA 70B foundation model in order to achieve a performance that is comparable with GPT-3.5 on some tasks. The FreeWilly models were inspired by Microsoft's "Orca: Progressive Learning from Complex Explanation traces of GPT-4" paper. While our data generation processes are similar, our data sources differ.

RoBERTa is based on BERT's language-masking strategy. The system learns to predict hidden sections of text in unannotated language examples. RoBERTa was implemented in PyTorch and modifies key hyperparameters of BERT. This includes removing BERT’s next-sentence-pretraining objective and training with larger mini-batches. This allows RoBERTa improve on the masked-language modeling objective, which is comparable to BERT. It also leads to improved downstream task performance. We are also exploring the possibility of training RoBERTa with a lot more data than BERT and for a longer time. We used both existing unannotated NLP data sets as well as CC-News which was a new set of public news articles.

Amazon Bedrock is an innovative service that allows FMs to be accessed by leading AI startups as well as Amazon via API. Bedrock makes it easy for customers to create and scale AI-based generative applications, using FMs. It democratizes access for all builders. Bedrock allows users to access a variety of powerful FMs that can be used for text or images, including Amazon Titan FMs. This is done through a scalable and reliable AWS managed service. Amazon Titan FMs have been trained on large datasets and are powerful general-purpose models. You can use them as-is or customize them privately with your own data to accomplish a specific task without having to annotate large volumes of data. Titan Text is a large language model that can be used for tasks like summarization, text creation (for example creating a blog), classification, open ended Q&A and information extraction. Automate natural language tasks, such as text generation and summarization.

This repository contains a research preview of LongLLaMA. It is a large language-model capable of handling contexts up to 256k tokens. LongLLaMA was built on the foundation of OpenLLaMA, and fine-tuned with the Focused Transformer method. LongLLaMA code was built on the foundation of Code Llama. We release a smaller base variant of the LongLLaMA (not instruction-tuned) on a permissive licence (Apache 2.0), and inference code that supports longer contexts for hugging face. Our model weights are a drop-in replacement for LLaMA (for short contexts up to 2048 tokens) in existing implementations. We also provide evaluation results, and comparisons with the original OpenLLaMA model.

It's as simple as changing one line in code to switch. Feature parity with industry-standard APIs ensures that your product runs faster and works the same way. GooseAI is a fully managed NLP as-a-Service delivered via API. In this respect, it is comparable to OpenAI. It is compatible with OpenAI’s completion API. Our state-of the-art selection GPT-based language models, uncompromising speed, and flexible alternative to your current provider will give you a jumpstart in your next project. We are proud to be able offer prices that are up to 70% lower than other providers and still deliver the same or better performance. Geese are integral to the ecosystem, just as the Mitochondria powerhouses cells. We were inspired by their beauty and elegance to fly high, just like geese.

Chinchilla has a large language. Chinchilla has the same compute budget of Gopher, but 70B more parameters and 4x as much data. Chinchilla consistently and significantly outperforms Gopher 280B, GPT-3 175B, Jurassic-1 178B, and Megatron-Turing (530B) in a wide range of downstream evaluation tasks. Chinchilla also uses less compute to perform fine-tuning, inference and other tasks. This makes it easier for downstream users to use. Chinchilla reaches a high-level average accuracy of 67.5% for the MMLU benchmark. This is a greater than 7% improvement compared to Gopher.

The ability of large language models to learn in zero- and few shots, despite being trained for hundreds of thousands or even millions of days, has been remarkable. These models are expensive to replicate, due to their high computational cost. The few models that are available via APIs do not allow access to the full weights of the model, making it difficult to study. Open Pre-trained Transformers is a suite decoder-only pre-trained transforms with parameters ranging from 175B to 125M. We aim to share this fully and responsibly with interested researchers. We show that OPT-175B has a carbon footprint of 1/7th that of GPT-3. We will also release our logbook, which details the infrastructure challenges we encountered, as well as code for experimenting on all of the released model.

Smaug 72B is an open-source large-language model (LLM), which is known for its key features. High Performance: It is currently ranked first on the Hugging face Open LLM leaderboard. This model has surpassed models such as GPT-3.5 across a range of benchmarks. This means that it excels in tasks such as understanding, responding to and generating text similar to human speech. Open Source: Smaug-72B, unlike many other advanced LLMs is available to anyone for free use and modification, fostering collaboration, innovation, and creativity in the AI community. Focus on Math and Reasoning: It excels at handling mathematical and reasoning tasks. This is attributed to the unique fine-tuning technologies developed by Abacus, the creators Smaug 72B. Based on Qwen 72B: This is a finely tuned version of another powerful LLM, called Qwen 72B, released by Alibaba. It further improves its capabilities. Smaug-72B is a significant advance in open-source AI.

Through scientific rigor and human insight, healthcare breakthroughs can change the world, bringing hope to humanity. We believe that AI can help in this area, through collaboration between researchers, healthcare organisations, and the wider ecosystem. Today, we are sharing exciting progress in these initiatives with the announcement that Google's large language model (LLM) for medical applications, called Med PaLM 2, will be available to a limited number of customers. In the coming weeks, it will be available to a small group of Google Cloud users for limited testing. We will explore use cases, share feedback, and investigate safe, responsible and meaningful ways to utilize this technology. Med-PaLM 2, which harnesses Google's LLMs aligned with the medical domain, is able to answer medical questions more accurately and safely. Med-PaLM 2 is the first LLM that has performed at an "expert" level on the MedQA dataset consisting of US Medical Licensing Examination-style questions.

A MLLM system that accepts any form of referral and grounds anything in response. Ferret Model- Hybrid Region representation + Spatial-aware visual sampler allows for fine-grained and open vocabulary referring and grounding. GRIT Dataset - A large-scale, hierarchical, robust ground-and refer instruction tuning dataset. Ferret Bench - A multimodal benchmark that requires Referring/Grounding as well as Semantics, Knowledge and Reasoning.

CodeGen is a model for program synthesis that is open-source. Trained on TPU v4. OpenAI Codex is competitive with TPU-v4.

OpenLLaMA, a permissively-licensed open source reproduction of Meta AI’s LLaMA 7B, is trained on the RedPajama data set. Our model weights are a drop-in replacement for LLaMA7B in existing implementations. We also offer a smaller 3B version of the LLaMA Model.

Gemini Ultra is an advanced new language model by Google DeepMind. It is the most powerful and largest model in the Gemini Family, which includes Gemini Pro & Gemini Nano. Gemini Ultra was designed to handle highly complex tasks such as machine translation, code generation, and natural language processing. It is the first language model that has outperformed human experts in the Massive Multitask Language Understanding test (MMLU), achieving a score 90%.

Pre-trained models of language have achieved state-of the-art results for various Natural Language Processing (NLP). GPT-3 has demonstrated that scaling up language models pre-trained can further exploit their immense potential. Recently, a framework named ERNIE 3.0 for pre-training large knowledge enhanced models was proposed. This framework trained a model that had 10 billion parameters. ERNIE 3.0 performed better than the current state-of-the art models on a variety of NLP tasks. In order to explore the performance of scaling up ERNIE 3.0, we train a hundred-billion-parameter model called ERNIE 3.0 Titan with up to 260 billion parameters on the PaddlePaddle platform. We also design a self supervised adversarial and a controllable model language loss to make ERNIE Titan generate credible texts.

FLAN-T5 was released in the paper Scaling Instruction-Finetuned Language Models - it is an enhanced version of T5 that has been finetuned in a mixture of tasks.

Phi-2 is a 2.7-billion-parameter language-model that shows outstanding reasoning and language-understanding capabilities. It represents the state-of-the art performance among language-base models with less than thirteen billion parameters. Phi-2 can match or even outperform models 25x larger on complex benchmarks, thanks to innovations in model scaling. Phi-2's compact size makes it an ideal playground for researchers. It can be used for exploring mechanistic interpretationability, safety improvements or fine-tuning experiments on a variety tasks. We have included Phi-2 in the Azure AI Studio catalog to encourage research and development of language models.

Mixtral 8x7B has open weights and is a high quality sparse mixture expert model (SMoE). Licensed under Apache 2.0. Mixtral outperforms Llama 70B in most benchmarks, with 6x faster Inference. It is the strongest model with an open-weight license and the best overall model in terms of cost/performance tradeoffs. It matches or exceeds GPT-3.5 in most standard benchmarks.