Announcing the Availability of Llama 3.1 Models in Amazon Bedrock

Meta has unveiled its most advanced models yet, the Llama 3.1 series, now available in Amazon Bedrock. These models, known as Llama 3.1, come in three sizes: 8B, 70B, and 405B parameters, showcasing state-of-the-art performance across various industry benchmarks. The new models offer groundbreaking capabilities for generative artificial intelligence (AI) applications.

Enhanced Context Length and Multilingual Support

One of the most significant improvements in the Llama 3.1 models is their support for a 128K context length, a substantial increase from the 8K tokens in the previous Llama 3 models. This extended context length enhances the models’ ability to handle more extensive and complex inputs, making them ideal for applications requiring long-form text generation and advanced reasoning. Moreover, these models excel in multilingual dialogue, supporting eight languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

Llama 3.1 Models: Detailed Overview

Llama 3.1 405B (Preview)

The Llama 3.1 405B model is the largest publicly available large language model (LLM) according to Meta. It is designed for enterprise-level applications and research and development (R&D). This model is particularly effective for tasks such as synthetic data generation, multilingual translation, coding, math, and advanced reasoning. It excels in generating long-form text, understanding context, and making decisions.

Llama 3.1 70B

The Llama 3.1 70B model is ideal for content creation, conversational AI, language understanding, and enterprise applications. It is proficient in text summarization, text classification, sentiment analysis, dialogue systems, and code generation. This model is also suitable for tasks that require following instructions and understanding nuances in language.

Llama 3.1 8B

The Llama 3.1 8B model is designed for applications with limited computational resources. It is particularly effective in text summarization, text classification, sentiment analysis, and language translation, requiring low-latency inferencing.

Performance Evaluation

Meta has rigorously evaluated the Llama 3.1 models across more than 150 benchmark datasets, covering a wide range of languages and extensive human evaluations. The performance results show that Llama 3.1 outperforms the previous Llama 3 models in every major benchmarking category.

Responsible AI and Data Governance

Amazon Bedrock provides robust features for building secure and reliable generative AI applications. Users can take advantage of Llama 3.1’s responsible AI capabilities, combined with data governance and model evaluation features, to ensure safe and ethical AI interactions.

Guardrails for Amazon Bedrock

Guardrails allow users to create multiple configurations tailored to specific use cases, promoting safe interactions between users and generative AI applications. These guardrails help monitor and analyze user inputs and model responses, detect hallucinations, and evaluate performance across different models.

Model Evaluation on Amazon Bedrock

Amazon Bedrock enables users to evaluate, compare, and select the best Llama models for their use cases. Users can choose between automatic evaluation with predefined metrics or human evaluation workflows for subjective metrics. The platform also provides built-in curated datasets or the option to use custom datasets.

Getting Started with Llama 3.1 Models

To start using Llama 3.1 models in Amazon Bedrock, users can access the Amazon Bedrock console in the US West (Oregon) Region. For the latest Llama 3.1 models, users need to request access separately for Llama 3.1 8B Instruct or Llama 3.1 70B Instruct. For access to the preview of the Llama 3.1 405B Instruct model, users should contact their AWS account team or submit a support ticket via the AWS Management Console.

Testing Llama 3.1 Models

Users can test the Llama 3.1 models in the Amazon Bedrock console by choosing "Text" or "Chat" under "Playgrounds" in the left menu pane. By selecting "Meta" as the category, users can choose from Llama 3.1 8B Instruct, Llama 3.1 70B Instruct, or Llama 3.1 405B Instruct models.

Using AWS CLI and SDKs

Users can also access the models using code examples in the AWS Command Line Interface (AWS CLI) and AWS SDKs. Here is a sample AWS CLI command:

bash<br /> aws bedrock-runtime invoke-model \<br /> --model-id meta.llama3-1-405b-instruct-v1:0 \<br /> --body "{\"prompt\":\" [INST]You are a very intelligent bot with exceptional critical thinking[/INST] I went to the market and bought 10 apples. I gave 2 apples to your friend and 2 to the helper. I then went and bought 5 more apples and ate 1. How many apples did I remain with? Let's think step by step.\",\"max_gen_len\":512,\"temperature\":0.5,\"top_p\":0.9}" \<br /> --cli-binary-format raw-in-base64-out \<br /> --region us-east-1 \<br /> invoke-model-output.txt<br />

Python Code Example

Here is a Python code example demonstrating how to use the Amazon Bedrock Converse API for text generation:

“`python
import boto3
from botocore.exceptions import ClientError

Create a Bedrock Runtime client in the AWS Region you want to use.

client = boto3.client("bedrock-runtime", region_name="us-east-1")

Set the model ID, e.g., Llama 3 8b Instruct.

model_id = "meta.llama3-1-405b-instruct-v1:0"

Start a conversation with the user message.

user_message = "Describe the purpose of a ‘hello world’ program in one line."
conversation = [
{
"role": "user",
"content": [{"text": user_message}],
}
]

try:

Send the message to the model, using a basic inference configuration.

response = client.converse(<br />
    modelId=model_id,<br />
    messages=conversation,<br />
    inferenceConfig={"maxTokens": 512, "temperature": 0.5, "topP": 0.9},<br />
)<br />

Extract and print the response text.

response_text = response["output"]["message"]["content"][0]["text"]<br />
print(response_text)<br />

except (ClientError, Exception) as e:
print(f"ERROR: Can’t invoke ‘{model_id}’. Reason: {e}")
exit(1)
“`

Integration with Amazon SageMaker JumpStart

All Llama 3.1 models (8B, 70B, and 405B) are available in Amazon SageMaker JumpStart. Users can discover and deploy Llama 3.1 models with a few clicks in Amazon SageMaker Studio or programmatically through the SageMaker Python SDK. SageMaker features such as SageMaker Pipelines and SageMaker Debugger help provide data security.

Fine-Tuning and Custom Models

Fine-tuning for Llama 3.1 models in Amazon Bedrock and Amazon SageMaker JumpStart will be available soon. Users will be able to import custom models into Amazon Bedrock and build fine-tuned models in SageMaker JumpStart.

Deployment on AWS

For customers who prefer self-managed machine learning workflows, AWS Trainium and AWS Inferentia-powered Amazon Elastic Compute Cloud (Amazon EC2) instances offer high performance and cost-effective deployment of Llama 3.1 models on AWS.

Customer Insights

Parkin Kent, Business Development Manager at Meta, highlights the power of the collaboration between Meta and Amazon in pushing the boundaries of generative AI. Notable customers like Nomura and TaskUs are leveraging Llama models in Amazon Bedrock to democratize generative AI and enhance their services.

Availability

Llama 3.1 8B and 70B models from Meta are now generally available, and the Llama 405B model is in preview in Amazon Bedrock in the US West (Oregon) Region. To request access to the preview of Llama 3.1 405B, users should contact their AWS account team or submit a support ticket. Check the full Region list for future updates.

Conclusion

The Llama 3.1 models represent a significant advancement in AI capabilities, offering enhanced performance and new features for generative AI applications. Users can start experimenting with these models in Amazon Bedrock and take advantage of the robust security and evaluation features provided by AWS.

Give Llama 3.1 a try in the Amazon Bedrock console today, and share your feedback through AWS re:Post or your usual AWS Support contacts. Visit the community.aws site for deep-dive technical content and discover how the Builder communities are using Amazon Bedrock in their solutions.

Update

As of July 23, 2024, the blog post has been updated to include new screenshots for model access and a customer video featuring TaskUs.