Today marks a significant milestone for Amazon Web Services (AWS) as they announce the general availability of a new broker type, Express brokers, for Amazon Managed Streaming for Apache Kafka (Amazon MSK). This new type of broker is engineered to enhance the performance of Amazon MSK by offering up to three times the throughput per broker, scaling up to 20 times faster, and reducing recovery times by 90% compared to the existing Standard brokers. This development is aimed at providing users with an optimized and more efficient streaming solution while maintaining compatibility with the existing client applications and ensuring low latency performance.
Amazon MSK is recognized as a fully-managed service on the AWS platform, designed to simplify the process of building and running scalable and highly available applications that leverage Apache Kafka. The introduction of Express brokers is a step towards enhancing compute and storage elasticity for Kafka applications, especially when using Amazon MSK provisioned clusters.
Key Features of Express Brokers
- Simplified Operations with Hands-Free Storage Management: Express brokers offer unlimited storage capabilities without the need for preprovisioning, thus eliminating bottlenecks related to disk storage. This advancement simplifies cluster sizing, as it only requires evaluation of ingress and egress throughput divided by the recommended per-broker throughput. By removing the need for proactive disk capacity monitoring and scaling, cluster management becomes more straightforward, thereby improving resilience by eliminating a potential point of failure.
- Increased Throughput with Fewer Brokers: Express brokers are designed to handle higher throughput per broker, allowing users to deploy smaller clusters for equivalent workloads. With Standard brokers, throughput considerations must include client traffic and background operations; however, Express brokers utilize opinionated settings and resource isolation to safely manage higher ingress rates without compromising performance or availability during cluster events.
- Faster Scaling Capabilities: The design of Express brokers facilitates reduced data movement during scaling, making them up to 20 times faster than their Standard counterparts. This rapid scaling capability allows for quick and reliable cluster resizing, enabling users to monitor each broker’s ingress throughput capacity and add brokers within minutes, thereby eliminating the need for over-provisioning in anticipation of traffic spikes.
- Greater Resilience with Rapid Recovery: Express brokers are built for mission-critical applications that require high resilience. They come preconfigured with best-practice defaults, including 3-way replication (RF=3), which significantly reduces failures due to misconfiguration. The brokers are also capable of recovering 90% faster from transient failures compared to standard Apache Kafka brokers, with rebalancing and recovery processes utilizing minimal cluster resources. This reduces the risk of increased resource utilization and the need for continuous monitoring when right-sizing clusters.
Choosing the Right Broker for Your Needs
Amazon MSK provides various options depending on your workload and personal preference:
- MSK Provisioned: Offers the most flexibility in terms of configuration range.
- MSK Serverless: Provides flexibility with an emphasis on compute-only capacity management.
- Standard Brokers: Requires customer-managed cluster rebalancing.
- Express Brokers: Managed by MSK, offering the fastest cluster rebalancing.
Express brokers present a cost-effective, resilient, and low-overhead option for managing Kafka workloads. For those who prefer a fully abstracted Apache Kafka experience without any infrastructure management, Amazon MSK Serverless is available. This option scales automatically and operates on a pay-per-use consumption model.
Getting Started with Express Brokers
To begin using Express brokers, AWS provides a Sizing and Pricing worksheet to help you estimate the cluster size required to accommodate your workload and provide a rough estimate of the total monthly cost. The primary factor influencing the size of your cluster is the throughput requirements of your workload, although other elements such as partition and connection count should also be considered.
For instance, if your application demands 30 MBps of data ingress and 80 MBps of data egress capacity, three express.m7g.large brokers can meet your throughput needs. AWS provides guidelines on the recommended maximum ingress, egress, and partition counts per instance size to ensure sustainable and safe operations.
Instance Size Recommendations
- express.m7g.large: 15.6 MBps ingress, 31.2 MBps egress
- express.m7g.4xlarge: 124.9 MBps ingress, 249.8 MBps egress
- express.m7g.16xlarge: 500.0 MBps ingress, 1000.0 MBps egress
Once you have determined the necessary number and size of Express brokers for your workload, you can create an Amazon MSK provisioned cluster via the AWS Management Console or by using the CreateCluster API. During the creation process, select the Express brokers option and choose the desired compute capacity for the broker. The setup supports Apache Kafka version 3.6.0 and Graviton-based instances, without the need for preprovisioned storage.
To further customize your cluster configurations, visit the Express broker configurations section in the Amazon MSK developer guide. If you prefer using the AWS Command Line Interface (AWS CLI) for cluster creation, the create-cluster command is available, along with a JSON file specifying the subnets for distributing broker nodes.
Scaling and Partition Management
Express brokers support both vertical and horizontal scaling. Vertical scaling involves changing the instance size to double throughput without requiring partition reassignment. Horizontal scaling, which involves adding brokers in sets of three, requires partition reassignment for new brokers to handle traffic. A significant advantage of Express brokers is the ability to add brokers and rebalance partitions within minutes, compared to several hours required for Standard brokers.
Additional Considerations
- Data Migration: You can migrate data from an existing Kafka or MSK cluster to a new cluster with Express brokers using Amazon MSK Replicator. This tool copies both data and metadata to the new cluster.
- Monitoring: Cluster monitoring is possible at both the cluster and broker levels with Amazon CloudWatch metrics. Open monitoring with Prometheus is also available, utilizing the JMX Exporter and Node Exporter.
- Security: AWS Key Management Service (KMS) integrates with Amazon MSK to provide server-side encryption for Express brokers. Users can specify a KMS key for data encryption at rest, or Amazon MSK can create a managed key automatically.
Availability and Pricing
Express brokers are now available in several AWS regions, including US East (Ohio and N. Virginia), US West (Oregon), Asia Pacific (Singapore, Sydney, and Tokyo), and Europe (Frankfurt, Ireland, and Stockholm). Pricing for Express brokers involves an hourly rate for broker instance usage, billed at one-second resolution, with costs varying based on the broker instance size and active brokers in your MSK clusters. Additional charges apply for data written to Express brokers, billed at per-byte resolution. For detailed pricing information, visit the Amazon MSK pricing page.
Express brokers represent a promising development for users looking to enhance the efficiency and performance of their streaming applications on Amazon MSK, offering a combination of cost savings, improved resilience, and reduced operational overhead.
For more Information, Refer to this article.
