AWS CodeBuild Introduces Parallel Test Execution to Accelerate Your Development Process

Amazon Web Services (AWS) has announced a significant enhancement to its AWS CodeBuild service: the support for parallel test execution. This new feature allows developers to run test suites concurrently, thereby reducing build times and enhancing overall efficiency. In this detailed report, we will explore what this means for developers and how you can leverage this feature to optimize your development workflow.

Understanding the Impact of Parallel Test Execution

With the introduction of parallel test execution, AWS CodeBuild enables developers to perform testing tasks concurrently. This advancement is particularly beneficial for projects with extensive test suites that traditionally consume a substantial amount of time when executed sequentially. For instance, in a demonstration project developed to showcase this capability, the total test time decreased from 35 minutes to just six minutes, including the time required to set up the environments.

The Challenge of Long Test Times

Long test times have long been a challenge for developers, especially those working with continuous integration (CI) systems. As projects grow in complexity, both in terms of codebase and team size, the time needed to execute comprehensive test suites can increase dramatically. This extension in test duration often delays the delivery of new features and bug fixes, while also impacting developer productivity by requiring them to wait for build results before proceeding further.

Such delays can lead to frustration and erode trust in the CI process, ultimately slowing down the entire software delivery cycle. Moreover, prolonged test durations can result in resource contention, increased costs due to wasted computing power, and reduced overall efficiency.

How Parallel Test Execution Works in CodeBuild

By supporting parallel test execution, CodeBuild allows tests to run simultaneously across multiple build environments. This is achieved through a method known as sharding, where each build node executes a subset of the test suite independently. CodeBuild provides environment variables that identify the current node number and total number of nodes, helping determine which tests each node should run.

Implementing Parallel Test Execution

To enable parallel execution, developers need to configure the batch fanout section in their buildspec.xml file. This configuration specifies the desired level of parallelism and other parameters. Additionally, the codebuild-tests-run utility can be used in the build step alongside the appropriate test commands and chosen splitting method.

Tests are divided based on the sharding strategy selected. CodeBuild offers two main sharding strategies:

• Equal-distribution: This strategy sorts test files alphabetically and distributes them equally across parallel environments. Any changes in test file names or quantities might result in reassignment across shards.
• Stability: This strategy uses a consistent hashing algorithm to maintain existing file-to-shard assignments when new files are added or removed.
  

  Automatic Merging of Test Reports
  

  One of the significant advantages of parallel test execution is the automatic merging of test reports. CodeBuild consolidates the reports into a single summary, simplifying result analysis. This merged report includes aggregated pass/fail statuses, test durations, and failure details, reducing the need for manual report processing. Users can view the merged results in the CodeBuild console, retrieve them via the AWS Command Line Interface (AWS CLI), or integrate them with other reporting tools.

  Step-by-Step Implementation Guide
  

  To illustrate the implementation of parallel testing, consider a basic Python project with hundreds of tests. In this demonstration, the test suite is run on ten compute environments in parallel. The following steps outline the process:

  1. Create a buildspec.yml file: The file contains essential configurations, including build-fanout parameters, to run tests in parallel.
  2. Configure the buildspec.yml file:
     “`yaml
     version: 0.2

     batch:
     fast-fail: false
     build-fanout:
     parallelism: 10
     ignore-failure: false

     phases:
     install:
     commands:

     • echo ‘Installing Python dependencies’
     • dnf install -y python3 python3-pip
     • pip3 install –upgrade pip
     • pip3 install pytest
       build:
       commands:
     • echo ‘Running Python Tests’
     • |
       codebuild-tests-run \
       –test-command ‘python -m pytest –junitxml=report/testreport.xml’ \
       –files-search "codebuild-glob-search ‘tests/test*.py’" \
       –sharding-strategy ‘equal-distribution’
       post_build:
       commands:
     • echo "Test execution completed"

       reports:
       pytest_reports:
       files:

     • "*.xml"
       base-directory: "report"
       file-format: JUNITXML
       “`

  3. Set up the CodeBuild project: Create a batch build configuration for the project and configure it to run in batch mode.
  4. Trigger the test suite execution: Initiate the build by committing new code or manually starting it through the console.
  5. Analyze the results: Once the tests are complete, access the merged test reports via the Reports tab in the CodeBuild console.
     

     Additional Features and Customization
     

     This new parallel test execution capability is compatible with all testing frameworks. The AWS CodeBuild documentation includes examples for various frameworks, including Django, Elixir, Go, Java (Maven), Javascript (Jest), Kotlin, PHPUnit, Pytest, Ruby (Cucumber), and Ruby (RSpec).

     For test frameworks that do not accept space-separated lists, the codebuild-tests-run CLI provides flexibility through the CODEBUILD_CURRENT_SHARD_FILES environment variable. This variable contains a newline-separated list of test file paths for the current build shard, enabling adaptation to different test framework requirements.

     Furthermore, developers can customize how tests are split across environments by writing their own sharding scripts and using the CODEBUILD_BATCH_BUILD_IDENTIFIER environment variable, allowing for framework-specific parallelization or optimization.

     Availability and Pricing
     

     Parallel test execution is available on all three compute modes offered by CodeBuild: on-demand, reserved capacity, and AWS Lambda compute. This feature is accessible in all AWS Regions where CodeBuild is available, with no additional cost beyond the standard CodeBuild pricing for the compute resources used.

     Conclusion
     

     AWS CodeBuild’s parallel test execution capability represents a significant advancement in optimizing the software development process. By reducing test times and improving efficiency, developers can accelerate their development cycles and enhance productivity. To explore this feature further, visit the AWS CodeBuild documentation and get started with parallelizing your tests today.

For more Information, Refer to this article.