In today’s dynamic world of software development, ensuring that plugins integrate seamlessly and function reliably is a significant challenge. Mattermost, a platform known for its open-source messaging and collaboration solutions, faced such a challenge. They found that their existing approach, which heavily relied on mocks for plugin testing, was proving to be inadequate. The limitations were clear: tests were fragile and often failed when the codebase evolved, and critical integration issues went unnoticed. This prompted Mattermost to adopt a new strategy using Testcontainers, an open-source tool that provides isolated Docker environments, streamlining complex integration testing processes.
The Old Testing Method
Previously, Mattermost utilized mocks extensively to test their plugins. Although this method had some advantages, it came with considerable drawbacks. The tests were brittle, meaning they frequently broke when the code underwent changes. Consequently, developers were constantly tasked with updating these mocks to align with the latest code changes, making the tests cumbersome to develop and maintain. Moreover, the reliance on mocks meant that the integration aspect of testing was often neglected. This oversight could result in unexpected issues when the software was deployed in a production environment.
Additionally, the previous method lacked automation, making the testing process labor-intensive and susceptible to human errors. These challenges underscored the need for a more robust testing strategy, which led to the adoption of Testcontainers for complex integration testing.
Embracing Testcontainers for Integration Testing
Testcontainers for Go
Mattermost has implemented Testcontainers for Go to create isolated testing environments for their plugins. This setup includes the Mattermost server, a PostgreSQL server, and occasionally an API mock server. The plugin is installed on the Mattermost server, and testing is conducted using regular API calls or end-to-end testing frameworks like Playwright.
Mattermost developed a tailored Testcontainers module for the Mattermost server. This module incorporates PostgreSQL as a dependency, ensuring the testing environment closely mirrors the production environment. It allows developers to easily install and configure any plugin on the Mattermost server.
To enhance the system’s isolation, the Mattermost module includes separate containers for the server and the PostgreSQL database, connected via an internal Docker network. Furthermore, the Mattermost module offers utility functions that allow direct database access, Mattermost API interactions through the Go client, and administrative actions such as creating users, channels, teams, and making configuration changes. These utilities are invaluable for conducting complex testing operations, such as API calls, user/team/channel creation, configuration adjustments, or even executing SQL queries.
This comprehensive approach to testing ensures thorough validation of all aspects of the Mattermost server and its plugins, thereby enhancing their reliability and functionality. Below is a code example of how this setup can be implemented:
go<br /> pluginConfig := map[string]any{}<br /> options := []mmcontainer.MattermostCustomizeRequestOption{<br /> mmcontainer.WithPlugin("sample.tar.gz", "sample", pluginConfig),<br /> }<br /> mattermost, err := mmcontainer.RunContainer(context.Background(), options...)<br /> defer mattermost.Terminate(context.Background())<br />
Once the Mattermost instance is initialized, a test can be conducted as follows:
go<br /> func TestSample(t *testing.T) {<br /> client, err := mattermost.GetClient()<br /> require.NoError(t, err)<br /> reqURL := client.URL + "/plugins/sample/sample-endpoint"<br /> resp, err := client.DoAPIRequest(context.Background(), http.MethodGet, reqURL, "", "")<br /> require.NoError(t, err, "cannot fetch url %s", reqURL)<br /> defer resp.Body.Close()<br /> bodyBytes, err := io.ReadAll(resp.Body)<br /> require.NoError(t, err)<br /> require.Equal(t, 200, resp.StatusCode)<br /> assert.Contains(t, string(bodyBytes), "sample-response")<br /> }<br />
Developers have the flexibility to decide when to tear down the Mattermost instance and recreate it, whether once per test or once per set of tests, based on their specific needs and the tests’ nature.
Testcontainers for Node.js
In addition to Testcontainers for Go, Mattermost leverages Testcontainers for Node.js to set up testing environments. Testcontainers for Node.js provides similar functionality to its Go counterpart, enabling the setup of isolated environments for running JavaScript tests with tools like Playwright. This setup allows for integration testing that interacts directly with the plugin user interface. The code for Testcontainers for Node.js is available on GitHub.
This approach offers the same benefits as Testcontainers for Go and allows the use of interface-based testing tools like Playwright. Here’s an example of how it can be implemented:
javascript<br /> test.beforeAll(async () => { mattermost = await RunContainer() })<br /> test.afterAll(async () => { await mattermost.stop(); })<br /> <br /> test.describe('sample slash command', () => {<br /> test('try to run a sample slash command', async ({ page }) => {<br /> const url = mattermost.url()<br /> await login(page, url, "regularuser", "regularuser")<br /> await expect(page.getByLabel('town square public channel')).toBeVisible();<br /> await page.getByTestId('post_textbox').fill("/sample run")<br /> await page.getByTestId('SendMessageButton').click();<br /> await expect(page.getByText('Sample command result', { exact: true })).toBeVisible();<br /> await logout(page)<br /> });<br /> });<br />
Using these two approaches, Mattermost can create integration tests covering both API and interface interactions without relying on mocks or synthetic environments. This ensures a high degree of isolation and avoids issues caused by contaminated environments during end-to-end testing.
Practical Applications of the New Approach
Currently, Mattermost employs this testing strategy for two of their plugins:
1. Mattermost AI Copilot
The Mattermost AI Copilot plugin assists users in their daily tasks by leveraging AI large language models (LLMs) to provide features like thread and meeting summarization and context-based interrogation. This plugin has a rich interface, making the Testcontainers for Node and Playwright approach ideal for testing the system through its interface. Additionally, the plugin requires API calls to AI LLMs. To avoid resource-heavy tasks, an API mock simulates these API interactions, ensuring comprehensive server-side and interface testing.
2. Mattermost MS Teams Plugin
Designed to connect MS Teams and Mattermost seamlessly, this plugin synchronizes messages between both platforms. For this plugin, the primary focus is on API calls, so Testcontainers for Go is used to directly interact with the API using a Go client. This plugin also depends on the Microsoft Graph API, and an API mock is used to test the plugin without relying on the third-party service. Some integration tests are conducted with the actual Teams API using the same Testcontainers infrastructure to confirm accurate handling of Microsoft Graph calls.
Benefits of Testcontainers Libraries
Using Testcontainers for integration testing offers several advantages:
- Isolation: Each test runs in its own Docker container, ensuring complete isolation from other tests. This prevents interference and ensures each test starts with a clean environment.
- Repeatability: The automated setup of the testing environment makes tests highly repeatable. Developers can run tests multiple times with consistent results, enhancing test reliability.
- Ease of Use: Testcontainers simplifies the process by managing Docker container setup and teardown, allowing developers to focus on writing tests rather than managing the testing environment.
Simplifying Testing with Testcontainers
Mattermost’s use of Testcontainers libraries for complex integration testing in their plugins demonstrates the tool’s power and versatility. By creating well-isolated and repeatable testing environments, Mattermost ensures their plugins are thoroughly tested and highly reliable.
Additional Resources
To learn more about Testcontainers and Mattermost’s testing strategies, you can explore more through various online resources and repositories.
By adopting Testcontainers, Mattermost has not only improved their testing accuracy but also streamlined their development process, ensuring a robust and reliable product for their users.
For more Information, Refer to this article.
