Performance Testing with K6

k6 is an open-source load testing tool for testing the performance of APIs, microservices, and websites and it’s a developer-centric load testing tool built for making performance testing a productive and enjoyable experience. Using k6, you’ll be able to catch performance regression and problems earlier, allowing you to build resilient systems and robust applications.

k6 is a JavaScript-based load testing tool that has two operational modes:

• k6 Open Source — tool for users to self manage (Free)
• k6 Cloud — SaaS for engineering teams (Paid)

Why k6?

k6 supports a set of built-in and custom metrics that can be used to measure various things. Below are some of the most important features that k6 can be satisfied with.

• Scriptability​ & Automatability
• Low learning complexity​
• Easy integration with CI/CD, reporting, and APM platforms
• High accuracy and performance​
• Comprehensive documentation
• High community support

Ragnar Lönn, who contributed to the early development of k6, has published a comprehensive load testing tool evaluation in 2020 which, even though may seem a little biased towards k6, has some very detailed reviews and comparisons on famous open source tools we have today.

Pros & Cons of usingk6

Following are some advantages and disadvantages of k6.

Pros:

• Provides a seamless dev experience through its immersive scripting APIs.
• Lightweight, fast, and very easy to use.
• Real-time test results can be published to various platforms including Grafana (via InfluxDB), DataDog, and AWS CloudWatch.
• Can define thresholds within the script, which would be ideal to automate certain SLA-based pass/fail criteria in CI/CD pipelines.
• Can integrate with numerous external systems.
• So many out-of-the-box options to control the throughput.
• k6 claims that a single load generator can sustain up to 30,000–40,000 simultaneous users (VUs). This would usually generate closer to 300,000 requests per second (RPS). This is much higher than the test with the highest RPS we have right now.

Cons:

• Not generating a comprehensive HTML report like in JMeter.
• Even though the scripting syntax is similar to NodeJS, some key fundamental differences are due to the k6 architecture. Because of this, if someone jumps straight into k6 scripting, without a proper understanding of k6 architecture, chances are high for him/her to end up with an inefficient perf script even if that person is an expert of Node/JS.
• Distributed load generation is not yet possible out-of-the-box. However, we can implement that using a single Grafana + InfluxDB setup. Also, k6 claims that its architecture enables a single node to generate large volumes of load.

Installation of k6

The easiest way to install k6 on your machine is to follow this k6 guide provided by the k6 team. They have provided the instructions OS-wise and you can choose the installation guide according to your OS.

There is also an official docker image. It is not recommended for beginners to build from the source, but you always have the option to do that.

To enhance your scripting experience I recommend you to install k6 plugins/extensions available for your IDE. Below are a couple of handy extensions if you are using Visual Studio Code:

• mjacobson.snippets-for-k6
• k6.k6

Sample Folder Structure

You can create your own folder structure in k6 and there is no rule for the folder structure. With k6, one main advantage we can gain is the ability to locate the performance test scripts with the application codebase itself. I have used PetStore which is an open-source swagger to conduct a sample test scenario using k6. Below shown is a sample folder structure that I have used in it:

Sample Folder Structure

data is where we can keep all our test data such as .csv files needed for the load test.

lib is where we can keep utility files needed for k6 scripting.

tests is where we can keep individual k6 test files. Each test file should act like a “Thread Group“ in JMeter’s context where typically, it should contain one or more related endpoints that are controlled by the same test conditions and configurations.

config.js is the file responsible to hold test configurations. The configurations like Ramp-up, Hold duration, Teardown, and Service host can be stored here.

main.js is like a test suite that combines all tests from the tests folder for k6 to run all of them together in a single load test. In a way, this acts similar to the Test Plan we had in JMeter. Typically, this main.js should contain “All Tests” for the corresponding component. You may create more such “Test Suite” files for different test combinations if required.

k6 Test Life-cycle

k6 has its own test life cycle and the test should be written in a way that complies with the given modal by the k6. Following is the test life cycle modal given by the k6 team. It’s better to get a clear idea of this before going for the scripting at once.

// 1. init code
export let options = {
  // 2. options object
}
export function setup() {
  // 3. setup code
}
export default function (data) {
  // 4. VU code
}
export function teardown(data) {
  // 5. teardown code
}

• init code — contain imports and variable declarations that typically happen before everything else. Here you can import k6 Script API and other JavaScript modules (libraries) that you desire to make use of.
• Objects object — is where the test configurations are defined for k6 such as several virtual users (VUs), test duration, throughput, thresholds, etc.
• setup code — is executed once before the test enters the VU code. The state can be passed through to the VU code via the data object.
• VU code — is the function that is getting iterated during the load test. This is where we place our API requests and their assertions, etc. There can be multiple VU code functions in a single test file. k6 supports a feature called virtual users. This means that you can use separate “smart” virtual users to test your system. The code in this section which is inside the exported “default” function, is run over and over inside each VU and the aggregated results of all these VUs are processed and reported by k6.
• teardown code — is executed once after the VU code execution is completed. Any housekeeping tasks can be placed here.

For more information regarding the test life cycle of a k6 script read through the k6 documentation to understand each segment’s role in a k6 script.

NOTE: Even though k6 is a JavaScript-based tool, it is not NodeJS, nor is it a browser. Packages that rely on APIs provided by NodeJS, for instance, the os and fs modules, will not work in k6. The same goes for browser-specific APIs like the window object. Please go through the k6 Modules which give more understanding on this.

You can use console.log() to print various attributes available through the response object for debugging purposes. Below is a common and useful console log:

console.log(response.request.method + ‘ ‘+ response.request.url + ‘ -> ‘ + response.status + ‘ ‘ + response.body.toString());

A Sample VU Code

Sample test

• By using the papaparse library we can import our test data files to the test.
• Inside the getPet method, we are creating the parameters that we need to send the request.
• In the request(res), the URL should be sent as the first parameter and the parameters(params) should be the second one.
• In the URL I have used a random value as the pet_id which is taken from the pets.csv.
• Using the check I have asserted whether our response is equal to 200.

Do k6 have the Test Suites?

k6 is built to run one test file at a time and doesn’t support the concept of “test suites“ out-of-the-box. But if you are writing only one or few related endpoints (for the sake of simplicity and maintainability) in a single test file, then you might want to run all of them together to simulate the load test. To overcome this I have come up with my own strategy to implement the concept of “test-suites“ for our tests as you can see below.

• Here, we are importing our individual tests into a single k6 (JavaScript) file as in the below sample code.
• Then we can create the options object on the fly by consolidating scenarios, thresholds, etc. defined in individual tests.
• Next, we can build the VU code segment using the same coming from each imported test file.

This way, we will end up with another executable k6 script that resembles a “test suite“. This is similar to a Test Plan in Jmeter.

But for the above “test suite“ concept to work, you’ll need to ensure the following in your individual test files:

• All test files should have a similar options format as expected by the “test suite“
• VU function in each test file should have its own unique name

A Sample Test Suite

main.js

Test Configurations

In k6, test configurations are set up in the form of a JavaScript object called options in each test file. k6 has a detailed document on this.

Here the executor is used to control our throughput as we want. In my example, I have used the Ramping Arrival Rate as the executor. For more details on these executors, you can have a look at this k6 document.

Below is a sample options object that I used in my tests.

Sample Option object

• Our start rate is zero. That means when the test is starting we don’t have any requests processing.
• timeUnit is used to generate the given throughput with the given time period. For example, in stages, I have given the target as 20 and the timeUnit as 10s. That means 20 transactions will be happening at a time period of 10seconds.
• preAllocatedVUs is something like threads. I have given it as 10 and that means 10 threads(virtual users) will be allocated when the test is starting.
• maxVUs is the maximum number of threads(virtual users) that the test can have. The test will achieve the given throughput with the preAllocatedVUs. If that amount is not enough to achieve the throughput, the test will take more VUs(virtual users) up to 100.
• stages is used to control our throughput. From the start to 30seconds which is the RampUp period, we want only a throughput of 20. From 31s to 300s which is the Hold Duration, we want a throughput of 20. When the time reached to 310seconds after starting the test which is the Tear Down period, the throughput will be gradually decreased to 0.
• thresholds can be used to see the status of our test, whether it has passed or failed. After a test in Jmeter, we manually check our test results and analyze the results like 90th response time, average, etc. But in k6 we don’t need to do that manually. We can configure that using the thresholds. In the above example, I have checked the 90th value of the latency is below 750. If not the test will be failed. Also, I have checked the overall failure rate is below 0.001.

Test Execution

Ok now after scripting the test cases all of you might be looking at how to execute the test cases. If you have done everything correctly(installation, scripting) you just need do is issuing the below command, from a CLI, from the same directory where your script is located:

$ k6 run script.js

Also, you can run it with 10 virtual users (VU) and over a period of 30 seconds, as follows:

$ k6 run -u 10 -d 30s script.js

It basically runs k6 on your machine using 10 virtual users over the course of 30 seconds, and it checks to see if the test URL returns 200 (OK) in all tests. For more details, you can refer to the Executing k6 scripts locally.

The terminal output of a sample test script for k6

The terminal output of a sample test script for k6

Automating Pass/Fail Criteria

By using the Thresholds feature in k6, we can automate the pass/fail criteria of a test in the k6 test script itself. In JMeter (and many other tools) we don’t have this feature. For more details, you can look at Thresholds documentation given by the k6 team which clearly describes this.

Integrations

Another major advantage that has in k6 is the easy integration with a large array of external platforms. This will ease the things in execution as well as results analysis. Let’s look at few tools that the k6 can be integrated.

Jenkins

There are two common ways to run k6 tests as part of a CI process:

• k6 run to run a test locally on the CI server(You can use the k6 docker image to run the load tests in Jenkins.).
• k6 cloud to run a test on the k6 Cloud from one or multiple geographic locations.

For more details on this, you can refer to the documentation on load testing with Jenkins given by the k6 team.

Grafana

All of you know Grafana is open-source analytics & monitoring solution, we can use Grafana to monitor our test results. It is very easy to integrate k6 with Grafana + InfluxDB setup.

Sample Grafana Dashboard

DataDog

DataDog is another monitoring tool that we can use to monitor our tests. Integrating k6 with DataDog, which is given by the k6 team is also easy and straightforward.

Sample DataDog Dashboard

Distributed Load Generation

k6 claims its architecture is so improved that a single load generator can handle an enormous load compared to other tools.

“k6 is different from many other load testing tools in the way it handles hardware resources. A single k6 process will efficiently use all CPU cores on a load generator machine. A single instance of k6 is often enough to generate a load of 30.000–40.000 simultaneous users (VUs). This amount of VUs can generate upwards of 300,000 requests per second (RPS).

Unless you need more than 100,000–300,000 requests per second (6–12M requests per minute), a single instance of k6 will likely be sufficient for your needs.“

Hope you learn something useful. See you in the next one ✌️

References

• https://k6.io/
• https://k6.io/blog/comparing-best-open-source-load-testing-tools/#history-and-status