mirror of https://github.com/Qortal/Brooklyn
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
774 lines
26 KiB
774 lines
26 KiB
.. SPDX-License-Identifier: GPL-2.0 |
|
|
|
=========== |
|
Using KUnit |
|
=========== |
|
|
|
The purpose of this document is to describe what KUnit is, how it works, how it |
|
is intended to be used, and all the concepts and terminology that are needed to |
|
understand it. This guide assumes a working knowledge of the Linux kernel and |
|
some basic knowledge of testing. |
|
|
|
For a high level introduction to KUnit, including setting up KUnit for your |
|
project, see Documentation/dev-tools/kunit/start.rst. |
|
|
|
Organization of this document |
|
============================= |
|
|
|
This document is organized into two main sections: Testing and Common Patterns. |
|
The first covers what unit tests are and how to use KUnit to write them. The |
|
second covers common testing patterns, e.g. how to isolate code and make it |
|
possible to unit test code that was otherwise un-unit-testable. |
|
|
|
Testing |
|
======= |
|
|
|
What is KUnit? |
|
-------------- |
|
|
|
"K" is short for "kernel" so "KUnit" is the "(Linux) Kernel Unit Testing |
|
Framework." KUnit is intended first and foremost for writing unit tests; it is |
|
general enough that it can be used to write integration tests; however, this is |
|
a secondary goal. KUnit has no ambition of being the only testing framework for |
|
the kernel; for example, it does not intend to be an end-to-end testing |
|
framework. |
|
|
|
What is Unit Testing? |
|
--------------------- |
|
|
|
A `unit test <https://martinfowler.com/bliki/UnitTest.html>`_ is a test that |
|
tests code at the smallest possible scope, a *unit* of code. In the C |
|
programming language that's a function. |
|
|
|
Unit tests should be written for all the publicly exposed functions in a |
|
compilation unit; so that is all the functions that are exported in either a |
|
*class* (defined below) or all functions which are **not** static. |
|
|
|
Writing Tests |
|
------------- |
|
|
|
Test Cases |
|
~~~~~~~~~~ |
|
|
|
The fundamental unit in KUnit is the test case. A test case is a function with |
|
the signature ``void (*)(struct kunit *test)``. It calls a function to be tested |
|
and then sets *expectations* for what should happen. For example: |
|
|
|
.. code-block:: c |
|
|
|
void example_test_success(struct kunit *test) |
|
{ |
|
} |
|
|
|
void example_test_failure(struct kunit *test) |
|
{ |
|
KUNIT_FAIL(test, "This test never passes."); |
|
} |
|
|
|
In the above example ``example_test_success`` always passes because it does |
|
nothing; no expectations are set, so all expectations pass. On the other hand |
|
``example_test_failure`` always fails because it calls ``KUNIT_FAIL``, which is |
|
a special expectation that logs a message and causes the test case to fail. |
|
|
|
Expectations |
|
~~~~~~~~~~~~ |
|
An *expectation* is a way to specify that you expect a piece of code to do |
|
something in a test. An expectation is called like a function. A test is made |
|
by setting expectations about the behavior of a piece of code under test; when |
|
one or more of the expectations fail, the test case fails and information about |
|
the failure is logged. For example: |
|
|
|
.. code-block:: c |
|
|
|
void add_test_basic(struct kunit *test) |
|
{ |
|
KUNIT_EXPECT_EQ(test, 1, add(1, 0)); |
|
KUNIT_EXPECT_EQ(test, 2, add(1, 1)); |
|
} |
|
|
|
In the above example ``add_test_basic`` makes a number of assertions about the |
|
behavior of a function called ``add``; the first parameter is always of type |
|
``struct kunit *``, which contains information about the current test context; |
|
the second parameter, in this case, is what the value is expected to be; the |
|
last value is what the value actually is. If ``add`` passes all of these |
|
expectations, the test case, ``add_test_basic`` will pass; if any one of these |
|
expectations fails, the test case will fail. |
|
|
|
It is important to understand that a test case *fails* when any expectation is |
|
violated; however, the test will continue running, potentially trying other |
|
expectations until the test case ends or is otherwise terminated. This is as |
|
opposed to *assertions* which are discussed later. |
|
|
|
To learn about more expectations supported by KUnit, see |
|
Documentation/dev-tools/kunit/api/test.rst. |
|
|
|
.. note:: |
|
A single test case should be pretty short, pretty easy to understand, |
|
focused on a single behavior. |
|
|
|
For example, if we wanted to properly test the add function above, we would |
|
create additional tests cases which would each test a different property that an |
|
add function should have like this: |
|
|
|
.. code-block:: c |
|
|
|
void add_test_basic(struct kunit *test) |
|
{ |
|
KUNIT_EXPECT_EQ(test, 1, add(1, 0)); |
|
KUNIT_EXPECT_EQ(test, 2, add(1, 1)); |
|
} |
|
|
|
void add_test_negative(struct kunit *test) |
|
{ |
|
KUNIT_EXPECT_EQ(test, 0, add(-1, 1)); |
|
} |
|
|
|
void add_test_max(struct kunit *test) |
|
{ |
|
KUNIT_EXPECT_EQ(test, INT_MAX, add(0, INT_MAX)); |
|
KUNIT_EXPECT_EQ(test, -1, add(INT_MAX, INT_MIN)); |
|
} |
|
|
|
void add_test_overflow(struct kunit *test) |
|
{ |
|
KUNIT_EXPECT_EQ(test, INT_MIN, add(INT_MAX, 1)); |
|
} |
|
|
|
Notice how it is immediately obvious what all the properties that we are testing |
|
for are. |
|
|
|
Assertions |
|
~~~~~~~~~~ |
|
|
|
KUnit also has the concept of an *assertion*. An assertion is just like an |
|
expectation except the assertion immediately terminates the test case if it is |
|
not satisfied. |
|
|
|
For example: |
|
|
|
.. code-block:: c |
|
|
|
static void mock_test_do_expect_default_return(struct kunit *test) |
|
{ |
|
struct mock_test_context *ctx = test->priv; |
|
struct mock *mock = ctx->mock; |
|
int param0 = 5, param1 = -5; |
|
const char *two_param_types[] = {"int", "int"}; |
|
const void *two_params[] = {¶m0, ¶m1}; |
|
const void *ret; |
|
|
|
ret = mock->do_expect(mock, |
|
"test_printk", test_printk, |
|
two_param_types, two_params, |
|
ARRAY_SIZE(two_params)); |
|
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ret); |
|
KUNIT_EXPECT_EQ(test, -4, *((int *) ret)); |
|
} |
|
|
|
In this example, the method under test should return a pointer to a value, so |
|
if the pointer returned by the method is null or an errno, we don't want to |
|
bother continuing the test since the following expectation could crash the test |
|
case. `ASSERT_NOT_ERR_OR_NULL(...)` allows us to bail out of the test case if |
|
the appropriate conditions have not been satisfied to complete the test. |
|
|
|
Test Suites |
|
~~~~~~~~~~~ |
|
|
|
Now obviously one unit test isn't very helpful; the power comes from having |
|
many test cases covering all of a unit's behaviors. Consequently it is common |
|
to have many *similar* tests; in order to reduce duplication in these closely |
|
related tests most unit testing frameworks - including KUnit - provide the |
|
concept of a *test suite*. A *test suite* is just a collection of test cases |
|
for a unit of code with a set up function that gets invoked before every test |
|
case and then a tear down function that gets invoked after every test case |
|
completes. |
|
|
|
Example: |
|
|
|
.. code-block:: c |
|
|
|
static struct kunit_case example_test_cases[] = { |
|
KUNIT_CASE(example_test_foo), |
|
KUNIT_CASE(example_test_bar), |
|
KUNIT_CASE(example_test_baz), |
|
{} |
|
}; |
|
|
|
static struct kunit_suite example_test_suite = { |
|
.name = "example", |
|
.init = example_test_init, |
|
.exit = example_test_exit, |
|
.test_cases = example_test_cases, |
|
}; |
|
kunit_test_suite(example_test_suite); |
|
|
|
In the above example the test suite, ``example_test_suite``, would run the test |
|
cases ``example_test_foo``, ``example_test_bar``, and ``example_test_baz``; |
|
each would have ``example_test_init`` called immediately before it and would |
|
have ``example_test_exit`` called immediately after it. |
|
``kunit_test_suite(example_test_suite)`` registers the test suite with the |
|
KUnit test framework. |
|
|
|
.. note:: |
|
A test case will only be run if it is associated with a test suite. |
|
|
|
``kunit_test_suite(...)`` is a macro which tells the linker to put the specified |
|
test suite in a special linker section so that it can be run by KUnit either |
|
after late_init, or when the test module is loaded (depending on whether the |
|
test was built in or not). |
|
|
|
For more information on these types of things see the |
|
Documentation/dev-tools/kunit/api/test.rst. |
|
|
|
Common Patterns |
|
=============== |
|
|
|
Isolating Behavior |
|
------------------ |
|
|
|
The most important aspect of unit testing that other forms of testing do not |
|
provide is the ability to limit the amount of code under test to a single unit. |
|
In practice, this is only possible by being able to control what code gets run |
|
when the unit under test calls a function and this is usually accomplished |
|
through some sort of indirection where a function is exposed as part of an API |
|
such that the definition of that function can be changed without affecting the |
|
rest of the code base. In the kernel this primarily comes from two constructs, |
|
classes, structs that contain function pointers that are provided by the |
|
implementer, and architecture-specific functions which have definitions selected |
|
at compile time. |
|
|
|
Classes |
|
~~~~~~~ |
|
|
|
Classes are not a construct that is built into the C programming language; |
|
however, it is an easily derived concept. Accordingly, pretty much every project |
|
that does not use a standardized object oriented library (like GNOME's GObject) |
|
has their own slightly different way of doing object oriented programming; the |
|
Linux kernel is no exception. |
|
|
|
The central concept in kernel object oriented programming is the class. In the |
|
kernel, a *class* is a struct that contains function pointers. This creates a |
|
contract between *implementers* and *users* since it forces them to use the |
|
same function signature without having to call the function directly. In order |
|
for it to truly be a class, the function pointers must specify that a pointer |
|
to the class, known as a *class handle*, be one of the parameters; this makes |
|
it possible for the member functions (also known as *methods*) to have access |
|
to member variables (more commonly known as *fields*) allowing the same |
|
implementation to have multiple *instances*. |
|
|
|
Typically a class can be *overridden* by *child classes* by embedding the |
|
*parent class* in the child class. Then when a method provided by the child |
|
class is called, the child implementation knows that the pointer passed to it is |
|
of a parent contained within the child; because of this, the child can compute |
|
the pointer to itself because the pointer to the parent is always a fixed offset |
|
from the pointer to the child; this offset is the offset of the parent contained |
|
in the child struct. For example: |
|
|
|
.. code-block:: c |
|
|
|
struct shape { |
|
int (*area)(struct shape *this); |
|
}; |
|
|
|
struct rectangle { |
|
struct shape parent; |
|
int length; |
|
int width; |
|
}; |
|
|
|
int rectangle_area(struct shape *this) |
|
{ |
|
struct rectangle *self = container_of(this, struct shape, parent); |
|
|
|
return self->length * self->width; |
|
}; |
|
|
|
void rectangle_new(struct rectangle *self, int length, int width) |
|
{ |
|
self->parent.area = rectangle_area; |
|
self->length = length; |
|
self->width = width; |
|
} |
|
|
|
In this example (as in most kernel code) the operation of computing the pointer |
|
to the child from the pointer to the parent is done by ``container_of``. |
|
|
|
Faking Classes |
|
~~~~~~~~~~~~~~ |
|
|
|
In order to unit test a piece of code that calls a method in a class, the |
|
behavior of the method must be controllable, otherwise the test ceases to be a |
|
unit test and becomes an integration test. |
|
|
|
A fake just provides an implementation of a piece of code that is different than |
|
what runs in a production instance, but behaves identically from the standpoint |
|
of the callers; this is usually done to replace a dependency that is hard to |
|
deal with, or is slow. |
|
|
|
A good example for this might be implementing a fake EEPROM that just stores the |
|
"contents" in an internal buffer. For example, let's assume we have a class that |
|
represents an EEPROM: |
|
|
|
.. code-block:: c |
|
|
|
struct eeprom { |
|
ssize_t (*read)(struct eeprom *this, size_t offset, char *buffer, size_t count); |
|
ssize_t (*write)(struct eeprom *this, size_t offset, const char *buffer, size_t count); |
|
}; |
|
|
|
And we want to test some code that buffers writes to the EEPROM: |
|
|
|
.. code-block:: c |
|
|
|
struct eeprom_buffer { |
|
ssize_t (*write)(struct eeprom_buffer *this, const char *buffer, size_t count); |
|
int flush(struct eeprom_buffer *this); |
|
size_t flush_count; /* Flushes when buffer exceeds flush_count. */ |
|
}; |
|
|
|
struct eeprom_buffer *new_eeprom_buffer(struct eeprom *eeprom); |
|
void destroy_eeprom_buffer(struct eeprom *eeprom); |
|
|
|
We can easily test this code by *faking out* the underlying EEPROM: |
|
|
|
.. code-block:: c |
|
|
|
struct fake_eeprom { |
|
struct eeprom parent; |
|
char contents[FAKE_EEPROM_CONTENTS_SIZE]; |
|
}; |
|
|
|
ssize_t fake_eeprom_read(struct eeprom *parent, size_t offset, char *buffer, size_t count) |
|
{ |
|
struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent); |
|
|
|
count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset); |
|
memcpy(buffer, this->contents + offset, count); |
|
|
|
return count; |
|
} |
|
|
|
ssize_t fake_eeprom_write(struct eeprom *parent, size_t offset, const char *buffer, size_t count) |
|
{ |
|
struct fake_eeprom *this = container_of(parent, struct fake_eeprom, parent); |
|
|
|
count = min(count, FAKE_EEPROM_CONTENTS_SIZE - offset); |
|
memcpy(this->contents + offset, buffer, count); |
|
|
|
return count; |
|
} |
|
|
|
void fake_eeprom_init(struct fake_eeprom *this) |
|
{ |
|
this->parent.read = fake_eeprom_read; |
|
this->parent.write = fake_eeprom_write; |
|
memset(this->contents, 0, FAKE_EEPROM_CONTENTS_SIZE); |
|
} |
|
|
|
We can now use it to test ``struct eeprom_buffer``: |
|
|
|
.. code-block:: c |
|
|
|
struct eeprom_buffer_test { |
|
struct fake_eeprom *fake_eeprom; |
|
struct eeprom_buffer *eeprom_buffer; |
|
}; |
|
|
|
static void eeprom_buffer_test_does_not_write_until_flush(struct kunit *test) |
|
{ |
|
struct eeprom_buffer_test *ctx = test->priv; |
|
struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; |
|
struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; |
|
char buffer[] = {0xff}; |
|
|
|
eeprom_buffer->flush_count = SIZE_MAX; |
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); |
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0); |
|
|
|
eeprom_buffer->flush(eeprom_buffer); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); |
|
} |
|
|
|
static void eeprom_buffer_test_flushes_after_flush_count_met(struct kunit *test) |
|
{ |
|
struct eeprom_buffer_test *ctx = test->priv; |
|
struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; |
|
struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; |
|
char buffer[] = {0xff}; |
|
|
|
eeprom_buffer->flush_count = 2; |
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); |
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); |
|
} |
|
|
|
static void eeprom_buffer_test_flushes_increments_of_flush_count(struct kunit *test) |
|
{ |
|
struct eeprom_buffer_test *ctx = test->priv; |
|
struct eeprom_buffer *eeprom_buffer = ctx->eeprom_buffer; |
|
struct fake_eeprom *fake_eeprom = ctx->fake_eeprom; |
|
char buffer[] = {0xff, 0xff}; |
|
|
|
eeprom_buffer->flush_count = 2; |
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 1); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0); |
|
|
|
eeprom_buffer->write(eeprom_buffer, buffer, 2); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[0], 0xff); |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[1], 0xff); |
|
/* Should have only flushed the first two bytes. */ |
|
KUNIT_EXPECT_EQ(test, fake_eeprom->contents[2], 0); |
|
} |
|
|
|
static int eeprom_buffer_test_init(struct kunit *test) |
|
{ |
|
struct eeprom_buffer_test *ctx; |
|
|
|
ctx = kunit_kzalloc(test, sizeof(*ctx), GFP_KERNEL); |
|
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx); |
|
|
|
ctx->fake_eeprom = kunit_kzalloc(test, sizeof(*ctx->fake_eeprom), GFP_KERNEL); |
|
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->fake_eeprom); |
|
fake_eeprom_init(ctx->fake_eeprom); |
|
|
|
ctx->eeprom_buffer = new_eeprom_buffer(&ctx->fake_eeprom->parent); |
|
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ctx->eeprom_buffer); |
|
|
|
test->priv = ctx; |
|
|
|
return 0; |
|
} |
|
|
|
static void eeprom_buffer_test_exit(struct kunit *test) |
|
{ |
|
struct eeprom_buffer_test *ctx = test->priv; |
|
|
|
destroy_eeprom_buffer(ctx->eeprom_buffer); |
|
} |
|
|
|
Testing against multiple inputs |
|
------------------------------- |
|
|
|
Testing just a few inputs might not be enough to have confidence that the code |
|
works correctly, e.g. for a hash function. |
|
|
|
In such cases, it can be helpful to have a helper macro or function, e.g. this |
|
fictitious example for ``sha1sum(1)`` |
|
|
|
.. code-block:: c |
|
|
|
#define TEST_SHA1(in, want) \ |
|
sha1sum(in, out); \ |
|
KUNIT_EXPECT_STREQ_MSG(test, out, want, "sha1sum(%s)", in); |
|
|
|
char out[40]; |
|
TEST_SHA1("hello world", "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed"); |
|
TEST_SHA1("hello world!", "430ce34d020724ed75a196dfc2ad67c77772d169"); |
|
|
|
|
|
Note the use of ``KUNIT_EXPECT_STREQ_MSG`` to give more context when it fails |
|
and make it easier to track down. (Yes, in this example, ``want`` is likely |
|
going to be unique enough on its own). |
|
|
|
The ``_MSG`` variants are even more useful when the same expectation is called |
|
multiple times (in a loop or helper function) and thus the line number isn't |
|
enough to identify what failed, like below. |
|
|
|
In some cases, it can be helpful to write a *table-driven test* instead, e.g. |
|
|
|
.. code-block:: c |
|
|
|
int i; |
|
char out[40]; |
|
|
|
struct sha1_test_case { |
|
const char *str; |
|
const char *sha1; |
|
}; |
|
|
|
struct sha1_test_case cases[] = { |
|
{ |
|
.str = "hello world", |
|
.sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", |
|
}, |
|
{ |
|
.str = "hello world!", |
|
.sha1 = "430ce34d020724ed75a196dfc2ad67c77772d169", |
|
}, |
|
}; |
|
for (i = 0; i < ARRAY_SIZE(cases); ++i) { |
|
sha1sum(cases[i].str, out); |
|
KUNIT_EXPECT_STREQ_MSG(test, out, cases[i].sha1, |
|
"sha1sum(%s)", cases[i].str); |
|
} |
|
|
|
|
|
There's more boilerplate involved, but it can: |
|
|
|
* be more readable when there are multiple inputs/outputs thanks to field names, |
|
|
|
* E.g. see ``fs/ext4/inode-test.c`` for an example of both. |
|
* reduce duplication if test cases can be shared across multiple tests. |
|
|
|
* E.g. if we wanted to also test ``sha256sum``, we could add a ``sha256`` |
|
field and reuse ``cases``. |
|
|
|
* be converted to a "parameterized test", see below. |
|
|
|
Parameterized Testing |
|
~~~~~~~~~~~~~~~~~~~~~ |
|
|
|
The table-driven testing pattern is common enough that KUnit has special |
|
support for it. |
|
|
|
Reusing the same ``cases`` array from above, we can write the test as a |
|
"parameterized test" with the following. |
|
|
|
.. code-block:: c |
|
|
|
// This is copy-pasted from above. |
|
struct sha1_test_case { |
|
const char *str; |
|
const char *sha1; |
|
}; |
|
struct sha1_test_case cases[] = { |
|
{ |
|
.str = "hello world", |
|
.sha1 = "2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", |
|
}, |
|
{ |
|
.str = "hello world!", |
|
.sha1 = "430ce34d020724ed75a196dfc2ad67c77772d169", |
|
}, |
|
}; |
|
|
|
// Need a helper function to generate a name for each test case. |
|
static void case_to_desc(const struct sha1_test_case *t, char *desc) |
|
{ |
|
strcpy(desc, t->str); |
|
} |
|
// Creates `sha1_gen_params()` to iterate over `cases`. |
|
KUNIT_ARRAY_PARAM(sha1, cases, case_to_desc); |
|
|
|
// Looks no different from a normal test. |
|
static void sha1_test(struct kunit *test) |
|
{ |
|
// This function can just contain the body of the for-loop. |
|
// The former `cases[i]` is accessible under test->param_value. |
|
char out[40]; |
|
struct sha1_test_case *test_param = (struct sha1_test_case *)(test->param_value); |
|
|
|
sha1sum(test_param->str, out); |
|
KUNIT_EXPECT_STREQ_MSG(test, out, test_param->sha1, |
|
"sha1sum(%s)", test_param->str); |
|
} |
|
|
|
// Instead of KUNIT_CASE, we use KUNIT_CASE_PARAM and pass in the |
|
// function declared by KUNIT_ARRAY_PARAM. |
|
static struct kunit_case sha1_test_cases[] = { |
|
KUNIT_CASE_PARAM(sha1_test, sha1_gen_params), |
|
{} |
|
}; |
|
|
|
.. _kunit-on-non-uml: |
|
|
|
KUnit on non-UML architectures |
|
============================== |
|
|
|
By default KUnit uses UML as a way to provide dependencies for code under test. |
|
Under most circumstances KUnit's usage of UML should be treated as an |
|
implementation detail of how KUnit works under the hood. Nevertheless, there |
|
are instances where being able to run architecture-specific code or test |
|
against real hardware is desirable. For these reasons KUnit supports running on |
|
other architectures. |
|
|
|
Running existing KUnit tests on non-UML architectures |
|
----------------------------------------------------- |
|
|
|
There are some special considerations when running existing KUnit tests on |
|
non-UML architectures: |
|
|
|
* Hardware may not be deterministic, so a test that always passes or fails |
|
when run under UML may not always do so on real hardware. |
|
* Hardware and VM environments may not be hermetic. KUnit tries its best to |
|
provide a hermetic environment to run tests; however, it cannot manage state |
|
that it doesn't know about outside of the kernel. Consequently, tests that |
|
may be hermetic on UML may not be hermetic on other architectures. |
|
* Some features and tooling may not be supported outside of UML. |
|
* Hardware and VMs are slower than UML. |
|
|
|
None of these are reasons not to run your KUnit tests on real hardware; they are |
|
only things to be aware of when doing so. |
|
|
|
Currently, the KUnit Wrapper (``tools/testing/kunit/kunit.py``) (aka |
|
kunit_tool) only fully supports running tests inside of UML and QEMU; however, |
|
this is only due to our own time limitations as humans working on KUnit. It is |
|
entirely possible to support other emulators and even actual hardware, but for |
|
now QEMU and UML is what is fully supported within the KUnit Wrapper. Again, to |
|
be clear, this is just the Wrapper. The actualy KUnit tests and the KUnit |
|
library they are written in is fully architecture agnostic and can be used in |
|
virtually any setup, you just won't have the benefit of typing a single command |
|
out of the box and having everything magically work perfectly. |
|
|
|
Again, all core KUnit framework features are fully supported on all |
|
architectures, and using them is straightforward: Most popular architectures |
|
are supported directly in the KUnit Wrapper via QEMU. Currently, supported |
|
architectures on QEMU include: |
|
|
|
* i386 |
|
* x86_64 |
|
* arm |
|
* arm64 |
|
* alpha |
|
* powerpc |
|
* riscv |
|
* s390 |
|
* sparc |
|
|
|
In order to run KUnit tests on one of these architectures via QEMU with the |
|
KUnit wrapper, all you need to do is specify the flags ``--arch`` and |
|
``--cross_compile`` when invoking the KUnit Wrapper. For example, we could run |
|
the default KUnit tests on ARM in the following manner (assuming we have an ARM |
|
toolchain installed): |
|
|
|
.. code-block:: bash |
|
|
|
tools/testing/kunit/kunit.py run --timeout=60 --jobs=12 --arch=arm --cross_compile=arm-linux-gnueabihf- |
|
|
|
Alternatively, if you want to run your tests on real hardware or in some other |
|
emulation environment, all you need to do is to take your kunitconfig, your |
|
Kconfig options for the tests you would like to run, and merge them into |
|
whatever config your are using for your platform. That's it! |
|
|
|
For example, let's say you have the following kunitconfig: |
|
|
|
.. code-block:: none |
|
|
|
CONFIG_KUNIT=y |
|
CONFIG_KUNIT_EXAMPLE_TEST=y |
|
|
|
If you wanted to run this test on an x86 VM, you might add the following config |
|
options to your ``.config``: |
|
|
|
.. code-block:: none |
|
|
|
CONFIG_KUNIT=y |
|
CONFIG_KUNIT_EXAMPLE_TEST=y |
|
CONFIG_SERIAL_8250=y |
|
CONFIG_SERIAL_8250_CONSOLE=y |
|
|
|
All these new options do is enable support for a common serial console needed |
|
for logging. |
|
|
|
Next, you could build a kernel with these tests as follows: |
|
|
|
|
|
.. code-block:: bash |
|
|
|
make ARCH=x86 olddefconfig |
|
make ARCH=x86 |
|
|
|
Once you have built a kernel, you could run it on QEMU as follows: |
|
|
|
.. code-block:: bash |
|
|
|
qemu-system-x86_64 -enable-kvm \ |
|
-m 1024 \ |
|
-kernel arch/x86_64/boot/bzImage \ |
|
-append 'console=ttyS0' \ |
|
--nographic |
|
|
|
Interspersed in the kernel logs you might see the following: |
|
|
|
.. code-block:: none |
|
|
|
TAP version 14 |
|
# Subtest: example |
|
1..1 |
|
# example_simple_test: initializing |
|
ok 1 - example_simple_test |
|
ok 1 - example |
|
|
|
Congratulations, you just ran a KUnit test on the x86 architecture! |
|
|
|
In a similar manner, kunit and kunit tests can also be built as modules, |
|
so if you wanted to run tests in this way you might add the following config |
|
options to your ``.config``: |
|
|
|
.. code-block:: none |
|
|
|
CONFIG_KUNIT=m |
|
CONFIG_KUNIT_EXAMPLE_TEST=m |
|
|
|
Once the kernel is built and installed, a simple |
|
|
|
.. code-block:: bash |
|
|
|
modprobe example-test |
|
|
|
...will run the tests. |
|
|
|
.. note:: |
|
Note that you should make sure your test depends on ``KUNIT=y`` in Kconfig |
|
if the test does not support module build. Otherwise, it will trigger |
|
compile errors if ``CONFIG_KUNIT`` is ``m``. |
|
|
|
Writing new tests for other architectures |
|
----------------------------------------- |
|
|
|
The first thing you must do is ask yourself whether it is necessary to write a |
|
KUnit test for a specific architecture, and then whether it is necessary to |
|
write that test for a particular piece of hardware. In general, writing a test |
|
that depends on having access to a particular piece of hardware or software (not |
|
included in the Linux source repo) should be avoided at all costs. |
|
|
|
Even if you only ever plan on running your KUnit test on your hardware |
|
configuration, other people may want to run your tests and may not have access |
|
to your hardware. If you write your test to run on UML, then anyone can run your |
|
tests without knowing anything about your particular setup, and you can still |
|
run your tests on your hardware setup just by compiling for your architecture. |
|
|
|
.. important:: |
|
Always prefer tests that run on UML to tests that only run under a particular |
|
architecture, and always prefer tests that run under QEMU or another easy |
|
(and monetarily free) to obtain software environment to a specific piece of |
|
hardware. |
|
|
|
Nevertheless, there are still valid reasons to write an architecture or hardware |
|
specific test: for example, you might want to test some code that really belongs |
|
in ``arch/some-arch/*``. Even so, try your best to write the test so that it |
|
does not depend on physical hardware: if some of your test cases don't need the |
|
hardware, only require the hardware for tests that actually need it. |
|
|
|
Now that you have narrowed down exactly what bits are hardware specific, the |
|
actual procedure for writing and running the tests is pretty much the same as |
|
writing normal KUnit tests. One special caveat is that you have to reset |
|
hardware state in between test cases; if this is not possible, you may only be |
|
able to run one test case per invocation. |
|
|
|
.. TODO([email protected]): Add an actual example of an architecture- |
|
dependent KUnit test. |
|
|
|
KUnit debugfs representation |
|
============================ |
|
When kunit test suites are initialized, they create an associated directory |
|
in ``/sys/kernel/debug/kunit/<test-suite>``. The directory contains one file |
|
|
|
- results: "cat results" displays results of each test case and the results |
|
of the entire suite for the last test run. |
|
|
|
The debugfs representation is primarily of use when kunit test suites are |
|
run in a native environment, either as modules or builtin. Having a way |
|
to display results like this is valuable as otherwise results can be |
|
intermixed with other events in dmesg output. The maximum size of each |
|
results file is KUNIT_LOG_SIZE bytes (defined in ``include/kunit/test.h``).
|
|
|