diff --git a/test/README.md b/test/README.md index cfc5f932..4daa88e1 100644 --- a/test/README.md +++ b/test/README.md @@ -6,7 +6,7 @@ Our projects use both **`bazel tests`** and **`pytest`**. You can run bazel tests with: `bazel test //...`. For more verbosity in python tests use **`-vvv`** or **`--log-cli-level=DEBUG`** for pytest. -### To run all unit tests, use one of the following command: +### To run all python tests, use one of the following command: * **Using Pytest:** ```bash pytest unit -vvv @@ -17,87 +17,89 @@ For more verbosity in python tests use **`-vvv`** or **`--log-cli-level=DEBUG`** python3 -m unittest discover unit -vvv ``` -### Running a Subset of Tests -Specify the directory containing your desired tests. For example, to run tests in `my_test_dir`: -```bash -pytest unit/my_test_dir -vvv -# OR -python3 -m unittest discover unit/my_test_dir -vvv -``` - -## Adding New Unit Tests +## Adding New Tests, -1. **Create a Test Folder** +### Create a Test Folder Inside the `unit` directory, create a folder for your new test. This folder should contain: - All source/header files needed for the test - `BUILD` -2. **Creating the BUILD File** - - Create the `cc_binary/library` targets. - - Create the `codechecker_test` targets. - - Create `unit_test` targets to assert on the outputs of the codechecker targets. (See `unit/unit_test.bzl` for documentation) - - Make sure that all failing `codechecker_test` targets get the `"manual"` tag. For example: - ``` - # This is a test I expect to fail - codechecker_test( - name = "codechecker_fail", - tags = [ - "manual", - ], - targets = [ - "test_fail", - ], - ) - ``` - - Tip: To test these failing tests, create a unit_test target and assert the bug being found. - -3. **Create a python test if you must** - - If you are writing a python test, have an `__init__.py` file in the test directory! - - Your test script must follow the naming convention: - ```text - test_*.py - ``` - - At the top of your test file, include the following snippet to correctly handle module imports: - ```python - from common.base import TestBase - ``` - - Create your test class by extending `TestBase` and implement your test methods. -> [!WARNING] -> You should include this line in your test class, this sets the current working directory: -> ```python -> __test_path__ = os.path.dirname(os.path.abspath(__file__)) -> ``` - -**For a test template look into unit/basic** +--- +### Create a skylib test -## Testing on open source projects +With skylib we can test anything that does not use the output of +`ctx.actions.run` actions (i.e. anything known at Bazel's analysis time). +Skylib provides two different kinds of tests: **unit tests** and +**analysis tests**. -### To run all FOSS tests, use one of the following command: -* **Using Pytest:** - ```bash - pytest foss -vvv - ``` +For more in depth information check the skylib documentation for [analysis](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/analysis_test_doc.md) and [unit tests](https://github.com/bazelbuild/bazel-skylib/blob/main/docs/unittest_doc.md). -* **Using Unittest:** - ```bash - python3 -m unittest discover foss -vvv - ``` +- **Unit tests** assert on a single Starlark function — call it with + known inputs and check the return value. (See example in: `test/unit/basic/analysis_test.bzl`.) +- **Analysis tests** build a real Bazel target and then inspect the + providers it returns (e.g. `DefaultInfo`, `CcInfo`, or custom + providers) without executing any actions. (See example in: `test/unit/basic/unit_test.bzl`.) + +Both are created in a `.bzl` file, instantiated from a `BUILD` file, +and run with `bazel test`. + +--- +### Creating unit tests asserting on the output of a rule + +We can test the output of `ctx.actions.run` actions using the `unit_test` macro found in `test/unit/unit_test.bzl`. +You may use the tests under `test/unit/basic` as template. +- Create the `cc_binary/library` targets. +- Create the `codechecker_test` targets. +- Create `unit_test` targets to assert on the outputs of the codechecker targets. (See `unit/unit_test.bzl` for documentation) +- Make sure that all failing `codechecker_test` targets get the `"manual"` tag. For example: +``` +# This is a test I expect to fail +codechecker_test( + name = "codechecker_fail", + tags = [ + "manual", + ], + targets = [ + "test_fail", + ], +) +``` +- Tip: To test these failing tests, create a unit_test target and assert the bug being found. + +--- +### Create a custom python test if you must + + In case you are writing integration tests, or tests that cannot be satisfied by the previous two solutions, + create a custom `py_test` target. To make thing nicer wrap the py_test into a macro like with `unit_test.blz`. + + For reference you may use: + - `test/unit/unit_test.bzl` + - `test/foss` + - `test/caching` + +## Testing on open source projects + +You can run all FOSS test with `bazel test //test/foss:*`. ## Add a new open source project: -1. Create a folder in the foss folder with the name of the project. -2. The folder should contain: - - init.sh - -3. The init.sh script should: - - Take the folder to which the project should be cloned/downloaded as the single command line argument - - Clone the test project into said folder - - To ensure the project doesn't change over time, check out a specific tag or commit instead of a branch! - - Copy the .bazelversion file, if it exists, from the root of codechecker_bazel into the projects directory. - This file is usually set by developers using bazelisk, and is also used in CI. - - Append the MODULE.template file to the MODULE.bazel file of the project. - If the project ships no MODULE.bazel, or misses a module dependency its - BUILD files refer to, add the missing bazel_dep() calls in init.sh. - - Append the codechecker rules to the BUILD file of the project. - - There can be only two targets, codechecker_test and per_file_test +Add a new `foss_test` target to the BUILD file in `test/foss`. +For each foss test you have to: + - Give an url to an archive of the codebase (get it from the releases page). + - In case the name of the target you want to test on differs from the name given to the test, define it with `target`. + - Define which tests should be run. (possible values are: `":codechecker_per_file"`, `":codechecker_test"`, `":compile_commands"`) + +Example: +```python +foss_test( + name = "cpuinfo_bazel", + target = "cpuinfo", + tests = [ + ":codechecker_per_file", + ":codechecker_test", + ":compile_commands", + ], + url = "https://github.com/pytorch/cpuinfo/archive/66ee79c0.tar.gz", +) +``` diff --git a/test/unit/basic/BUILD b/test/unit/basic/BUILD index bafae905..10769726 100644 --- a/test/unit/basic/BUILD +++ b/test/unit/basic/BUILD @@ -24,6 +24,14 @@ load( "//test/unit:unit_test.bzl", "unit_test", ) +load( + ":analysis_test.bzl", + "analysis_test_suite", +) +load( + ":unit_test.bzl", + "unit_test_suite", +) cc_library( name = "basic_target", @@ -68,3 +76,7 @@ unit_test( regex_patterns = ["[a-z]"], require_patterns_in_each_file = False, ) + +analysis_test_suite(name = "analysis_tests") + +unit_test_suite(name = "unit_tests") diff --git a/test/unit/basic/analysis_test.bzl b/test/unit/basic/analysis_test.bzl new file mode 100644 index 00000000..27ebbd06 --- /dev/null +++ b/test/unit/basic/analysis_test.bzl @@ -0,0 +1,72 @@ +# Copyright 2026 Ericsson AB +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Minimal skylib analysis test example. + +Verifies that compile_commands_aspect populates SourceFilesInfo with +the expected source file. +""" + +load("@bazel_skylib//lib:unittest.bzl", "analysistest", "asserts") +load( + "//src:compile_commands.bzl", + "SourceFilesInfo", + "compile_commands_aspect", +) + +# --------------------------------------------------------------------------- +# Test: compilation database is non-empty +# --------------------------------------------------------------------------- + +def _compilation_db_nonempty_test_impl(ctx): + env = analysistest.begin(ctx) + + target = analysistest.target_under_test(env) + db = target[SourceFilesInfo].compilation_db.to_list() + + asserts.true( + env, + len(db) > 0, + "compilation_db should contain at least one entry", + ) + + return analysistest.end(env) + +compilation_db_nonempty_test = analysistest.make( + _compilation_db_nonempty_test_impl, + extra_target_under_test_aspects = [compile_commands_aspect], +) + +# --------------------------------------------------------------------------- +# Test suite +# --------------------------------------------------------------------------- + +def analysis_test_suite(name): + """Instantiates the basic skylib analysis tests. + + Args: + name: Name prefix for the generated test targets. + """ + + compilation_db_nonempty_test( + name = name + "_compilation_db_nonempty", + target_under_test = ":basic_target", + ) + + native.test_suite( + name = name, + tests = [ + ":" + name + "_compilation_db_nonempty", + ], + ) diff --git a/test/unit/basic/unit_test.bzl b/test/unit/basic/unit_test.bzl new file mode 100644 index 00000000..8c956543 --- /dev/null +++ b/test/unit/basic/unit_test.bzl @@ -0,0 +1,83 @@ +# Copyright 2026 Ericsson AB +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Minimal skylib unit test example. + +Tests pure Starlark functions without needing real Bazel targets. +""" + +load("@bazel_skylib//lib:unittest.bzl", "asserts", "unittest") +load("//src:per_file.bzl", "check_valid_file_type") + +# --------------------------------------------------------------------------- +# Helpers — fake struct to stand in for a File object +# --------------------------------------------------------------------------- + +def _fake_file(basename): + return struct(basename = basename) + +# --------------------------------------------------------------------------- +# Test: check_valid_file_type accepts C/C++ files +# --------------------------------------------------------------------------- + +def _valid_file_types_test_impl(ctx): + env = unittest.begin(ctx) + + asserts.true(env, check_valid_file_type(_fake_file("main.c"))) + asserts.true(env, check_valid_file_type(_fake_file("main.cc"))) + asserts.true(env, check_valid_file_type(_fake_file("main.cpp"))) + asserts.true(env, check_valid_file_type(_fake_file("main.cxx"))) + + return unittest.end(env) + +valid_file_types_test = unittest.make(_valid_file_types_test_impl) + +# --------------------------------------------------------------------------- +# Test: check_valid_file_type rejects non-C/C++ files +# --------------------------------------------------------------------------- + +def _invalid_file_types_test_impl(ctx): + env = unittest.begin(ctx) + + asserts.false(env, check_valid_file_type(_fake_file("script.py"))) + asserts.false(env, check_valid_file_type(_fake_file("data.json"))) + asserts.false(env, check_valid_file_type(_fake_file("header.h"))) + asserts.false(env, check_valid_file_type(_fake_file("header.hpp"))) + asserts.false(env, check_valid_file_type(_fake_file("BUILD"))) + asserts.false(env, check_valid_file_type(_fake_file("rules.bzl"))) + + return unittest.end(env) + +invalid_file_types_test = unittest.make(_invalid_file_types_test_impl) + +# --------------------------------------------------------------------------- +# Test suite +# --------------------------------------------------------------------------- + +def unit_test_suite(name): + """Instantiates the basic skylib unit tests. + + Args: + name: Name prefix for the generated test targets. + """ + valid_file_types_test(name = name + "_valid_file_types") + invalid_file_types_test(name = name + "_invalid_file_types") + + native.test_suite( + name = name, + tests = [ + ":" + name + "_valid_file_types", + ":" + name + "_invalid_file_types", + ], + )