forked from mirrors/linux
bc41d57fab
Update the documentation to specify linking to a relevant GitLab issue or email report for each new flake entry. Added specific GitLab issue urls for amdgpu, i915, msm and xe driver. Acked-by: Maxime Ripard <mripard@kernel.org> Acked-by: Rodrigo Vivi <rodrigo.vivi@intel.com> #intel and xe Acked-by: Abhinav Kumar <quic_abhinavk@quicinc.com> # msm Acked-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org> # msm Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com> Acked-by: Alex Deucher <alexander.deucher@amd.com> Signed-off-by: Helen Koike <helen.koike@collabora.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240930095255.2071586-1-vignesh.raman@collabora.com
167 lines
6.2 KiB
ReStructuredText
167 lines
6.2 KiB
ReStructuredText
.. SPDX-License-Identifier: GPL-2.0+
|
|
|
|
=========================================
|
|
Automated testing of the DRM subsystem
|
|
=========================================
|
|
|
|
Introduction
|
|
============
|
|
|
|
Making sure that changes to the core or drivers don't introduce regressions can
|
|
be very time-consuming when lots of different hardware configurations need to
|
|
be tested. Moreover, it isn't practical for each person interested in this
|
|
testing to have to acquire and maintain what can be a considerable amount of
|
|
hardware.
|
|
|
|
Also, it is desirable for developers to check for regressions in their code by
|
|
themselves, instead of relying on the maintainers to find them and then
|
|
reporting back.
|
|
|
|
There are facilities in gitlab.freedesktop.org to automatically test Mesa that
|
|
can be used as well for testing the DRM subsystem. This document explains how
|
|
people interested in testing it can use this shared infrastructure to save
|
|
quite some time and effort.
|
|
|
|
|
|
Relevant files
|
|
==============
|
|
|
|
drivers/gpu/drm/ci/gitlab-ci.yml
|
|
--------------------------------
|
|
|
|
This is the root configuration file for GitLab CI. Among other less interesting
|
|
bits, it specifies the specific version of the scripts to be used. There are
|
|
some variables that can be modified to change the behavior of the pipeline:
|
|
|
|
DRM_CI_PROJECT_PATH
|
|
Repository that contains the Mesa software infrastructure for CI
|
|
|
|
DRM_CI_COMMIT_SHA
|
|
A particular revision to use from that repository
|
|
|
|
UPSTREAM_REPO
|
|
URL to git repository containing the target branch
|
|
|
|
TARGET_BRANCH
|
|
Branch to which this branch is to be merged into
|
|
|
|
IGT_VERSION
|
|
Revision of igt-gpu-tools being used, from
|
|
https://gitlab.freedesktop.org/drm/igt-gpu-tools
|
|
|
|
drivers/gpu/drm/ci/testlist.txt
|
|
-------------------------------
|
|
|
|
IGT tests to be run on all drivers (unless mentioned in a driver's \*-skips.txt
|
|
file, see below).
|
|
|
|
drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-fails.txt
|
|
----------------------------------------------------------
|
|
|
|
Lists the known failures for a given driver on a specific hardware revision.
|
|
|
|
drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-flakes.txt
|
|
-----------------------------------------------------------
|
|
|
|
Lists the tests that for a given driver on a specific hardware revision are
|
|
known to behave unreliably. These tests won't cause a job to fail regardless of
|
|
the result. They will still be run.
|
|
|
|
Each new flake entry must be associated with a link to the email reporting the
|
|
bug to the author of the affected driver or the relevant GitLab issue. The entry
|
|
must also include the board name or Device Tree name, the first kernel version
|
|
affected, the IGT version used for tests, and an approximation of the failure rate.
|
|
|
|
They should be provided under the following format::
|
|
|
|
# Bug Report: $LORE_URL_OR_GITLAB_ISSUE
|
|
# Board Name: broken-board.dtb
|
|
# Linux Version: 6.6-rc1
|
|
# IGT Version: 1.28-gd2af13d9f
|
|
# Failure Rate: 100
|
|
flaky-test
|
|
|
|
Use the appropriate link below to create a GitLab issue:
|
|
amdgpu driver: https://gitlab.freedesktop.org/drm/amd/-/issues
|
|
i915 driver: https://gitlab.freedesktop.org/drm/i915/kernel/-/issues
|
|
msm driver: https://gitlab.freedesktop.org/drm/msm/-/issues
|
|
xe driver: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues
|
|
|
|
drivers/gpu/drm/ci/${DRIVER_NAME}-${HW_REVISION}-skips.txt
|
|
-----------------------------------------------------------
|
|
|
|
Lists the tests that won't be run for a given driver on a specific hardware
|
|
revision. These are usually tests that interfere with the running of the test
|
|
list due to hanging the machine, causing OOM, taking too long, etc.
|
|
|
|
|
|
How to enable automated testing on your tree
|
|
============================================
|
|
|
|
1. Create a Linux tree in https://gitlab.freedesktop.org/ if you don't have one
|
|
yet
|
|
|
|
2. In your kernel repo's configuration (eg.
|
|
https://gitlab.freedesktop.org/janedoe/linux/-/settings/ci_cd), change the
|
|
CI/CD configuration file from .gitlab-ci.yml to
|
|
drivers/gpu/drm/ci/gitlab-ci.yml.
|
|
|
|
3. Request to be added to the drm/ci-ok group so that your user has the
|
|
necessary privileges to run the CI on https://gitlab.freedesktop.org/drm/ci-ok
|
|
|
|
4. Next time you push to this repository, you will see a CI pipeline being
|
|
created (eg. https://gitlab.freedesktop.org/janedoe/linux/-/pipelines)
|
|
|
|
5. The various jobs will be run and when the pipeline is finished, all jobs
|
|
should be green unless a regression has been found.
|
|
|
|
|
|
How to update test expectations
|
|
===============================
|
|
|
|
If your changes to the code fix any tests, you will have to remove one or more
|
|
lines from one or more of the files in
|
|
drivers/gpu/drm/ci/${DRIVER_NAME}_*_fails.txt, for each of the test platforms
|
|
affected by the change.
|
|
|
|
|
|
How to expand coverage
|
|
======================
|
|
|
|
If your code changes make it possible to run more tests (by solving reliability
|
|
issues, for example), you can remove tests from the flakes and/or skips lists,
|
|
and then the expected results if there are any known failures.
|
|
|
|
If there is a need for updating the version of IGT being used (maybe you have
|
|
added more tests to it), update the IGT_VERSION variable at the top of the
|
|
gitlab-ci.yml file.
|
|
|
|
|
|
How to test your changes to the scripts
|
|
=======================================
|
|
|
|
For testing changes to the scripts in the drm-ci repo, change the
|
|
DRM_CI_PROJECT_PATH and DRM_CI_COMMIT_SHA variables in
|
|
drivers/gpu/drm/ci/gitlab-ci.yml to match your fork of the project (eg.
|
|
janedoe/drm-ci). This fork needs to be in https://gitlab.freedesktop.org/.
|
|
|
|
|
|
How to incorporate external fixes in your testing
|
|
=================================================
|
|
|
|
Often, regressions in other trees will prevent testing changes local to the
|
|
tree under test. These fixes will be automatically merged in during the build
|
|
jobs from a branch in the target tree that is named as
|
|
${TARGET_BRANCH}-external-fixes.
|
|
|
|
If the pipeline is not in a merge request and a branch with the same name
|
|
exists in the local tree, commits from that branch will be merged in as well.
|
|
|
|
|
|
How to deal with automated testing labs that may be down
|
|
========================================================
|
|
|
|
If a hardware farm is down and thus causing pipelines to fail that would
|
|
otherwise pass, one can disable all jobs that would be submitted to that farm
|
|
by editing the file at
|
|
https://gitlab.freedesktop.org/gfx-ci/lab-status/-/blob/main/lab-status.yml.
|