If an MR title starts with WIP: or Draft:, push commit will NOT trigger a merge-request pipeline
If a commit message starts with test ci:, pushing a commit will trigger a merge-request pipeline even when the MR title starts with WIP: or Draft:.
If a commit message starts with WIP: or Draft:, push commit will NOT trigger a pipeline

What if Expected Jobs ARE NOT Created?

check the file patterns

If you found a job that is not running as expected with some file changes, a git commit to improve the pattern will be appreciated.
please add MR labels to run additional tests

MR labels for additional jobs

Supported MR Labels

build
build_docs
component_ut[_esp32/esp32s2/...]
custom_test[_esp32/esp32s2/...]
docker
docs
example_test[_esp32/esp32s2/...]
fuzzer_test
host_test
integration_test
iperf_stress_test
macos
macos_test
nvs_coverage
submodule
unit_test[_esp32/esp32s2/...]
weekend_test
windows

There are two general labels (not recommended since these two labels will trigger a lot of jobs)

target_test: includes all target for example_test, custom_test, component_ut, unit_test, integration_test
all_test: includes all test labels

Usages

We have two ways to run additional jobs

Add these labels in the MR labels
Add these labels in the commit message (not the first line). For example:
```
ci: detect file changes to assign jobs

test labels: example_test_esp32, custom_test_esp32
```
The additional test labels line should start with test label(s): and the labels should be separated by space or comma.

How to trigger a `detached` pipeline without pushing new commits?

Go to MR web page -> Pipelines tab -> click Run pipeline button

How to Develop With `rules.yml`?

General Concepts

pattern: Defined in an array. A GitLab job will be created if the changed files in this MR matched one of the patterns. For example:
```
.patterns-python-files: &patterns-python-files
  - "**/*.py"
```
label: (deprecated). Defined in an if clause, similar as the previous bot command. A GitLab job will be created if the pipeline variables contains variables in BOT_LABEL_xxx format. For example:
```
.if-label-build_docs: &if-label-build_docs
  if: '$BOT_LABEL_BUILD_DOCS'
```

title: Defined in an if clause. A GitLab job will be created if this title included in the MR labels or in the commit message title. For example:

.if-title-docs: &if-title-docs
  if: '$CI_MERGE_REQUEST_LABELS =~ /^(?:\w+,)*docs(?:,\w+)*$/i || $CI_COMMIT_TITLE =~ /\((?:\w+\s+)*docs(?:\s+\w+)*\)$/i'

rule: A combination of various patterns, labels, and titles. It will be used by GitLab YAML extends keyword to tell GitLab in what conditions will this job be created. For example:

.rules:build:docs:
  rules:
    - <<: *if-protected
    - <<: *if-label-build
    - <<: *if-title-build
    - <<: *if-label-build_docs
    - <<: *if-title-build_docs
    - <<: *if-label-docs
    - <<: *if-title-docs
    - <<: *if-dev-push
      changes: *patterns-docs

An example for GitLab job on how to use extends:

check_docs_lang_sync:
  extends:
    - .pre_check_job_template
    - .rules:build:docs
  script:
    - cd docs
    - ./check_lang_folder_sync.sh

How to Add a New `Job`?

check if there's a suitable .rules:<rules-you-need> template

if there is, put this in the job extends. All done, now you can close this window. (extends could be array or string)
if there isn't
1. check How to Add a New Rules Template?, create a suitable one
2. follow step 1

How to Add a New `Rules` Template?

check if this rule is related to labels, patterns

if it is, please refer to dependencies/README.md and add new rules by auto-generating
if it isn't, please continue reading

check if there's a suitable .if-<if-anchor-you-need> anchor

if there is, create a rule following rules Template Naming Rules.For detail information, please refer to GitLab Documentation rules-if. Here's an example.
```
.rules:dev:
  rules:
    - <<: *if-trigger
    - <<: *if-dev-push
```
if there isn't
1. check How to Add a New if Anchor?, create a suitable one
2. follow step 1

How to Add a New `if` Anchor?

Create an if anchor following if Anchors Naming Rules. For detailed information about how to write the condition clause, please refer to GitLab Documentation `only/except (advanced). Here's an example.

.if-schedule: &if-schedule:
  if: '$CI_PIPELINE_SOURCE == "schedule"'

Naming Rules

Common Naming Rules

if a phrase has multi words, use _ to concatenate them.

e.g. regular_test

if a name has multi phrases, use - to concatenate them.

e.g. regular_test-example_test

`if` Anchors Naming Rules

if it's a label: .if-label-<label_name>
if it's a ref: .if-ref-<ref_name>
if it's a branch: .if-branch-<branch_name>
if it's a tag: .if-tag-<tag_name>
if it's multi-type combination: .if-ref-<release_name>-branch-<branch_name>

Common Phrases/Abbreviations
- no_label
  
  $BOT_TRIGGER_WITH_LABEL == null
- protected
  
  ($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_BRANCH =~ /^release\/v/ || $CI_COMMIT_TAG =~ /^v\d+\.\d+(\.\d+)?($|-)/)
- target_test
  
  a combination of example_test, custom_test, unit_test, component_ut, integration_test and all targets

`rules` Template Naming Rules

if it's tag related: .rules:tag:<tag_1>-<tag_2>
if it's label related: .rules:labels:<label_1>-<label_2>
if it's test related: .rules:test:<test_type>
if it's build related: .rules:build:<build_type>
if it's pattern related: .rules:patterns:<patterns>

Reusable Shell Script `tools/ci/utils.sh`

It is used to put all the reusable shell scripts as small functions. If you want to set before_script: [] for you job, now you can set extends: .before_script_slim instead. it will only run source tools/ci/utils.sh

If you're developing CI shell scripts, you can use these functions without source them. They're already included in all before_script

To run these commands in shell script locally, place source tools/ci/utils.sh at the very beginning.

Functions

add_gitlab_ssh_keys
add_github_ssh_keys
add_doc_server_ssh_keys
fetch_submodules
get_all_submodules

error: log in red color
warning: log in orange color
info: log in green color
run_cmd: run the command with duration seconds info
retry_failed: run the command with duration seconds info, retry when failed

README.md

IDF CI

General Workflow

What if Expected Jobs ARE NOT Created?

MR labels for additional jobs

Supported MR Labels

Usages

How to trigger a detached pipeline without pushing new commits?

How to Develop With rules.yml?

General Concepts

How to Add a New Job?

How to Add a New Rules Template?

How to Add a New if Anchor?

Naming Rules

Common Naming Rules

if Anchors Naming Rules

rules Template Naming Rules

Reusable Shell Script tools/ci/utils.sh