test_runner: extend tag filter with boolean expression DSL by atlowChemi · Pull Request #63054 · nodejs/node

atlowChemi · 2026-04-30T16:29:26Z

Summary

Builds on #63221, which landed the tags option, inheritance, reporter
event payloads, and a literal-tag-name filter. This PR upgrades the
filter to a boolean expression DSL:

Grammar: and/&&, or/||, not/!, parentheses for grouping,
and * wildcards inside identifiers (bare * matches any tagged
test). Standard precedence (not > and > or); binary operators are
left-associative. Word forms (and/or/not) require whitespace
separation; punctuation forms do not.
--experimental-test-tag-filter and the testTagFilters run()
option now accept an expression rather than a literal tag name. The
flag is still repeatable; multiple expressions still AND together.
Untagged tests: any include expression evaluates false against an
empty tag set; not X evaluates true. So not flaky keeps every
untagged test, while db excludes them.
Tag value validation tightens: tag names may no longer contain
whitespace, operator characters (& | ! ( ) *), or the reserved
words and/or/not in any casing. This is a breaking change
relative to test_runner: add tags option and tag-name filter #63221 - acceptable since the feature is at Stability 1.0
(Early development).
Malformed expressions fail fast at parent-process startup before any
test file is spawned.

Prior art

Tag filtering with boolean composition is well-trodden in the JS
testing ecosystem. The expression syntax here is closest to Vitest's:

Vitest test tags - https://vitest.dev/guide/test-tags
mocha-tags - https://www.npmjs.com/package/mocha-tags
jest-runner-groups - https://www.npmjs.com/package/jest-runner-groups
jest-tags - https://github.com/gitbrik/jest-tags

Vitest additionally lets a tag carry per-test config overrides such as
timeout or retry (with priority resolution between overlapping
tags). That's intentionally out of scope for this PR - landing it
would commit to a separate tags: [{ name, timeout, retry, priority }]
config surface that's orthogonal to filtering. It can be layered on
later without breaking the union-inheritance semantics established in
#63221.

nodejs-github-bot · 2026-04-30T16:29:32Z

Review requested:

@nodejs/config
@nodejs/test_runner

codecov · 2026-04-30T18:06:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.34%. Comparing base (8c8190f) to head (7c3d927).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #63054      +/-   ##
==========================================
+ Coverage   90.33%   90.34%   +0.01%     
==========================================
  Files         759      759              
  Lines      248456   248856     +400     
  Branches    46871    46970      +99     
==========================================
+ Hits       224439   224831     +392     
+ Misses      15454    15449       -5     
- Partials     8563     8576      +13

Files with missing lines	Coverage Δ
lib/internal/test_runner/runner.js	`94.77% <100.00%> (+0.33%)`	⬆️
lib/internal/test_runner/tag_filter.js	`100.00% <100.00%> (ø)`
lib/internal/test_runner/utils.js	`66.30% <100.00%> (-0.05%)`	⬇️

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JakobJingleheimer · 2026-04-30T20:18:32Z

Sweet! I'll try to carve out time on Saturday to reciew

JakobJingleheimer

Sorry for the delay. GitHub is having a bit of a stroke. Every time I try to add a comment to the review "Something went wrong" and I have to copy it, reload, paste, and try a couple more times.

I'll try to finish tomorrow (I got through 10 of 18 files, which doesn't include tag_filter.js).

JakobJingleheimer · 2026-05-06T18:35:02Z

-  fail(nesting, loc, testNumber, name, details, directive, testId) {
+  fail(nesting, loc, testNumber, name, details, directive, testId, tags) {


General comment: this is getting to be a bit of a junk-drawer of arguments 😬

I agree, perhaps we could open a good first issue Issues that are suitable for first-time contributors. issue to handle this

atlowChemi · 2026-05-06T19:11:09Z

GitHub is having a bit of a stroke.

@JakobJingleheimer github having issues? no way 😂

aduh95 · 2026-05-18T12:00:39Z

 This feature requires `--allow-worker` if used with the [Permission Model][].

-### `--experimental-test-tag-filter=<tag>`
+### `--experimental-test-tag-filter='<expr>'`


I'm not sure adding quotes helps here, to me at least that seems more confusing (e.g. git docs do not do that: https://git-scm.com/docs/git-log)

Suggested change

### `--experimental-test-tag-filter='<expr>'`

### `--experimental-test-tag-filter=<expr>`

aduh95 · 2026-05-18T12:09:58Z

+* Identifiers—any non-whitespace, non-operator characters. A literal
+  identifier matches a tag of the same value (case-insensitive).
+* `*` wildcards inside an identifier match any sequence of characters.
+  A bare `*` matches any tagged test.
+* Boolean operators with two equivalent forms:
+  * `and` / `&&`
+  * `or` / `||`
+  * `not` / `!`
+* Parentheses for grouping.


I'm not too keen on creating a new language for that. Let's either use JS (e.g. --experimental-test-tag-filter='(tag) => tag.startsWith("foo-")'), RegEx (e.g. --experimental-test-tag-filter=/^foo-/), or let's not add it into core

Probably better if we have this conversation at: #63054 (comment)?

Having it in JS means we have a problem (can't serialize functions, so isolation: 'process' is not going to work straightforward). Using regex doesn't solve the problem of "is X or Y but not Z", which JS/DSL would..

MoLow

After discussing this face to face with @atlowChemi and @benjamingr I see the value in this. Considered some alternatives for allowing usage of tags via CLI, without the need to use run directly. it seems this is the best option.
compared to passing a function or a regex, and due to the fact it exists in other langauages and other test runners in the echosystem

benjamingr · 2026-08-09T19:42:33Z

I was also convinced:

Every other framework (e.g. vitest) even in other languages (e.g. pytest) does this as well as userland libraries for test tags.
Regex looks at a single value, so it can't do "AND" like the array at the moment
If we allow "recursive" array + regex, we're effectively inventing a DSL just less readable than every other test runner
Function requires either a serializable function (which is a footgun, closures and stuff) or an expensive postMessage back and forth for evaluating it on the parent

So I'm +1 on this, it took me using the feature a bunch to understand this

atlowChemi · 2026-08-09T19:48:38Z

or an expensive postMessage back and forth for evaluating it on the parent

This would also be expensive as it would mean the tag evaluation which currently happens synchronously in the Test constructor, would have to become async (as we would have to wait for the message event for the response) and would require some noisy changes to ensure the test can't start while the evaluation hasn't completed

atlowChemi · 2026-08-09T20:32:51Z

@JakobJingleheimer @aduh95 are you OK with landing this? (see Moshe & Benjamin's recent comments)

Upgrades the experimental tag filter introduced by stage 1 to accept a boolean expression instead of a literal tag name. Grammar: `and`/`&&`, `or`/`||`, `not`/`!`, parentheses for grouping, and `*` wildcards inside identifiers. Standard precedence (`not > and > or`); binary operators are left-associative. Word forms require whitespace separation; punctuation forms do not. Untagged tests evaluate `false` for any include expression and `true` for `not X`, so excluding tags does not accidentally remove untagged tests. The flag and `testTagFilters` option are still repeatable; multiple expressions still AND together. Malformed expressions fail fast at the parent process at startup. Tag value validation tightens to reject whitespace, operator characters (`& | ! ( ) *`), and the reserved words `and`/`or`/`not` in any casing - a breaking change relative to the stage 1 ship, acceptable at Stability 1.0 (Early development). Signed-off-by: atlowChemi <chemi@atlow.co.il>

nodejs-github-bot · 2026-08-10T11:32:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/75730/

nodejs-github-bot · 2026-08-10T13:36:31Z

CI: https://ci.nodejs.org/job/node-test-pull-request/75732/

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Apr 30, 2026

atlowChemi added the test_runner Issues and PRs related to the test runner subsystem. label Apr 30, 2026

MoLow reviewed Apr 30, 2026

View reviewed changes

Comment thread doc/api/cli.md

JakobJingleheimer added this to Test Runner Board Apr 30, 2026

TheOneTheOnlyJJ reviewed May 1, 2026

View reviewed changes

Comment thread test/parallel/test-runner-tag-filter-cli.mjs Outdated

atlowChemi force-pushed the test_runner-tag-filter branch 3 times, most recently from 3591324 to 31a7737 Compare May 5, 2026 16:47

JakobJingleheimer reviewed May 6, 2026

View reviewed changes

atlowChemi force-pushed the test_runner-tag-filter branch from 31a7737 to 853f715 Compare May 7, 2026 19:13

atlowChemi mentioned this pull request May 10, 2026

test_runner: add tags option and tag-name filter #63221

Merged

atlowChemi force-pushed the test_runner-tag-filter branch 2 times, most recently from 131d524 to 3f583b5 Compare May 12, 2026 17:51

atlowChemi changed the title ~~test_runner: add experimental tag-based test filtering~~ test_runner: extend tag filter with boolean expression DSL May 12, 2026

aduh95 reviewed May 18, 2026

View reviewed changes

MoLow approved these changes Aug 9, 2026

View reviewed changes

benjamingr approved these changes Aug 9, 2026

View reviewed changes

atlowChemi force-pushed the test_runner-tag-filter branch from 3f583b5 to f66e962 Compare August 9, 2026 20:27

MoLow approved these changes Aug 9, 2026

View reviewed changes

atlowChemi force-pushed the test_runner-tag-filter branch from e4a46c1 to 7c3d927 Compare August 9, 2026 21:08

benjamingr approved these changes Aug 9, 2026

View reviewed changes

MoLow added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 10, 2026

github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 10, 2026

		fail(nesting, loc, testNumber, name, details, directive, testId) {
		fail(nesting, loc, testNumber, name, details, directive, testId, tags) {

	### `--experimental-test-tag-filter='<expr>'`
	### `--experimental-test-tag-filter=<expr>`

Uh oh!

Uh oh!

Conversation

atlowChemi commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Prior art

Uh oh!

nodejs-github-bot commented Apr 30, 2026

Uh oh!

Uh oh!

codecov Bot commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JakobJingleheimer commented Apr 30, 2026

Uh oh!

Uh oh!

JakobJingleheimer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JakobJingleheimer May 6, 2026

Choose a reason for hiding this comment

Uh oh!

atlowChemi May 6, 2026

Choose a reason for hiding this comment

Uh oh!

atlowChemi commented May 6, 2026

Uh oh!

aduh95 May 18, 2026

Choose a reason for hiding this comment

Uh oh!

aduh95 May 18, 2026

Choose a reason for hiding this comment

Uh oh!

atlowChemi May 19, 2026

Choose a reason for hiding this comment

Uh oh!

MoLow left a comment

Choose a reason for hiding this comment

Uh oh!

benjamingr commented Aug 9, 2026

Uh oh!

atlowChemi commented Aug 9, 2026

Uh oh!

atlowChemi commented Aug 9, 2026

Uh oh!

nodejs-github-bot commented Aug 10, 2026

Uh oh!

nodejs-github-bot commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

atlowChemi commented Apr 30, 2026 •

edited

Loading

codecov Bot commented Apr 30, 2026 •

edited

Loading