-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(baseline): Allow force creating fresh baseline #2124
Open
richardwerkman
wants to merge
43
commits into
master
Choose a base branch
from
baseline-command
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
43 commits
Select commit
Hold shift + click to select a range
853cbd9
init
dc5686f
Merge branch 'master' into baseline-command
richardwerkman e986e68
Fix unit tests
89b9cb6
Merge branch 'baseline-command' of https://github.com/stryker-mutator…
5c7705d
Merge branch 'master' of https://github.com/stryker-mutator/stryker-n…
68d4e39
Baseline recreate added
67db04d
merge main
4871f39
fix merge
36a4ce9
try to enable baseline by command
e180534
Fixed command
f1f6cf3
Fix mutation level
7dd8a89
merged master
e44bb01
Remove using
9ebf7f7
merged master
e1b4d14
Update docs + try to add target option to command
ca9b354
Merge master, fix and add tests, consistency (#2712)
danihengeveld 60d6d41
fix using
02193a7
Merge branch 'master' into baseline-command
richardwerkman 8f39401
Fix: StrykerCLITests info text test (#2713)
danihengeveld 938920d
Merge branch 'baseline-command' of https://github.com/stryker-mutator…
dc0f060
Pass command line arguments and options to subcommands (#2718)
danihengeveld 4d5303b
merged master
97d5c54
Merge branch 'baseline-command' of https://github.com/stryker-mutator…
0d9a5e6
Fix build
b530afa
Refactor but I broke it...
4f6c3cc
Merged master
a02f5c9
Fix commands
548d06b
Reset mutation level
ef6392f
Update docs
552c9a3
Remove "with-baseline.enabled"
0d62dda
Update docs and add help text for commands
e51eebd
Update pipeline docs
937f26f
merged master
d0fa85a
.
d7ac653
Fix tests
079311a
Merge branch 'master' into baseline-command
richardwerkman e87f5bb
Fix sonar
4c21604
Merge branch 'baseline-command' of https://github.com/stryker-mutator…
1202182
merged master
0380d09
Fix docs
0ba1d08
Fix unit tests
0dc89f4
merged
20cab04
Remove double logging and refactor
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -462,7 +462,7 @@ Config file: `"since": { }` | |||||
|
||||||
Use git information to test only code changes since the given target. Stryker will only report on mutants within the changed code. All other mutants will not have a result. | ||||||
|
||||||
If you wish to test only changed sources and tests but would like to have a complete mutation report see [with-baseline](#with-baseline-flag-committish). | ||||||
If you wish to test only changed sources and tests but would like to have a complete mutation report see [with-baseline](#with-baseline-committish). | ||||||
|
||||||
Set the diffing target on the command line by passing a committish with the since flag in the format `--since:<committish>`. | ||||||
Set the diffing target in the config file by setting the [since target](#sincetarget-committish) option. | ||||||
|
@@ -502,36 +502,26 @@ Use [globbing syntax](https://en.wikipedia.org/wiki/Glob_(programming)) for wild | |||||
|
||||||
## Baseline | ||||||
|
||||||
### `with-baseline` <`flag`> [`:committish`] | ||||||
### `with-baseline` <`committish`> | ||||||
|
||||||
Default: `false` | ||||||
Command line: `--with-baseline:feat-2` | ||||||
Config file: `"baseline": { }` | ||||||
Command line: `with-baseline feat-2` | ||||||
Config file: `"baseline": { }` | ||||||
|
||||||
Enabling `with-baseline` saves the mutation report to a storage location such as the filesystem. The mutation report is loaded at the start of the next mutation run. Any changed source code or unit test results in a reset of the mutants affected by the change. For unchanged mutants the previous result is reused. This feature expands on the [since](#since-flag-committish) feature by providing you with a full report after a partial mutation testrun. | ||||||
Enabling `baseline`, alias `with-baseline`, saves the mutation report to a storage location such as the filesystem. The mutation report is loaded at the start of the next mutation run. Any changed source code or unit test results in a reset of the mutants affected by the change. For unchanged mutants the previous result is reused. This feature expands on the [since](#since-flag-committish) feature by providing you with a full report after a partial mutation testrun. The target of the baseline is a `committish` passed as an extra argument. The target determines which committish is used to establish or locate a baseline. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why does not still need to be a committish? We are trying to move away from git for baseline. |
||||||
|
||||||
The report name is based on the current branch name or the [project-info.version](#project-infoversion-committish). | ||||||
|
||||||
Set the diffing target on the command line by passing a committish with the since flag. | ||||||
Set the diffing target in the config file by setting the [since target](#sincetarget-committish) option. | ||||||
|
||||||
*\* The baseline and since features are mutually exclusive. This feature implicitly enables the [since](#since-flag-committish) feature for now.* | ||||||
|
||||||
### `baseline.enabled` <`flag`> | ||||||
|
||||||
Default: `null` | ||||||
Command line: `N/A` | ||||||
Config file: `"baseline": { "enabled": false }` | ||||||
|
||||||
Enable or disable [with-baseline](#with-baseline-flag-committish). If the enabled property is not set but the `baseline` object exists in the config file it is assumed to be enabled. Use this option to (temporarily) disable `with-baseline` without having to delete the other baseline configuration. | ||||||
|
||||||
### `baseline.fallback-version` <`string`> | ||||||
|
||||||
Default: [since-target](#since-flag-committish) | ||||||
Command line: `N/A` | ||||||
Config file: `"baseline": { "fallback-version": 'develop' }` | ||||||
Config file: `"baseline": { "fallback-version": 'develop' }` | ||||||
|
||||||
When [with-baseline](#with-baseline-flag-committish) is enabled and Stryker cannot find an existing report for the current branch the fallback version is used. When Stryker is still unable to find a baseline we will do a complete instead of partial testrun. The complete testrun will then be saved as the new baseline for the next mutation testrun. | ||||||
When [with-baseline](#with-baseline-committish) is enabled and Stryker cannot find an existing report for the current branch the fallback version is used. When Stryker is still unable to find a baseline we will do a complete instead of partial testrun. The complete testrun will then be saved as the new baseline for the next mutation testrun. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
**Example**: | ||||||
```json | ||||||
|
@@ -566,9 +556,9 @@ new baseline saved to: feat-2 | |||||
|
||||||
Default: `Disk` | ||||||
Command line: `N/A` | ||||||
Config file: `"baseline": { "provider": 'AzureFileStorage'}` | ||||||
Config file: `"baseline": { "provider": 'AzureFileStorage'}` | ||||||
|
||||||
Sets the storage provider for the baseline used by [with-baseline](#with-baseline-flag-committish). By default this is set to disk, when the dashboard [reporter](#reporter-string) is enabled this is automatically set to Dashboard. | ||||||
Sets the storage provider for the baseline used by [with-baseline](#with-baseline-committish). By default this is set to disk, when the dashboard [reporter](#reporter-string) is enabled this is automatically set to Dashboard. | ||||||
|
||||||
Supported storage providers are: | ||||||
|
||||||
|
@@ -611,6 +601,14 @@ Allowed permissions: `Read`, `Write`, `Create` | |||||
|
||||||
For more information on how to configure a SAS check the [Azure documentation](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview). | ||||||
|
||||||
### `baseline recreate` | ||||||
|
||||||
Default: `false` | ||||||
Command line: `with-baseline main recreate` | ||||||
Config file: `N/A` | ||||||
|
||||||
Sometimes your baseline can get corrupted or out of touch with reality. In that case the baseline can be recreated using this command. This will test all mutations in your project and save the result as the new baseline. | ||||||
|
||||||
## Troubleshooting | ||||||
|
||||||
### `verbosity` <`log-level`> | ||||||
|
@@ -630,7 +628,7 @@ All available loglevels are | |||||
|
||||||
### `log-to-file` <`flag`> | ||||||
|
||||||
Default: `false` | ||||||
Default: `false` | ||||||
Command line: `[-L|--log-to-file]` | ||||||
Config file: `N/A` | ||||||
|
||||||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.