Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start fixing flakey tests ❄️ #3329

Merged
merged 6 commits into from
Sep 26, 2024
Merged

Conversation

pixlwave
Copy link
Member

@pixlwave pixlwave commented Sep 25, 2024

First round of changes and a non-failing run. Probably best to merge this so the Preview Tests are more stable and look at other flakey unit tests separately.

The PR has individual commits for each change.

Copy link

github-actions bot commented Sep 25, 2024

Warnings
⚠️ Please provide a description for this PR.
⚠️ Some of the commits are missing ticket numbers. Please consider squashing all commits that don't have a tracking number.

Generated by 🚫 Danger Swift against 134b8f4

@pixlwave pixlwave added the pr-build for changes related to build, tools, CI/CD label Sep 25, 2024
Copy link

codecov bot commented Sep 25, 2024

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
831 3 828 0
View the top 2 failed tests by shortest run time
UserIndicatorControllerTests testChainedPresentation()
Stack Traces | 2.55s run time
XCTAssertEqual failed: ("3") is not equal to ("2") (.../UnitTests/Sources/UserIndicatorControllerTests.swift:61)
CompletionSuggestionServiceTests testUserSuggestionsIncludingAllUsers()
Stack Traces | 12.2s run time
Asynchronous wait failed: Exceeded timeout of 10 seconds, with unfulfilled expectations: "Awaiting publisher". (.../Sources/Extensions/XCTestCase.swift:43)
View the full list of 1 ❄️ flaky tests
PreviewTests test_timelineItemBubbledStylerView()

Flake rate in main: 35.71% (Passed 36 times, Failed 20 times)

Stack Traces | 13.1s run time
failed - Snapshot "Pinned messages" does not match reference.

@−
"file:.../__Snapshots__/PreviewTests/test_timelineItemBubbledStylerView-iPhone-16-en-GB.Pinned-messages.png"
@+
"file:.../tmp/PreviewTests/test_timelineItemBubbledStylerView-iPhone-16-en-GB.Pinned-messages.png"

To configure output for a custom diff tool, use 'withSnapshotTesting'. For example:

    withSnapshotTesting(diffTool: .ksdiff) {
      // ...
    }

The percentage of pixels that match 0.8688965 is less than required 1.0
The lowest perceptual color precision 0.18624997 is less than required 0.98 (.../PreviewTests/Sources/PreviewTests.swift:84)

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

Copy link

sonarcloud bot commented Sep 26, 2024

@pixlwave pixlwave marked this pull request as ready for review September 26, 2024 13:21
@pixlwave pixlwave requested a review from a team as a code owner September 26, 2024 13:21
@pixlwave pixlwave requested review from stefanceriu and removed request for a team September 26, 2024 13:21
Copy link
Member

@stefanceriu stefanceriu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very very nice! Fingers crossed 🤞

@pixlwave pixlwave merged commit 4d7d687 into develop Sep 26, 2024
7 checks passed
@pixlwave pixlwave deleted the doug/try-fix-lots-of-tests branch September 26, 2024 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-build for changes related to build, tools, CI/CD
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants