Introduce a waitForScripting helper function and use it in all scripting integration tests
#18405
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pting integration tests Code inspection uncovered that quite a few integration tests don't wait for scripting to be ready before proceeding, which is a source of intermittent failures, especially on slower machines where e.g. creating the sandbox takes longer. This commit fixes the issues by introducing a `waitForScripting` helper function and calling it consistently in all scripting integration tests.
timvandermeij
force-pushed
the
intermittent-waitforscripting
branch
from
0b4f04a
to
b540b63
Compare
|
I have looked into the impact of this change with diff 1 from #18396 (comment). This idea is to bring tests that don't properly await scripting initialization to the surface by slowing down sandbox creation, similar to what would happen on a slower machine such as the Windows bot. Before this patch running only the scripting integration tests with diff 1 applied resulted in 5 failures (note that we have seen some of them before on the bots): However, note that we are lucky with only 5 failures. There were quite a few more tests that didn't await scripting initialization, but those passed more-or-less "accidentally" because they are in the same Therefore, the only reason there wasn't more breakage is because we always run the integration tests in a fixed order, which is in contrast to the unit tests that we run in a random order and which would be nice to fix later. If we would run them in a random order those tests can't depend on the first test in the block to fix this for them anymore. This patch therefore also gets rid of an implicit dependency in the tests, which is a step towards eventually randomizing them. After this patch running only the scripting integration tests with diff 1 applied resulted in no failures anymore: |
|
/botio integrationtest |
From: Bot.io (Windows)ReceivedCommand cmd_integrationtest from @timvandermeij received. Current queue size: 0 Live output at: http://54.193.163.58:8877/4c9b86873c4b736/output.txt |
From: Bot.io (Linux m4)ReceivedCommand cmd_integrationtest from @timvandermeij received. Current queue size: 0 Live output at: http://54.241.84.105:8877/c169167d256b2da/output.txt |
From: Bot.io (Linux m4)SuccessFull output at http://54.241.84.105:8877/c169167d256b2da/output.txt Total script time: 7.95 mins
|
From: Bot.io (Windows)SuccessFull output at http://54.193.163.58:8877/4c9b86873c4b736/output.txt Total script time: 16.84 mins
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Code inspection uncovered that quite a few integration tests don't wait for scripting to be ready before proceeding, which is a source of intermittent failures, especially on slower machines where e.g. creating the sandbox takes longer.
This commit fixes the issues by introducing a
waitForScriptinghelper function and calling it consistently in all scripting integration tests.Fixes a part of #18396.