How to split slow RSpec test files by test examples (by individual it)?
❗ RSpec requirement: You need
RSpec >= 3.3.0 in order to use this feature.
In order to split RSpec slow test files by test examples across parallel CI nodes you need to set environment variable:
It allows a single test file to be auto split by test examples between parallel jobs (parallel CI nodes).
Thanks to that your CI build speed can be faster. It works with Knapsack Pro Regular Mode and Queue Mode.
If you find any issue to enable the feature, see common problems.
What are the benefits of RSpec split by test examples feature?
Your slow test files are the bottleneck and delay the whole CI build time. When you split slow test files by test examples in parallel CI nodes it means you remove the bottleneck and enable yourself to run much more parallel CI nodes than before to get much faster CI build time.
The more parallel CI nodes you use the more slow test files will be detected by
knapsack_progem and they will be split by test examples.
We recommend running at least 2 CI builds after you increase the number of parallel CI nodes. This way Knapsack Pro API can learn about your test examples execution time to better predict tests split in the future CI builds.
How to verify that RSpec split by test examples works as expected?
You can open one of your latest CI builds from the user dashboard.
When slow test files are detected then they will be highlighted in yellow.
|Test file path (2 files)||Time execution|
|spec/controllers/api/v3/books_controller_spec.rb||4 minutes 18.77 seconds|
|spec/features/books_spec.rb||2 minutes 23.824 seconds|
When you use RSpec split by examples feature then tests will be split by examples. RSpec adds ID of the test example
[1:1] at the end of a spec file path, for instance,
Here is a list of test files split by examples:
|Test file path (7 files)||Time execution|
|spec/controllers/api/v3/books_controller_spec.rb[1:1]||1 minutes 18.77 seconds|
As you can see test file paths to test examples are not highlighted in yellow when they are not a bottleneck anymore.
Sometimes it can happen that you have a single test example (
it test which does a super slow work, for instance, capybara test). In such a case when the test file path for the test example is highlighted in yellow then this is your bottleneck. You can refactor the test or think how to write 2 smaller and faster test examples.
How does the split by slow RSpec test files work?
Let's see an example. You have 2 test files:
spec/features/dashboard_spec.rb it takes 7 minutes to run. spec/features/homepage_spec.rb it takes 3 minutes to run.
In total, those 2 test files take 10 minutes to execute.
You run the test suite on 2 parallel CI nodes. In a perfect scenario, we would like to run each parallel job for 5 minutes because 10 minutes / 2 parallel jobs are 5 minutes. But we can't because one of the test files is slow. File
spec/features/dashboard_spec.rb takes 7 minutes to execute which is more than expected 5 minutes per parallel job. Because of this problem, your CI build will take 7 minutes to run instead of 5 minutes.
To fix the above problem we can use knapsack_pro auto split slow RSpec test files by test examples feature.
knapsack_pro knows that ideal expected time per parallel CI node is 5 minutes so it will look for test files that take more than
70% * 5 minutes = 3.5 minutes to run. It means if the test file takes
>= 3.5 minutes then it will be split by test examples between parallel CI nodes. We use the 70% threshold to adjust to randomness in the execution time of test files. The slower the test file the more probable is that it will run not always the same amount of time. This is especially happening for feature/capybara specs or when CI provider has some performance hiccup.
Is the test file split only by
RSpec test file is split by test examples. In RSpec syntax test example can be marked as
Why does knapsack_pro not split all test files by test examples?
Running test files by test examples adds additional overhead for memory usage in RSpec. If you have a very large test suite (>= 1000 test files) then running all test files by test examples would lead to running out of RAM on your CI provider.
There would be no point splitting all test files by test examples. Instead,
knapsack_pro gem determines which test files pose a real bottleneck for your CI build and ensures those specific files are split. It guarantees optimal performance.
Why do I see an error like:
LoadError: cannot load such file -- example-ruby-gem?
example-ruby-gem gem in your
Rakefile so when
knapsack_pro gem is running a rake task the
example-ruby-gem gem is not loaded. You can fix your project configuration by not loading
Or if you need it, then please ensure you add
:test group in the
Why do I see an error: Don't know how to build task
If you see an error like:
Don't know how to build task 'knapsack_pro:rspec_test_example_detector' (See the list of available tasks with `rake --tasks`)
It probably means bundler can't find the rake task. You can try to remove the default prefix
bundle exec used by
knapsack_pro gem by setting
Warning: don't use deprecated RSpec
If you use the deprecated RSpec option
run_all_when_everything_filtered (learn) it can cause skipping some of the tests when
knapsack_pro will try to run a test example and a test file at the same time. Do not use below deprecated flags!
# DO NOT USE THIS DEPRECATED OPTIONS RSpec.configure do |c| c.filter_run :focus => true c.run_all_when_everything_filtered = true end
The latest RSpec version provides a new option named
filter_run_when_matching (learn). You can use it instead of the above deprecated option.
# this can be used instead of run_all_when_everything_filtered RSpec.configure do |c| c.filter_run_when_matching :focus end
How to auto split slow test files for other test runners
See other examples how to auto split test files by test cases on parallel jobs (CI nodes).