How to fail the CI build if one of the test files duration exceeds a certain limit?

You can add Knapsack Pro Queue Hook that will be run after all tests are executed on the CI node using Knapsack Pro Queue Mode.

In the hook, you can read the recorded time execution of test files to detect the slowest test file. Then you can check if it exceeds a certain limit.

KnapsackPro::Hooks::Queue.after_queue do |queue_id|
  THE_SLOWEST_TEST_FILE_TIME_EXECUTION_LIMIT = 120 # in seconds

  # all recorded test files by knapsack_pro gem on particular CI node index
  test_files = []
  Dir.glob(".knapsack_pro/queue/#{queue_id}/*.json").each do |file|
    report = JSON.parse(File.read(file))
    test_files += report
  end

  slowest_test_file = test_files.max_by do |test_file|
    test_file['time_execution']
  end

  if slowest_test_file['time_execution'].to_f > THE_SLOWEST_TEST_FILE_TIME_EXECUTION_LIMIT
    puts '!'*50
    puts "The slowest test file took #{slowest_test_file['time_execution']} seconds and exceeded allowed max limit: #{THE_SLOWEST_TEST_FILE_TIME_EXECUTION_LIMIT} seconds. File path: #{slowest_test_file['path']}"
    puts '!'*50

    File.open('tmp/slowest_test_file_exceeded_allowed_limit.txt', 'w+') do |f|
      f.write(slowest_test_file.to_json)
    end
  end
end

If the file tmp/slowest_test_file_exceeded_allowed_limit.txt is created on the disk, then it means that a limit has been reached (THE_SLOWEST_TEST_FILE_TIME_EXECUTION_LIMIT).

You can create a bash script that will detect the file and exit with status 1 to make your CI build to fail.

You can add a new step in your CI pipeline and run the script bin/slowest_test_file_exceeded_allowed_limit.

#!/bin/bash
# bin/slowest_test_file_exceeded_allowed_limit

if [ -f tmp/slowest_test_file_exceeded_allowed_limit.txt ]; then
  cat tmp/slowest_test_file_exceeded_allowed_limit.txt
  echo
  echo "Too slow test file found. Fail CI build!"
  exit 1
fi

Make sure the file tmp/slowest_test_file_exceeded_allowed_limit.txt does not exist before each new CI build run. For most CI providers each new CI build happens using an isolated machine with a fresh copy of your repository so the tmp directory should not contain the file tmp/slowest_test_file_exceeded_allowed_limit.txt.