https://git.ruby-lang.org/ruby.git/commit/?id=01138f5853

From 01138f5853a16068fb5a45ea39d3fc35fe664cb7 Mon Sep 17 00:00:00 2001
From: Yusuke Endoh <mame@r...>
Date: Sun, 16 Feb 2020 19:55:19 +0900
Subject: Make OpenSSL::OSSL#test_memcmp_timing robust

The test was too fragile.  Actually, it fails on one of our CIs
immediately after it was merged to ruby/ruby.

https://gist.github.com/ko1/7ea4a5826641f79e2f9e041d83e45dba#file-brlog-trunk_clang_40-20200216-101730-L532-L535
https://gist.github.com/ko1/1c657746092b871359d8bf9e0ad28921#file-brlog-trunk-test4-20200216-104518-L473-L476

* Two measurements, a-b and a-c, must be interative instead of
  sequential; the execution time will be easily affected by disturbance
  (say, cron job or some external process invoked during measurement)

* The comparison of the two results must be relative instead of
  absolute; slow machine may take several tens of seconds for each
  execution, and one delta second is too small.  The test cases of a, b,
  and c are very extreme, so if the target method has a bug, the two
  execution times would be very different.  So I think it is enough to
  check if the difference is less than 10 times.

This change is the same as https://github.com/ruby/openssl/pull/332

diff --git a/test/openssl/test_ossl.rb b/test/openssl/test_ossl.rb
index f517b1d..f83e813 100644
--- a/test/openssl/test_ossl.rb
+++ b/test/openssl/test_ossl.rb
@@ -52,10 +52,13 @@ class OpenSSL::OSSL < OpenSSL::SSLTestCase https://github.com/ruby/ruby/blob/trunk/test/openssl/test_ossl.rb#L52
     c = "y#{a}"
     a = "#{a}x"
 
-    n = 10_000
-    a_b_time = Benchmark.measure { n.times { OpenSSL.fixed_length_secure_compare(a, b) } }.real
-    a_c_time = Benchmark.measure { n.times { OpenSSL.fixed_length_secure_compare(a, c) } }.real
-    assert_in_delta(a_b_time, a_c_time, 1, "fixed_length_secure_compare timing test failed")
+    a_b_time = a_c_time = 0
+    100.times do
+      a_b_time += Benchmark.measure { 100.times { OpenSSL.fixed_length_secure_compare(a, b) } }.real
+      a_c_time += Benchmark.measure { 100.times { OpenSSL.fixed_length_secure_compare(a, c) } }.real
+    end
+    assert_operator(a_b_time, :<, a_c_time * 10, "fixed_length_secure_compare timing test failed")
+    assert_operator(a_c_time, :<, a_b_time * 10, "fixed_length_secure_compare timing test failed")
   end
 end
 
-- 
cgit v0.10.2


--
ML: ruby-changes@q...
Info: http://www.atdot.net/~ko1/quickml/