ruby-changes:54952
From: k0kubun <ko1@a...>
Date: Fri, 1 Mar 2019 23:09:46 +0900 (JST)
Subject: [ruby-changes:54952] k0kubun:r67158 (trunk): Stop sharding --jit-wait testing
k0kubun 2019-03-01 23:09:41 +0900 (Fri, 01 Mar 2019) New Revision: 67158 https://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=67158 Log: Stop sharding --jit-wait testing by just randomizing test order. The original motivation to shard --jit-wait tests was forcing to test major parts of code without actually stopping to test MJIT after TracePoint enablement. But it tends to increase the test time because we often compile the same thing in different shards. I made this decision because we seem to hit 1.5h timeout of Wercker these days, and Wercker is really bad at handling timeout (it does not report timeout as failure, but just keeps it "pending" state) https://app.wercker.com/ruby/ruby/runs/mjit-test2/5c78f15cc9e725000805b86c?step=5c79031d6c1e2c0008ac41c3 By randomizing this, we could test things randomly. The downside of this approach is that we may not be able to find a specific commit that caused a future failure by having TracePoint in a very early phase. Modified files: trunk/wercker.yml Index: wercker.yml =================================================================== --- wercker.yml (revision 67157) +++ wercker.yml (revision 67158) @@ -6,27 +6,7 @@ box: ruby:2.5-stretch https://github.com/ruby/ruby/blob/trunk/wercker.yml#L6 no-response-timeout: 30 command-timeout: 60 -### Code to generate test-all definition with --jit-wait ### -# This aims to relax no-output timeout, and to isolate TracePoint testing since JIT is not supporting trace_* insns for now. - -# allow_failures = [] -# dirs = Dir.glob('test/*/').sort -# index = 1 -# puts <<-EOS -# - script: -# name: make test-all#{index} -- others (JIT wait) -# code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTOPTS="#{dirs.map { |d| "--exclude #{d}" }.join(' ')} --color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" -# EOS -# dirs.each do |dir| -# index += 1 -# puts <<-EOS -# - script: -# name: make test-all#{index} -- #{dir} (JIT wait#{(', allow failure' if allow_failures.include?(dir))}) -# code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="#{dir}" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait"#{(' || true' if allow_failures.include?(dir))} -# EOS -# end - -# --jit + first half of --jit-wait. +# --jit + --jit-wait (test, test-spec) mjit-test1: steps: - install-packages: @@ -55,7 +35,7 @@ mjit-test1: https://github.com/ruby/ruby/blob/trunk/wercker.yml#L35 name: make test-all (JIT) code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit --jit-warnings" TESTOPTS="-v --color=never --job-status=normal --longest 10 --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit" - # --jit-wait + # --jit-wait (test, test-spec) - script: name: make test (JIT wait) code: /usr/bin/sudo -H -u test -- make test RUN_OPTS="--disable-gems --jit-wait --jit-warnings" @@ -63,122 +43,6 @@ mjit-test1: https://github.com/ruby/ruby/blob/trunk/wercker.yml#L43 name: make test-spec (JIT wait) code: /usr/bin/sudo -H -u test -- make test-spec RUN_OPTS="--disable-gems --jit-wait --jit-warnings" - # -- AUTO GENERATED 1st half START (by above code) --- - - script: - name: make test-all1 -- others (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTOPTS="--exclude test/-ext-/ --exclude test/base64/ --exclude test/benchmark/ --exclude test/bigdecimal/ --exclude test/cgi/ --exclude test/coverage/ --exclude test/csv/ --exclude test/date/ --exclude test/dbm/ --exclude test/digest/ --exclude test/drb/ --exclude test/dtrace/ --exclude test/erb/ --exclude test/etc/ --exclude test/excludes/ --exclude test/fiddle/ --exclude test/fileutils/ --exclude test/gdbm/ --exclude test/io/ --exclude test/irb/ --exclude test/json/ --exclude test/lib/ --exclude test/logger/ --exclude test/matrix/ --exclude test/minitest/ --exclude test/misc/ --exclude test/mkmf/ --exclude test/monitor/ --exclude test/net/ --exclude test/nkf/ --exclude test/objspace/ --exclude test/open-uri/ --exclude test/openssl/ --exclude test/optparse/ --exclude test/ostruct/ --exclude test/pathname/ --exclude test/psych/ --exclude test/rdoc/ --exclude test/readline/ - -exclude test/resolv/ --exclude test/rexml/ --exclude test/rinda/ --exclude test/ripper/ --exclude test/rss/ --exclude test/ruby/ --exclude test/rubygems/ --exclude test/scanf/ --exclude test/sdbm/ --exclude test/shell/ --exclude test/socket/ --exclude test/stringio/ --exclude test/strscan/ --exclude test/syslog/ --exclude test/testunit/ --exclude test/uri/ --exclude test/webrick/ --exclude test/win32ole/ --exclude test/yaml/ --exclude test/zlib/ --color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all2 -- test/-ext-/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/-ext-/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all3 -- test/base64/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/base64/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all4 -- test/benchmark/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/benchmark/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all5 -- test/bigdecimal/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/bigdecimal/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all6 -- test/cgi/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/cgi/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all7 -- test/coverage/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/coverage/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all8 -- test/csv/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/csv/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all9 -- test/date/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/date/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all10 -- test/dbm/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/dbm/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all11 -- test/digest/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/digest/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all12 -- test/drb/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/drb/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all13 -- test/dtrace/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/dtrace/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all14 -- test/erb/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/erb/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all15 -- test/etc/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/etc/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all16 -- test/excludes/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/excludes/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all17 -- test/fiddle/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/fiddle/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all18 -- test/fileutils/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/fileutils/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all19 -- test/gdbm/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/gdbm/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all20 -- test/io/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/io/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all21 -- test/irb/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/irb/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all22 -- test/json/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/json/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all23 -- test/lib/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/lib/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all24 -- test/logger/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/logger/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all25 -- test/matrix/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/matrix/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all26 -- test/minitest/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/minitest/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all27 -- test/misc/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/misc/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all28 -- test/mkmf/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/mkmf/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all29 -- test/monitor/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/monitor/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all30 -- test/net/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/net/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all31 -- test/nkf/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/nkf/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all32 -- test/objspace/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/objspace/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all33 -- test/open-uri/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/open-uri/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all34 -- test/openssl/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/openssl/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all35 -- test/optparse/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/optparse/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all36 -- test/ostruct/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/ostruct/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all37 -- test/pathname/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/pathname/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all38 -- test/psych/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/psych/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - # -- AUTO GENERATED 1st half END --- after-steps: - wantedly/pretty-slack-notify: webhook_url: $SLACK_WEBHOOK_URL @@ -187,7 +51,7 @@ mjit-test1: https://github.com/ruby/ruby/blob/trunk/wercker.yml#L51 notify_on: "failed" branches: ^trunk$ -# second half of --jit-wait. +# --jit-wait (test-all) mjit-test2: steps: - install-packages: @@ -205,74 +69,11 @@ mjit-test2: https://github.com/ruby/ruby/blob/trunk/wercker.yml#L69 name: make all install code: /usr/bin/sudo -H -u test -- make -j$(nproc) all install - # -- AUTO GENERATED 2nd half START (by above code) --- - - script: - name: make test-all39 -- test/rdoc/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/rdoc/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all40 -- test/readline/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/readline/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all41 -- test/resolv/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/resolv/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all42 -- test/rexml/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/rexml/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all43 -- test/rinda/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/rinda/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all44 -- test/ripper/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/ripper/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all45 -- test/rss/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/rss/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" + # --jit-wait (test-all) - script: - name: make test-all46 -- test/ruby/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/ruby/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all47 -- test/rubygems/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/rubygems/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all48 -- test/scanf/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/scanf/" TESTOPTS="--color=never --job-status=normal --subprocess-timeout-scale=3.0 --excludes=test/excludes/_wercker/jit-wait" - - script: - name: make test-all49 -- test/sdbm/ (JIT wait) - code: /usr/bin/sudo -H -u test -- make test-all RUN_OPTS="--disable-gems --jit-wait --jit-warnings" TESTS="test/sdbm/" TESTOPTS="--color=n (... truncated) -- ML: ruby-changes@q... Info: http://www.atdot.net/~ko1/quickml/