ruby-changes:27490
From: drbrain <ko1@a...>
Date: Fri, 1 Mar 2013 07:26:47 +0900 (JST)
Subject: [ruby-changes:27490] drbrain:r39542 (trunk): * lib/rubygems/ext/builder.rb: Fix incompatibilities when installing
drbrain 2013-03-01 07:25:55 +0900 (Fri, 01 Mar 2013) New Revision: 39542 http://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=rev&revision=39542 Log: * lib/rubygems/ext/builder.rb: Fix incompatibilities when installing extensions. Patch by Nobu. [ruby-trunk - Bug #7968] [ruby-trunk - Bug #7971] * lib/rubygems/ext/ext_conf_builder.rb: ditto. * lib/rubygems/installer.rb: ditto. * test/rubygems/test_gem_ext_ext_conf_builder.rb: Test for the above. * test/rubygems/test_gem_installer.rb: ditto. * lib/rubygems/commands/sources_command.rb: Prefer HTTPS over HTTP. * lib/rubygems/defaults.rb: ditto * lib/rubygems/dependency_resolver.rb: Ditto. * lib/rubygems/source.rb: ditto. * lib/rubygems/spec_fetcher.rb: ditto. * lib/rubygems/specification.rb: ditto. * lib/rubygems/test_utilities.rb: ditto. * test/rubygems/test_gem.rb: Test for the above. * test/rubygems/test_gem_commands_sources_command.rb: ditto. * test/rubygems/test_gem_dependency_resolver_api_set.rb: ditto. * test/rubygems/test_gem_remote_fetcher.rb: ditto. * test/rubygems/test_gem_source.rb: ditto. * test/rubygems/test_gem_spec_fetcher.rb: ditto. Added files: trunk/test/rubygems/test_gem_dependency_resolver_api_set.rb Modified files: trunk/ChangeLog trunk/lib/rubygems/commands/sources_command.rb trunk/lib/rubygems/defaults.rb trunk/lib/rubygems/dependency_resolver.rb trunk/lib/rubygems/ext/builder.rb trunk/lib/rubygems/ext/ext_conf_builder.rb trunk/lib/rubygems/installer.rb trunk/lib/rubygems/source.rb trunk/lib/rubygems/spec_fetcher.rb trunk/lib/rubygems/specification.rb trunk/lib/rubygems/test_utilities.rb trunk/test/rubygems/test_gem.rb trunk/test/rubygems/test_gem_commands_sources_command.rb trunk/test/rubygems/test_gem_ext_ext_conf_builder.rb trunk/test/rubygems/test_gem_installer.rb trunk/test/rubygems/test_gem_remote_fetcher.rb trunk/test/rubygems/test_gem_source.rb trunk/test/rubygems/test_gem_spec_fetcher.rb Index: ChangeLog =================================================================== --- ChangeLog (revision 39541) +++ ChangeLog (revision 39542) @@ -1,3 +1,27 @@ https://github.com/ruby/ruby/blob/trunk/ChangeLog#L1 +Fri Mar 1 07:21:41 2013 Eric Hodel <drbrain@s...> + + * lib/rubygems/ext/builder.rb: Fix incompatibilities when installing + extensions. Patch by Nobu. + [ruby-trunk - Bug #7968] [ruby-trunk - Bug #7971] + * lib/rubygems/ext/ext_conf_builder.rb: ditto. + * lib/rubygems/installer.rb: ditto. + * test/rubygems/test_gem_ext_ext_conf_builder.rb: Test for the above. + * test/rubygems/test_gem_installer.rb: ditto. + + * lib/rubygems/commands/sources_command.rb: Prefer HTTPS over HTTP. + * lib/rubygems/defaults.rb: ditto + * lib/rubygems/dependency_resolver.rb: Ditto. + * lib/rubygems/source.rb: ditto. + * lib/rubygems/spec_fetcher.rb: ditto. + * lib/rubygems/specification.rb: ditto. + * lib/rubygems/test_utilities.rb: ditto. + * test/rubygems/test_gem.rb: Test for the above. + * test/rubygems/test_gem_commands_sources_command.rb: ditto. + * test/rubygems/test_gem_dependency_resolver_api_set.rb: ditto. + * test/rubygems/test_gem_remote_fetcher.rb: ditto. + * test/rubygems/test_gem_source.rb: ditto. + * test/rubygems/test_gem_spec_fetcher.rb: ditto. + Fri Mar 1 03:25:00 2013 Zachary Scott <zachary@z...> * ext/psych/lib/psych.rb: rdoc for Psych overview by Adam Stankiewicz Index: lib/rubygems/spec_fetcher.rb =================================================================== --- lib/rubygems/spec_fetcher.rb (revision 39541) +++ lib/rubygems/spec_fetcher.rb (revision 39542) @@ -188,6 +188,8 @@ class Gem::SpecFetcher https://github.com/ruby/ruby/blob/trunk/lib/rubygems/spec_fetcher.rb#L188 list = {} Gem.sources.each_source do |source| + source = upgrade_http_source source + begin names = case type when :latest @@ -225,5 +227,31 @@ class Gem::SpecFetcher https://github.com/ruby/ruby/blob/trunk/lib/rubygems/spec_fetcher.rb#L227 cache[source.uri] ||= source.load_specs(type) end end + + def upgrade_http_source source + uri = source.uri + + return source unless uri.scheme.downcase == 'http' + + https_uri = uri.dup + https_uri.scheme = 'https' + https_uri += '/' + + Gem::RemoteFetcher.fetcher.fetch_path https_uri, nil, true + + say "Upgraded #{uri} to HTTPS" + + https_uri += uri.request_uri + + source.uri = URI https_uri.to_s # cast to URI::HTTPS + + source + rescue Gem::RemoteFetcher::FetchError + say "Upgrading #{uri} to HTTPS failed, continuing" if + Gem.configuration.really_verbose + + source + end + end Index: lib/rubygems/ext/builder.rb =================================================================== --- lib/rubygems/ext/builder.rb (revision 39541) +++ lib/rubygems/ext/builder.rb (revision 39542) @@ -16,13 +16,6 @@ class Gem::Ext::Builder https://github.com/ruby/ruby/blob/trunk/lib/rubygems/ext/builder.rb#L16 raise Gem::InstallError, "Makefile not found:\n\n#{results.join "\n"}" end - mf = Gem.read_binary 'Makefile' - mf = mf.gsub(/^RUBYARCHDIR\s*=\s*\$[^$]*/, "RUBYARCHDIR = #{dest_path}") - mf = mf.gsub(/^RUBYLIBDIR\s*=\s*\$[^$]*/, "RUBYLIBDIR = #{dest_path}") - mf = mf.gsub(/\s*\S+\.time$/, "") - - File.open('Makefile', 'wb') {|f| f.print mf} - # try to find make program from Ruby configure arguments first RbConfig::CONFIG['configure_args'] =~ /with-make-prog\=(\w+)/ make_program = $1 || ENV['MAKE'] || ENV['make'] Index: lib/rubygems/ext/ext_conf_builder.rb =================================================================== --- lib/rubygems/ext/ext_conf_builder.rb (revision 39541) +++ lib/rubygems/ext/ext_conf_builder.rb (revision 39542) @@ -6,16 +6,36 @@ https://github.com/ruby/ruby/blob/trunk/lib/rubygems/ext/ext_conf_builder.rb#L6 require 'rubygems/ext/builder' require 'rubygems/command' +require 'fileutils' +require 'tmpdir' class Gem::Ext::ExtConfBuilder < Gem::Ext::Builder def self.build(extension, directory, dest_path, results, args=[]) - cmd = "#{Gem.ruby} #{File.basename extension}" + pwd = Dir.pwd + cmd = "#{Gem.ruby} -r./siteconf #{File.join pwd, File.basename(extension)}" cmd << " #{args.join ' '}" unless args.empty? - run cmd, results - - make dest_path, results + Dir.mktmpdir("gem-install.") do |tmpdir| + Dir.chdir(tmpdir) do + open("siteconf.rb", "w") do |f| + f.puts "require 'rbconfig'" + f.puts "dest_path = #{dest_path.dump}" + %w[sitearchdir sitelibdir].each do |dir| + f.puts "RbConfig::MAKEFILE_CONFIG['#{dir}'] = dest_path" + f.puts "RbConfig::CONFIG['#{dir}'] = dest_path" + end + end + + begin + run cmd, results + + make dest_path, results + ensure + FileUtils.mv("mkmf.log", pwd) if $! and File.exist?("mkmf.log") + end + end + end results end Index: lib/rubygems/dependency_resolver.rb =================================================================== --- lib/rubygems/dependency_resolver.rb (revision 39541) +++ lib/rubygems/dependency_resolver.rb (revision 39542) @@ -69,6 +69,8 @@ module Gem https://github.com/ruby/ruby/blob/trunk/lib/rubygems/dependency_resolver.rb#L69 # and dependencies. # class APISpecification + attr_reader :set # :nodoc: + def initialize(set, api_data) @set = set @name = api_data[:name] @@ -80,6 +82,14 @@ module Gem https://github.com/ruby/ruby/blob/trunk/lib/rubygems/dependency_resolver.rb#L82 attr_reader :name, :version, :dependencies + def == other # :nodoc: + self.class === other and + @set == other.set and + @name == other.name and + @version == other.version and + @dependencies == other.dependencies + end + def full_name "#{@name}-#{@version}" end @@ -91,6 +101,7 @@ module Gem https://github.com/ruby/ruby/blob/trunk/lib/rubygems/dependency_resolver.rb#L101 class APISet def initialize @data = Hash.new { |h,k| h[k] = [] } + @dep_uri = URI 'https://rubygems.org/api/v1/dependencies' end # Return data for all versions of the gem +name+. @@ -100,8 +111,8 @@ module Gem https://github.com/ruby/ruby/blob/trunk/lib/rubygems/dependency_resolver.rb#L111 return @data[name] end - u = URI.parse "http://rubygems.org/api/v1/dependencies?gems=#{name}" - str = Net::HTTP.get(u) + uri = @dep_uri + "?gems=#{name}" + str = Gem::RemoteFetcher.fetcher.fetch_path uri Marshal.load(str).each do |ver| @data[ver[:name]] << ver @@ -134,8 +145,8 @@ module Gem https://github.com/ruby/ruby/blob/trunk/lib/rubygems/dependency_resolver.rb#L145 return if needed.empty? - u = URI.parse "http://rubygems.org/api/v1/dependencies?gems=#{needed.join ','}" - str = Net::HTTP.get(u) + uri = @dep_uri + "?gems=#{needed.sort.join ','}" + str = Gem::RemoteFetcher.fetcher.fetch_path uri Marshal.load(str).each do |ver| @data[ver[:name]] << ver Index: lib/rubygems/source.rb =================================================================== --- lib/rubygems/source.rb (revision 39541) +++ lib/rubygems/source.rb (revision 39542) @@ -141,4 +141,14 @@ class Gem::Source https://github.com/ruby/ruby/blob/trunk/lib/rubygems/source.rb#L141 fetcher = Gem::RemoteFetcher.fetcher fetcher.download spec, @uri.to_s, dir end + + ## + # Replaces the URI for this source with +uri+. Used for upgrading this + # source to HTTPS + + def uri= uri # :nodoc: + @api_uri = nil + @uri = uri + end + end Index: lib/rubygems/specification.rb =================================================================== --- lib/rubygems/specification.rb (revision 39541) +++ lib/rubygems/specification.rb (revision 39542) @@ -17,7 +17,7 @@ https://github.com/ruby/ruby/blob/trunk/lib/rubygems/specification.rb#L17 # s.authors = ["Ruby Coder"] # s.email = 'rubycoder@e...' # s.files = ["lib/example.rb"] -# s.homepage = 'http://rubygems.org/gems/example' +# s.homepage = 'https://rubygems.org/gems/example' # end # # Starting in RubyGems 1.9.0, a Specification can hold arbitrary Index: lib/rubygems/installer.rb =================================================================== --- lib/rubygems/installer.rb (revision 39541) +++ lib/rubygems/installer.rb (revision 39542) @@ -681,7 +681,7 @@ TEXT https://github.com/ruby/ruby/blob/trunk/lib/rubygems/installer.rb#L681 say results.join("\n") if Gem.configuration.really_verbose end rescue - extension_build_error(extension_dir, results.join("\n")) + extension_build_error(extension_dir, results.join("\n"), $@) end end end @@ -689,7 +689,7 @@ TEXT https://github.com/ruby/ruby/blob/trunk/lib/rubygems/installer.rb#L689 ## # Logs the build +output+ in +build_dir+, then raises ExtensionBuildError. - def extension_build_error(build_dir, output) + def extension_build_error(build_dir, output, backtrace = nil) gem_make_out = File.join build_dir, 'gem_make.out' open gem_make_out, 'wb' do |io| io.puts output end @@ -703,7 +703,7 @@ Gem files will remain installed in #{gem https://github.com/ruby/ruby/blob/trunk/lib/rubygems/installer.rb#L703 Results logged to #{gem_make_out} EOF - raise ExtensionBuildError, message + raise ExtensionBuildError, message, backtrace end ## Index: lib/rubygems/commands/sources_command.rb =================================================================== --- lib/rubygems/commands/sources_command.rb (revision 39541) +++ lib/rubygems/commands/sources_command.rb (revision 39542) @@ -65,6 +65,19 @@ class Gem::Commands::SourcesCommand < Ge https://github.com/ruby/ruby/blob/trunk/lib/rubygems/commands/sources_command.rb#L65 end if source_uri = options[:add] then + uri = URI source_uri + + if uri.scheme and uri.scheme.downcase == 'http' and + uri.host.downcase == 'rubygems.org' then + question = <<-QUESTION.chomp +https://rubygems.org is recommended for security over #{uri} + +Do you want to add this insecure source? + QUESTION + + terminate_interaction 1 unless ask_yes_no question + end + source = Gem::Source.new source_uri begin Index: lib/rubygems/defaults.rb =================================================================== --- lib/rubygems/defaults.rb (revision 39541) +++ lib/rubygems/defaults.rb (revision 39542) @@ -11,7 +11,7 @@ module Gem https://github.com/ruby/ruby/blob/trunk/lib/rubygems/defaults.rb#L11 # An Array of the default sources that come with RubyGems def self.default_sources - %w[http://rubygems.org/] + %w[https://rubygems.org/] end ## Index: lib/rubygems/test_utilities.rb =================================================================== --- lib/rubygems/test_utilities.rb (revision 39541) +++ lib/rubygems/test_utilities.rb (revision 39542) @@ -49,7 +49,7 @@ class Gem::FakeFetcher https://github.com/ruby/ruby/blob/trunk/lib/rubygems/test_utilities.rb#L49 @data[path] end - def fetch_path path, mtime = nil + def fetch_path path, mtime = nil, head = false data = find_data(path) if data.respond_to?(:call) then Index: test/rubygems/test_gem_spec_fetcher.rb =================================================================== --- test/rubygems/test_gem_spec_fetcher.rb (revision 39541) +++ test/rubygems/test_gem_spec_fetcher.rb (revision 39542) @@ -32,13 +32,14 @@ class TestGemSpecFetcher < Gem::TestCase https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_spec_fetcher.rb#L32 Gem::NameTuple.new(spec.name, spec.version, spec.original_platform) } - v = Gem.marshal_version - s_zip = util_gzip(Marshal.dump(Gem::NameTuple.to_basic(@specs))) - l_zip = util_gzip(Marshal.dump(Gem::NameTuple.to_basic(@latest_specs))) - p_zip = util_gzip(Marshal.dump(Gem::NameTuple.to_basic(@prerelease_specs))) - @fetcher.data["#{@gem_repo}specs.#{v}.gz"] = s_zip - @fetcher.data["#{@gem_repo}latest_specs.#{v}.gz"] = l_zip - @fetcher.data["#{@gem_repo}prerelease_specs.#{v}.gz"] = p_zip + @v = Gem.marshal_version + @s_zip = util_gzip(Marshal.dump(Gem::NameTuple.to_basic(@specs))) + @l_zip = util_gzip(Marshal.dump(Gem::NameTuple.to_basic(@latest_specs))) + @p_zip = util_gzip(Marshal.dump(Gem::NameTuple.to_basic(@prerelease_specs))) + + @fetcher.data["#{@gem_repo}specs.#{@v}.gz"] = @s_zip + @fetcher.data["#{@gem_repo}latest_specs.#{@v}.gz"] = @l_zip + @fetcher.data["#{@gem_repo}prerelease_specs.#{@v}.gz"] = @p_zip @sf = Gem::SpecFetcher.new @@ -200,7 +201,6 @@ class TestGemSpecFetcher < Gem::TestCase https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_spec_fetcher.rb#L201 assert_equal comp.sort, specs[@source].sort end - def test_available_specs_cache specs, _ = @sf.available_specs(:latest) @@ -240,5 +240,36 @@ class TestGemSpecFetcher < Gem::TestCase https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_spec_fetcher.rb#L240 assert_kind_of Gem::SourceFetchProblem, errors.first end + def test_upgrade_http_source + Gem.configuration.verbose = :really + + source = Gem::Source.new URI 'http://example' + same_source = nil + https_source = nil + + use_ui @ui do + same_source = @sf.upgrade_http_source source + end + + assert_equal URI('http://example'), same_source.uri + + @fetcher.data['https://example/'] = 'hello' + + use_ui @ui do + https_source = @sf.upgrade_http_source source + end + + assert_equal URI('https://example'), https_source.uri + + assert_empty @ui.error + + expected = <<-EXPECTED +Upgrading http://example to HTTPS failed, continuing +Upgraded http://example to HTTPS + EXPECTED + + assert_equal expected, @ui.output + end + end Index: test/rubygems/test_gem_dependency_resolver_api_set.rb =================================================================== --- test/rubygems/test_gem_dependency_resolver_api_set.rb (revision 0) +++ test/rubygems/test_gem_dependency_resolver_api_set.rb (revision 39542) @@ -0,0 +1,80 @@ https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_dependency_resolver_api_set.rb#L1 +require 'rubygems/test_case' +require 'rubygems/dependency_resolver' + +class TestGemDependencyResolverAPISet < Gem::TestCase + + def setup + super + + @DR = Gem::DependencyResolver + + @api_set = @DR::APISet.new + @uri = 'https://rubygems.org/api/v1/dependencies' + @fetcher = Gem::FakeFetcher.new + Gem::RemoteFetcher.fetcher = @fetcher + end + + def test_find_all + b_entry = { + :name => 'b', + :number => '2', + :platform => 'ruby', + :dependencies => [['a', '>= 0']], + } + + @fetcher.data["#{@uri}?gems=b"] = Marshal.dump [b_entry] + + b_req = @DR::DependencyRequest.new dep('b', '>= 0'), nil + + expected = [ + @DR::APISpecification.new(@api_set, b_entry) + ] + + assert_equal expected, @api_set.find_all(b_req) + end + + def test_prefetch + b_entry = { + :name => 'b', + :number => '2', + :platform => 'ruby', + :dependencies => [['a', '>= 0']], + } + + a_entry = { + :name => 'a', + :number => '2', + :platform => 'ruby', + :dependencies => [], + } + + @fetcher.data["#{@uri}?gems=a,b"] = Marshal.dump [a_entry, b_entry] + + a_req = @DR::DependencyRequest.new dep('a', '>= 0'), nil + b_req = @DR::DependencyRequest.new dep('b', '>= 0'), nil + + @api_set.prefetch([b_req, a_req]) + + assert_equal [a_entry], @api_set.versions('a') + assert_equal [b_entry], @api_set.versions('b') + end + + def test_versions_cache + entry = { + :name => 'b', + :number => '2', + :platform => 'ruby', + :dependencies => [['a', '>= 0']], + } + + @fetcher.data["#{@uri}?gems=b"] = Marshal.dump [entry] + + assert_equal [entry], @api_set.versions('b') + + @fetcher.data["#{@uri}?gems=b"] = 'garbage' + + assert_equal [entry], @api_set.versions('b'), 'version data must be cached' + end + +end + Index: test/rubygems/test_gem_remote_fetcher.rb =================================================================== --- test/rubygems/test_gem_remote_fetcher.rb (revision 39541) +++ test/rubygems/test_gem_remote_fetcher.rb (revision 39542) @@ -399,6 +399,9 @@ gems: https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_remote_fetcher.rb#L399 @fetcher.instance_variable_set :@a1, @a1 @fetcher.instance_variable_set :@a2, @a2 def @fetcher.fetch_path uri, mtime = nil, head = false + raise Gem::RemoteFetcher::FetchError.new 'no http upgrade', uri if + uri.scheme != 'http' + case uri.request_uri when /#{@a1.spec_name}/ then Gem.deflate Marshal.dump @a1 Index: test/rubygems/test_gem.rb =================================================================== --- test/rubygems/test_gem.rb (revision 39541) +++ test/rubygems/test_gem.rb (revision 39542) @@ -665,7 +665,7 @@ class TestGem < Gem::TestCase https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem.rb#L665 end def test_self_default_sources - assert_equal %w[http://rubygems.org/], Gem.default_sources + assert_equal %w[https://rubygems.org/], Gem.default_sources end def test_self_detect_gemdeps Index: test/rubygems/test_gem_ext_ext_conf_builder.rb =================================================================== --- test/rubygems/test_gem_ext_ext_conf_builder.rb (revision 39541) +++ test/rubygems/test_gem_ext_ext_conf_builder.rb (revision 39542) @@ -30,7 +30,7 @@ class TestGemExtExtConfBuilder < Gem::Te https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_ext_ext_conf_builder.rb#L30 Gem::Ext::ExtConfBuilder.build 'extconf.rb', nil, @dest_path, output end - assert_match(/^#{Gem.ruby} extconf.rb/, output[0]) + assert_match(/^#{Gem.ruby} .*extconf.rb/, output[0]) assert_equal "creating Makefile\n", output[1] case RUBY_PLATFORM when /mswin/ then @@ -107,10 +107,10 @@ class TestGemExtExtConfBuilder < Gem::Te https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_ext_ext_conf_builder.rb#L107 assert_match(/\Aextconf failed: -#{Gem.ruby} extconf.rb.* +#{Gem.ruby} .*extconf.rb.* checking for main\(\) in .*?nonexistent/m, error.message) - assert_match(/^#{Gem.ruby} extconf.rb/, output[0]) + assert_match(/^#{Gem.ruby} .*extconf.rb/, output[0]) end def test_class_make @@ -134,12 +134,6 @@ checking for main\(\) in .*?nonexistent/ https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_ext_ext_conf_builder.rb#L134 assert_equal make_command, output[0] assert_equal "#{make_command} install", output[2] - - edited_makefile = Gem.read_binary makefile_path - edited_makefile.gsub!(/\r/, '') if Gem.win_platform? - - assert_match "\nRUBYARCHDIR = #{@ext}$(target_prefix)\n", edited_makefile - assert_match "\nRUBYLIBDIR = #{@ext}$(target_prefix)\n", edited_makefile end def test_class_make_no_Makefile Index: test/rubygems/test_gem_source.rb =================================================================== --- test/rubygems/test_gem_source.rb (revision 39541) +++ test/rubygems/test_gem_source.rb (revision 39542) @@ -184,5 +184,16 @@ class TestGemSource < Gem::TestCase https://github.com/ruby/ruby/blob/trunk/test/rubygems/test_gem_source.rb#L184 end end + def test_uri_equals + @source.api_uri # cached + + refute_equal URI('https://secure.example'), @source.api (... truncated) -- ML: ruby-changes@q... Info: http://www.atdot.net/~ko1/quickml/