ruby-changes:50813
From: usa <ko1@a...>
Date: Wed, 28 Mar 2018 23:44:25 +0900 (JST)
Subject: [ruby-changes:50813] usa:r63020 (ruby_2_2): merge revision(s) 60584, 62954-62959, 63008:
usa 2018-03-28 23:44:20 +0900 (Wed, 28 Mar 2018) New Revision: 63020 https://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=63020 Log: merge revision(s) 60584,62954-62959,63008: webrick: support Proc objects as body responses * lib/webrick/httpresponse.rb (send_body): call send_body_proc (send_body_proc): new method (class ChunkedWrapper): new class * test/webrick/test_httpresponse.rb (test_send_body_proc): new test (test_send_body_proc_chunked): ditto [Feature #855] webrick: favor .write over << method This will make the next change to use IO.copy_stream easier-to-read. When we can drop Ruby 2.4 support in a few years, this will allow us to use writev(2) with multiple arguments for headers and chunked responses. * lib/webrick/cgi.rb (write): new wrapper method lib/webrick/httpresponse.rb: (send_header): use socket.write (send_body_io): ditto (send_body_string): ditto (send_body_proc): ditto (_write_data): ditto (ChunkedWrapper#write): ditto (_send_file): ditto ------------------------------------------------------------------------ r62954 | normal | 2018-03-28 17:05:52 +0900 (?\230?\176?\180, 28 3 2018) | 14 lines webrick/httpresponse: IO.copy_stream for regular files Remove the redundant _send_file method since its functionality is unnecessary with IO.copy_stream. IO.copy_stream also allows the use of sendfile under some OSes to speed up copies to non-TLS sockets. Testing with "curl >/dev/null" and "ruby -run -e httpd" to read a 1G file over Linux loopback reveals a reduction from around ~0.770 to ~0.490 seconds on the client side. * lib/webrick/httpresponse.rb (send_body_io): use IO.copy_stream (_send_file): remove [Feature #14237] ------------------------------------------------------------------------ r62955 | normal | 2018-03-28 17:05:57 +0900 (?\230?\176?\180, 28 3 2018) | 10 lines webrick: use IO.copy_stream for single range response This is also compatible with range responses generated by Rack::File (tested with rack 2.0.3). * lib/webrick/httpresponse.rb (send_body_io): use Content-Range * lib/webrick/httpservlet/filehandler.rb (make_partial_content): use File object for the single range case * test/webrick/test_filehandler.rb (get_res_body): use send_body to test result ------------------------------------------------------------------------ r62956 | normal | 2018-03-28 17:06:02 +0900 (?\230?\176?\180, 28 3 2018) | 7 lines test/webrick/test_filehandler.rb: stricter multipart range test We need to ensure we generate compatibile output in the face of future changes * test/webrick/test_filehandler.rb (test_make_partial_content): check response body ------------------------------------------------------------------------ r62957 | normal | 2018-03-28 17:06:08 +0900 (?\230?\176?\180, 28 3 2018) | 8 lines webrick: quiet warning for multi-part ranges Content-Length is ignored by WEBrick::HTTPResponse even if we calculate it, so instead we chunk responses to HTTP/1.1 clients and terminate HTTP/1.0 connections. * lib/webrick/httpservlet/filehandler.rb (make_partial_content): quiet warning ------------------------------------------------------------------------ r62958 | normal | 2018-03-28 17:06:13 +0900 (?\230?\176?\180, 28 3 2018) | 7 lines webrick/httpresponse: make ChunkedWrapper copy_stream-compatible The .write method needs to return the number of bytes written to avoid confusing IO.copy_stream. * lib/webrick/httpresponse.rb (ChunkedWrapper#write): return bytes written (ChunkedWrapper#<<): return self ------------------------------------------------------------------------ r62959 | normal | 2018-03-28 17:06:18 +0900 (?\230?\176?\180, 28 3 2018) | 9 lines webrick: use IO.copy_stream for multipart response Use the new Proc response body feature to generate a multipart range response dynamically. We use a flat array to minimize object overhead as much as possible; as many ranges may fit into an HTTP request header. * lib/webrick/httpservlet/filehandler.rb (multipart_body): new method (make_partial_content): use multipart_body get rid of test error/failure on Windows introduced at r62955 * lib/webrick/httpresponse.rb (send_body_io): use seek if NotImplementedError is raised in IO.copy_stream with offset. * lib/webrick/httpservlet/filehandler.rb (multipart_body): ditto. Modified directories: branches/ruby_2_2/ Modified files: branches/ruby_2_2/ChangeLog branches/ruby_2_2/lib/webrick/httpresponse.rb branches/ruby_2_2/lib/webrick/httpservlet/filehandler.rb branches/ruby_2_2/test/webrick/test_filehandler.rb branches/ruby_2_2/test/webrick/test_httpresponse.rb branches/ruby_2_2/version.h Index: ruby_2_2/ChangeLog =================================================================== --- ruby_2_2/ChangeLog (revision 63019) +++ ruby_2_2/ChangeLog (revision 63020) @@ -1,3 +1,101 @@ https://github.com/ruby/ruby/blob/trunk/ruby_2_2/ChangeLog#L1 +Wed Mar 28 23:41:53 2018 NAKAMURA Usaku <usa@r...> + + get rid of test error/failure on Windows introduced at r62955 + + * lib/webrick/httpresponse.rb (send_body_io): use seek if + NotImplementedError is raised in IO.copy_stream with offset. + + * lib/webrick/httpservlet/filehandler.rb (multipart_body): ditto. + +Wed Mar 28 23:41:53 2018 Eric Wong <normalperson@y...> + + webrick: support Proc objects as body responses + + * lib/webrick/httpresponse.rb (send_body): call send_body_proc + (send_body_proc): new method + (class ChunkedWrapper): new class + + * test/webrick/test_httpresponse.rb (test_send_body_proc): new test + (test_send_body_proc_chunked): ditto + [Feature #855] + + webrick: favor .write over << method + + This will make the next change to use IO.copy_stream + easier-to-read. When we can drop Ruby 2.4 support in a few + years, this will allow us to use writev(2) with multiple + arguments for headers and chunked responses. + + * lib/webrick/cgi.rb (write): new wrapper method + lib/webrick/httpresponse.rb: (send_header): use socket.write + (send_body_io): ditto + (send_body_string): ditto + (send_body_proc): ditto + (_write_data): ditto + (ChunkedWrapper#write): ditto + (_send_file): ditto + + webrick/httpresponse: IO.copy_stream for regular files + + Remove the redundant _send_file method since its functionality + is unnecessary with IO.copy_stream. IO.copy_stream also allows + the use of sendfile under some OSes to speed up copies to + non-TLS sockets. + + Testing with "curl >/dev/null" and "ruby -run -e httpd" to + read a 1G file over Linux loopback reveals a reduction from + around ~0.770 to ~0.490 seconds on the client side. + + * lib/webrick/httpresponse.rb (send_body_io): use IO.copy_stream + (_send_file): remove + [Feature #14237] + + webrick: use IO.copy_stream for single range response + + This is also compatible with range responses generated + by Rack::File (tested with rack 2.0.3). + + * lib/webrick/httpresponse.rb (send_body_io): use Content-Range + * lib/webrick/httpservlet/filehandler.rb (make_partial_content): + use File object for the single range case + * test/webrick/test_filehandler.rb (get_res_body): use send_body + to test result + + test/webrick/test_filehandler.rb: stricter multipart range test + + We need to ensure we generate compatibile output in + the face of future changes + + * test/webrick/test_filehandler.rb (test_make_partial_content): + check response body + + webrick: quiet warning for multi-part ranges + + Content-Length is ignored by WEBrick::HTTPResponse even if we + calculate it, so instead we chunk responses to HTTP/1.1 clients + and terminate HTTP/1.0 connections. + + * lib/webrick/httpservlet/filehandler.rb (make_partial_content): + quiet warning + + webrick/httpresponse: make ChunkedWrapper copy_stream-compatible + + The .write method needs to return the number of bytes written + to avoid confusing IO.copy_stream. + + * lib/webrick/httpresponse.rb (ChunkedWrapper#write): return bytes written + (ChunkedWrapper#<<): return self + + webrick: use IO.copy_stream for multipart response + + Use the new Proc response body feature to generate a multipart + range response dynamically. We use a flat array to minimize + object overhead as much as possible; as many ranges may fit + into an HTTP request header. + + * lib/webrick/httpservlet/filehandler.rb (multipart_body): new method + (make_partial_content): use multipart_body + Wed Mar 28 23:37:18 2018 Nobuyoshi Nakada <nobu@r...> pack.c: fix underflow Index: ruby_2_2/lib/webrick/httpservlet/filehandler.rb =================================================================== --- ruby_2_2/lib/webrick/httpservlet/filehandler.rb (revision 63019) +++ ruby_2_2/lib/webrick/httpservlet/filehandler.rb (revision 63020) @@ -86,6 +86,35 @@ module WEBrick https://github.com/ruby/ruby/blob/trunk/ruby_2_2/lib/webrick/httpservlet/filehandler.rb#L86 return false end + # returns a lambda for webrick/httpresponse.rb send_body_proc + def multipart_body(body, parts, boundary, mtype, filesize) + lambda do |socket| + begin + begin + first = parts.shift + last = parts.shift + socket.write( + "--#{boundary}#{CRLF}" \ + "Content-Type: #{mtype}#{CRLF}" \ + "Content-Range: bytes #{first}-#{last}/#{filesize}#{CRLF}" \ + "#{CRLF}" + ) + + begin + IO.copy_stream(body, socket, last - first + 1, first) + rescue NotImplementedError + body.seek(first, IO::SEEK_SET) + IO.copy_stream(body, socket, last - first + 1) + end + socket.write(CRLF) + end while parts[0] + socket.write("--#{boundary}--#{CRLF}") + ensure + body.close + end + end + end + def make_partial_content(req, res, filename, filesize) mtype = HTTPUtils::mime_type(filename, @config[:MimeTypes]) unless ranges = HTTPUtils::parse_range_header(req['range']) @@ -96,37 +125,27 @@ module WEBrick https://github.com/ruby/ruby/blob/trunk/ruby_2_2/lib/webrick/httpservlet/filehandler.rb#L125 if ranges.size > 1 time = Time.now boundary = "#{time.sec}_#{time.usec}_#{Process::pid}" - body = '' - ranges.each{|range| - first, last = prepare_range(range, filesize) - next if first < 0 - io.pos = first - content = io.read(last-first+1) - body << "--" << boundary << CRLF - body << "Content-Type: #{mtype}" << CRLF - body << "Content-Range: bytes #{first}-#{last}/#{filesize}" << CRLF - body << CRLF - body << content - body << CRLF + parts = [] + ranges.each {|range| + prange = prepare_range(range, filesize) + next if prange[0] < 0 + parts.concat(prange) } - raise HTTPStatus::RequestRangeNotSatisfiable if body.empty? - body << "--" << boundary << "--" << CRLF + raise HTTPStatus::RequestRangeNotSatisfiable if parts.empty? res["content-type"] = "multipart/byteranges; boundary=#{boundary}" - res.body = body + if req.http_version < '1.1' + res['connection'] = 'close' + else + res.chunked = true + end + res.body = multipart_body(io.dup, parts, boundary, mtype, filesize) elsif range = ranges[0] first, last = prepare_range(range, filesize) raise HTTPStatus::RequestRangeNotSatisfiable if first < 0 - if last == filesize - 1 - content = io.dup - content.pos = first - else - io.pos = first - content = io.read(last-first+1) - end res['content-type'] = mtype res['content-range'] = "bytes #{first}-#{last}/#{filesize}" res['content-length'] = last - first + 1 - res.body = content + res.body = io.dup else raise HTTPStatus::BadRequest end Index: ruby_2_2/lib/webrick/httpresponse.rb =================================================================== --- ruby_2_2/lib/webrick/httpresponse.rb (revision 63019) +++ ruby_2_2/lib/webrick/httpresponse.rb (revision 63020) @@ -302,6 +302,8 @@ module WEBrick https://github.com/ruby/ruby/blob/trunk/ruby_2_2/lib/webrick/httpresponse.rb#L302 def send_body(socket) # :nodoc: if @body.respond_to? :readpartial then send_body_io(socket) + elsif @body.respond_to?(:call) then + send_body_proc(socket) else send_body_string(socket) end @@ -404,9 +406,20 @@ module WEBrick https://github.com/ruby/ruby/blob/trunk/ruby_2_2/lib/webrick/httpresponse.rb#L406 end _write_data(socket, "0#{CRLF}#{CRLF}") else - size = @header['content-length'].to_i - _send_file(socket, @body, 0, size) - @sent_size = size + if %r{\Abytes (\d+)-(\d+)/\d+\z} =~ @header['content-range'] + offset = $1.to_i + size = $2.to_i - offset + 1 + else + offset = nil + size = @header['content-length'] + size = size.to_i if size + end + begin + @sent_size = IO.copy_stream(@body, socket, size, offset) + rescue NotImplementedError + @body.seek(offset, IO::SEEK_SET) + @sent_size = IO.copy_stream(@body, socket, size) + end end ensure @body.close @@ -435,24 +448,41 @@ module WEBrick https://github.com/ruby/ruby/blob/trunk/ruby_2_2/lib/webrick/httpresponse.rb#L448 end end - def _send_file(output, input, offset, size) - while offset > 0 - sz = @buffer_size < size ? @buffer_size : size - buf = input.read(sz) - offset -= buf.bytesize + def send_body_proc(socket) + if @request_method == "HEAD" + # do nothing + elsif chunked? + @body.call(ChunkedWrapper.new(socket, self)) + _write_data(socket, "0#{CRLF}#{CRLF}") + else + size = @header['content-length'].to_i + @body.call(socket) + @sent_size = size end + end - if size == 0 - while buf = input.read(@buffer_size) - _write_data(output, buf) - end - else - while size > 0 - sz = @buffer_size < size ? @buffer_size : size - buf = input.read(sz) - _write_data(output, buf) - size -= buf.bytesize - end + class ChunkedWrapper + def initialize(socket, resp) + @socket = socket + @resp = resp + end + + def write(buf) + return 0 if buf.empty? + socket = @socket + @resp.instance_eval { + size = buf.bytesize + data = "#{size.to_s(16)}#{CRLF}#{buf}#{CRLF}" + _write_data(socket, data) + data.clear + @sent_size += size + size + } + end + + def <<(*buf) + write(buf) + self end end Index: ruby_2_2/version.h =================================================================== --- ruby_2_2/version.h (revision 63019) +++ ruby_2_2/version.h (revision 63020) @@ -1,6 +1,6 @@ https://github.com/ruby/ruby/blob/trunk/ruby_2_2/version.h#L1 #define RUBY_VERSION "2.2.10" #define RUBY_RELEASE_DATE "2018-03-28" -#define RUBY_PATCHLEVEL 486 +#define RUBY_PATCHLEVEL 487 #define RUBY_RELEASE_YEAR 2018 #define RUBY_RELEASE_MONTH 3 Index: ruby_2_2/test/webrick/test_filehandler.rb =================================================================== --- ruby_2_2/test/webrick/test_filehandler.rb (revision 63019) +++ ruby_2_2/test/webrick/test_filehandler.rb (revision 63020) @@ -14,16 +14,10 @@ class WEBrick::TestFileHandler < Test::U https://github.com/ruby/ruby/blob/trunk/ruby_2_2/test/webrick/test_filehandler.rb#L14 end def get_res_body(res) - body = res.body - if defined? body.read - begin - body.read - ensure - body.close - end - else - body - end + sio = StringIO.new + sio.binmode + res.send_body(sio) + sio.string end def make_range_request(range_spec) @@ -75,6 +69,23 @@ class WEBrick::TestFileHandler < Test::U https://github.com/ruby/ruby/blob/trunk/ruby_2_2/test/webrick/test_filehandler.rb#L69 res = make_range_response(filename, "bytes=0-0, -2") assert_match(%r{^multipart/byteranges}, res["content-type"]) + body = get_res_body(res) + boundary = /; boundary=(.+)/.match(res['content-type'])[1] + off = filesize - 2 + last = filesize - 1 + + exp = "--#{boundary}\r\n" \ + "Content-Type: text/plain\r\n" \ + "Content-Range: bytes 0-0/#{filesize}\r\n" \ + "\r\n" \ + "#{IO.read(__FILE__, 1)}\r\n" \ + "--#{boundary}\r\n" \ + "Content-Type: text/plain\r\n" \ + "Content-Range: bytes #{off}-#{last}/#{filesize}\r\n" \ + "\r\n" \ + "#{IO.read(__FILE__, 2, off)}\r\n" \ + "--#{boundary}--\r\n" + assert_equal exp, body end def test_filehandler Index: ruby_2_2/test/webrick/test_httpresponse.rb =================================================================== --- ruby_2_2/test/webrick/test_httpresponse.rb (revision 63019) +++ ruby_2_2/test/webrick/test_httpresponse.rb (revision 63020) @@ -145,5 +145,38 @@ module WEBrick https://github.com/ruby/ruby/blob/trunk/ruby_2_2/test/webrick/test_httpresponse.rb#L145 } assert_equal 0, logger.messages.length end + + def test_send_body_proc + @res.body = Proc.new { |out| out.write('hello') } + IO.pipe do |r, w| + @res.send_body(w) + w.close + r.binmode + assert_equal 'hello', r.read + end + assert_equal 0, logger.messages.length + end + + def test_send_body_proc_chunked + @res.body = Proc.new { |out| out.write('hello') } + @res.chunked = true + IO.pipe do |r, w| + @res.send_body(w) + w.close + r.binmode + assert_equal "5\r\nhello\r\n0\r\n\r\n", r.read + end + assert_equal 0, logger.messages.length + end + + def test_set_error + status = 400 + message = 'missing attribute' + @res.status = status + error = WEBrick::HTTPStatus[status].new(message) + body = @res.set_error(error) + assert_match(/#{@res.reason_phrase}/, body) + assert_match(/#{message}/, body) + end end end Index: ruby_2_2 =================================================================== --- ruby_2_2 (revision 63019) +++ ruby_2_2 (revision 63020) Property changes on: ruby_2_2 ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /trunk:r60584,62954-62959 -- ML: ruby-changes@q... Info: http://www.atdot.net/~ko1/quickml/