Recent versions of Rails (or more honestly, Rack) introduced the ability to stream your responses to HTTP requests. This has a couple of great benefits, it starts the page rendering faster for the end user, and it prevents the creation of arbitrarily large buffers on your server (which can be a performance killer).  This is provided “for free” for a regular, templated HTML page.  If you’re generating very large XML or JSON files where you would like to do something similar, you’ve got to do it manually.

In order to stream data out of your app, you simply need to pass an object that responds to “each” (and doesn’t respond to “to_s”) to the render call.  Every time the callback passed in is invoked, Rack writes the string passed in to the outgoing stream.  In older versions of Rails and Rack we had an object that we could directly write to, but that’s no longer the case.

Under the hood, the ActiveSupport #to_xml commands use a Builder::XmlMarkup object in order to do their magic, and you can pass your own instance in using the optional parameter :builder.  Builder::XmlMarkup#new takes an optional parameter :target that is an object that needs to respond to :<<.  It will then call :<< on the target instead of just writing to an internal string.

So that leaves us with a way to stream that requires us to respond to a callback, and a way to write a stream that requires us to respond to a callback.  It’d be nice if one of them would give us an object we could pass to the other, but no such luck.  We’ll have to proxy the request across from the Builder to Rack.  To do this, I made a class that takes the object to be serialized and responds to each. When each is invoked, the object is serialized using a builder with a custom target. That custom target responds to << by invoking the callback passed in to it on initialization.  The code is easier to follow than words, so here’s an example:

class XmlStreamer
  def initialize(obj)
    @obj=obj
  end

  def each &callback
    @obj.to_xml(builder: Builder::XmlMarkup.new(target: XmlStreamerCallback.new(callback)))
  end
end

class XmlStreamerCallback
  def initialize(callback)
    @callback = callback
  end

  def << xml
    @callback.call(xml)
  end
end

# In our controller...
respond_with data do |f|
  f.xml  { render xml: XmlStreamer.new(data) }
end

This example is NOT a generic implementation of this pattern.  A full implementation should also manage the parameters that can be passed to to_xml, and may even be implemented as a method on the original object.  However, this should give you a starting point for streaming your own XML building.

I plan have a JSON version of this up soon.