Category Archives: Ruby

Streaming XML in Rails 3.2 using to_xml

Recent versions of Rails (or more honestly, Rack) introduced the ability to stream your responses to HTTP requests. This has a couple of great benefits, it starts the page rendering faster for the end user, and it prevents the creation of arbitrarily large buffers on your server (which can be a performance killer).  This is provided “for free” for a regular, templated HTML page.  If you’re generating very large XML or JSON files where you would like to do something similar, you’ve got to do it manually.

In order to stream data out of your app, you simply need to pass an object that responds to “each” (and doesn’t respond to “to_s”) to the render call.  Every time the callback passed in is invoked, Rack writes the string passed in to the outgoing stream.  In older versions of Rails and Rack we had an object that we could directly write to, but that’s no longer the case.

Under the hood, the ActiveSupport #to_xml commands use a Builder::XmlMarkup object in order to do their magic, and you can pass your own instance in using the optional parameter :builder.  Builder::XmlMarkup#new takes an optional parameter :target that is an object that needs to respond to :<<.  It will then call :<< on the target instead of just writing to an internal string.

So that leaves us with a way to stream that requires us to respond to a callback, and a way to write a stream that requires us to respond to a callback.  It’d be nice if one of them would give us an object we could pass to the other, but no such luck.  We’ll have to proxy the request across from the Builder to Rack.  To do this, I made a class that takes the object to be serialized and responds to each. When each is invoked, the object is serialized using a builder with a custom target. That custom target responds to << by invoking the callback passed in to it on initialization.  The code is easier to follow than words, so here’s an example:

class XmlStreamer
  def initialize(obj)
    @obj=obj
  end

  def each &callback
    @obj.to_xml(builder: Builder::XmlMarkup.new(target: XmlStreamerCallback.new(callback)))
  end
end

class XmlStreamerCallback
  def initialize(callback)
    @callback = callback
  end

  def << xml
    @callback.call(xml)
  end
end

# In our controller...
respond_with data do |f|
  f.xml  { render xml: XmlStreamer.new(data) }
end

This example is NOT a generic implementation of this pattern.  A full implementation should also manage the parameters that can be passed to to_xml, and may even be implemented as a method on the original object.  However, this should give you a starting point for streaming your own XML building.

I plan have a JSON version of this up soon.

Rails 3 Engines/Plugins and Static Assets

THIS INFORMATION IS OUT OF DATE!  Since Rails 3.2, you can easily include static assets in your engine using the “app/assets” folder.  This post is only applicable for older versions of Rails, and still not suggested for production systems.

Engine-ish

One of the big changes in Rails 3 is the move to “everything is an engine”.  You end up reading a lot about this in relation to plugins, mentioning how this architecture makes it so you can easily embed entire rails applications into others since they’re all just derivatives of Railties.  While this holds true for very simple cases, you’ll quickly find simply creating a rails app, and sticking it in vendor/plugins doesn’t work the way you’d expect from reading about it.  You really do need to create your plugin in a certain way to actually make it work.  Over here is a good writeup on creating a Rails Engine based plugin installed as a gem.

Serving Static Assets

Now, serving static assets from your plugin is a bit on the unintuitive side, and likely for a good reason.  You generally do not want to do this for any kind of production system. However, like most rules, there are times when breaking them makes pragmatic sense.  In this use case we have an internal tool that we want to be able to easily extend.  The maximum number of concurrent users will likely be in low teens, and serving static assets through the rails app is a non-issue.  Extending the app includes providing views, controllers, models, routes, and images.  The first three are extremely straightforward and intuitive, routes use a slightly different enclosing syntax in an Engine, but are otherwise identical.  Images, however are a bit tricky.

There are two ways to create an Engine based plugin, and they are a bit incompatible with each other.  You can either install your plugin as a gem, or as a plugin in the app.  In both cases, you are going to use the ActionDispatch::Static middleware to serve your content. I am also assuming that you are placing your content in the “public” directory in the root of your plugin’s file structure.

Gem Based Plugin

A gem based plugin is initialized via the definition of itself in your plugin’s lib directory.  You can see an example of this below.  In order to set it up correctly you should put your middleware line in an initializer in your declaration:

module MyEngine
  class Engine &lt; Rails::Engine
    initializer "static assets" do |app|
      app.middleware.use ::ActionDispatch::Static, "#{root}/public"
    end
  end
end

In App Plugin

Unfortunately, if the above file exists in a plugin installed to vendor/plugins your app will fail to load.  This is because rails automatically assumes all plugins are Engines, and initializes them as instances of the Plugin class which inherits from Engine.  It will autoload everything in lib, see the declaration, and then fail out with the incredible error message “[Your Plugin] is a Railtie/Engine and cannot be installed as plugin”. Which is somewhat misleading since a plugin is an Engine, just not explicitly.

To get around this issue, you just have to move where you do your initialization and setup. Inside your plugin’s init.rb (which should be in the root of your plugin’s file structure) is where you can do this kind of stuff.  Rails boots your Plugin, and then loads this file with the variable “config” set for you to do your work.  So it’s simply:

config.middleware.use ::ActionDispatch::Static, "#{root}/public"

Load Order

I strongly recommend using middleware.use as it will place your middleware after the parent app, and therefore will cause your plugins assets to be of lower priority than everything else in the stack. This means that files in the parent app’s public directory will load instead of the plugin’s in the case of a conflicting name.  You still have options though.  To give your plugin’s static files priority over everything except the parent app’s static files, use:

middleware.insert_after ::ActionDispatch::Static, ::ActionDispatch::Static, "#{root}/public"

To give your plugin top priority in static asset serving use:

middleware.insert_before ::ActionDispatch::Static, ::ActionDispatch::Static, "#{root}/public"

Cucumber, Service Based Applications, Middleware, and You

Over at Primedia, we’ve been trying to get a little bit better at our BDD, and have resolved to really use Cucumber wherever possible. This is an easy thing to say for most web apps, however ours have a number of critical parts that are external to the application itself. This means that full integration testing of the applications that use these services as well as testing the services themselves can be rather tricky. In order for us to get started there were two major problems that we had to tackle.

The first was how to test the service based apps outside of any consuming application. The specific one in question did not present any HTML pages at all, and only returned Javascript that was executed. A few ideas were floated around like dev/test only controllers or basic one-off apps that were specifically for testing the service. Eventually we cam across a solution that worked and seemed much cleaner. We combined the one-off app idea with rack middleware to create an application within our app that was only loaded in certain environments. We used Sinatra to make a simple application that would respond with test pages on routes prefixed with ‘/test’, and then added it with config.middleware.use. Combining this with Celerity and Capybara gave us a very clean testing solution for an application that only served Javascript.

There is one gotcha to this, and it only involves using Sinatra as middleware in Rails. Rails attempts to determine if the middleware you gave it is a proc, and in doing so prevents you from being able to use any middleware that can respond to Rack requests as a singleton (like Sinatra). Our quick fix was to monkey patch Sinatra to remove self.call, but a more permanent solution is fixing the way Rails 2.3 does its middleware. I hope to submit some patches for that soon.

module Sinatra
  class Base
    class << self
      remove_method :call
    end
  end
end

Our second hurdle was determining how to test our consuming apps using Cucumber. The main problem we had is that we use reverse proxying to make the services appear to be coming from the same domains as the rest of the apps. This was easily taken care of in manual tests, where we could simulate our load balancers using Apache or nginx, however with automated tests, this becomes a problem. The automated test suite doesn't run behind a proxying layer or a web server, it just hits the app directly. So, tackling that problem, I am releasing a gem I developed called rack-reverse-proxy. What this gem does is allow you to put reverse proxy rules directly into your middleware stack so they are independent of your web server. This is probably not a good idea for production systems, load balancers and web servers are much better at this sort of thing, and due to the way rack works I have to wait until the entire request is received before forwarding it. However it's great for development and testing, allowing you to test your app directly without the need of an intermediate web server layer.

  require 'rack/reverse_proxy'

  use Rack::ReverseProxy do
    # Forward the path /test* to http://example.com/test*
    reverse_proxy '/test', 'http://example.com/'

    # Forward the path /foo/* to http://example.com/bar/*
    reverse_proxy /^\/foo(\/.*)$/, 'http://example.com/bar$1'
  end

Updating Multiple Git Repos

Adam Lowe has a great writeup on setting up Vim for Ruby/Rails work.  It’s a pretty great solution, using pathogen.vim and the git repos for the bundles, but he didn’t mention how to update your bundles in an easy manner…

$ find . -name ".git"  -type d -prune -execdir git pull \;

This is a little command line snippet that will search for git repositories in all of the subdirectories of wherever you run it and update them.  Really handy in this case, because you can update all of your bundles at once, but it will work in any case where you have cloned a bunch of repos and want them to all stay up to date.

Passenger can’t find Rails

If you are running in to an issue where you are trying to get a Rails app up and running, but are receiving the error “Missing the Rails 2.x.x gem…” when you attempt to access it despite Rails being installed and available to your system, do not panic.  Rails is just giving you an extremely unhelpful error message.

Rails catches *any* problem loading the Rails gem, and then responds with that error.

 rescue Gem::LoadError => load_error
      $stderr.puts %(Missing the Rails #{version} gem. Please `gem install -v=#{version} rails`, update your RAILS_GEM_VERSION setting in config/environment.rb for the Rails version you do have installed, or comment out RAILS_GEM_VERSION to use the latest version installed.)
      exit 1

If you inspect load_error from the line above, you should see the actual error message. In my case it was that ActiveSupport required Rack v1.0.0, but Passenger had already instantiated Rack v1.1.0. This led to a failure when loading the Rails gem, which was of course incorrectly reported. My quick solution was to uninstall the new version of rack, and force the install of the older version.

As a side note: I have the world’s worst code highlighter.

Getting Ruby Going on App Engine

I gave a talk last night at the Atlanta Ruby Users Group meetup about using JRuby on Google App Engine.  It was more of an overview about what App Engine is, and how to get started with a Sinatra app.  The Github repository for the application is located here, and you can see the application in action here.

The application demonstrates a few app engine concepts.  In models.rb you’ll see a sharded counter implementation using the direct Datastore API, and a traditional model using Datamapper as your ORM (ODM?) to Datastore.  The counter also demonstrates usage of the Memcache API.  Inside app.rb you can see some simple usage of the Users API, including how to check to see if a user is logged in, present a user with a login screen, and interacting with the object.  Lastly, you will find an example of how to determine your app’s name and version in config.ru.

Following the presentation I received a number of requests for my slides, so they are included below.  Slide 13 illustrates pushing your own app to App Engine.  As I mentioned in the talk, I strongly suggest setting aside one of your app ids for testing new ideas, since you only get 10 ids total. Slides removed due to discontinuation of the slide tool.