Upgrading Gems with git bisect

28th Jan 2013

Today we had to upgrade Devise on the Government's Enterprise Finance Guarantee management application due to a recent security announcement.

On running the usual bundle update to grab the latest version of Devise and then running the test suite, one of our request specs started failing.

F

Failures:

  1) User creation and first login 
     Failure/Error: page.should have_content('Invalid username or password')
       expected there to be content "Invalid username or password" in "Enterprise Finance Guarantee\n\n  \n    \n      \n        Enterprise Finance Guarantee\n\n      \n    \n  \n\n  \n\n      Invalid email or password.\n\n    \n  Sign In\n\n\n\n  Username\n  Password\n\n  \n\n  Forgot your password?\n    \n\n  \n\n"
     # ./spec/requests/user_creation_and_first_login_spec.rb:38:in `block (2 levels) in '

Finished in 1.09 seconds
1 example, 1 failure

Failed examples:

rspec ./spec/requests/user_creation_and_first_login_spec.rb:11 # User creation and first login

How could I narrow down what had changed to cause the failure? If this were my own code I'd use git bisect, but we can also use it for gem dependencies using bundler's path option. Here's how we solved the problem:

Checkout the Devise source code.

cd ~/Source
git clone https://github.com/plataformatec/devise.git

Configure your application's Gemfile to use the local checkout of the gem.

gem 'devise', path: '~/Source/devise'

Start git bisect. If you're unfamiliar with this tool this article explains it really well.

cd devise
git bisect start

Mark our previous version as good and the one we're upgrading to as bad.

git bisect good v2.1.0
git bisect bad v2.2.3

From the application directory run the bundle and your test. Then from the devise directory mark it as either good or bad depending on whether the test passed or failed.

bundle && bundle exec rspec spec/requests/user_creation_and_first_login_spec.rb:11

Eventually, after a number of test runs, we get the result: This commit:

cde2229e598d0b91e75c039ead4311fa6a89a2e9 is the first bad commit
commit cde2229e598d0b91e75c039ead4311fa6a89a2e9
Author: Gabe Martin-Dempesy 
Date:   Sun Nov 18 15:08:57 2012 -0800

    Support alternate sign in error message when email record does not exist

    By default, the nonexistent error is still identical to the :invalid
    message, and must be customized by the developer to implement.

:040000 040000 f3d61b1f4def429b532b251601b0c6b00b7f3b0b 20f98c3ef29dde4cbe03ba7389a5fbcf34a19642 M	config
:040000 040000 5c2448231976fbe98589192ce23b9c8afeb7be27 2db6b1091741eddf4d3b30158c0d689acb17a5de M	lib

Once we knew the cause the fix was simple. Voila, passing tests.

.

Finished in 1.43 seconds
1 example, 0 failures

A more thorough read of Devise's well documented CHANGELOG might have got the answer quicker, but this approach made the computer do the work!