I have discovered, that composite_primary_keys breaks my SQL queries when :primary_key option specified both for has_many and has_one associations.

Step 0. Reproducing the bug

First of all we need to reproduce a bug. Please note: if you know where the problem is, you can skip this step. Let’s start from a simple example:

Now we will add dependencies to the config/environment.rb:

and to the config/environments/test.rb:

Okay. Now we are ready to start. Let’s generate some migrations:

Let’s create our factories:

And our spec which should fail:

Ok, it fails because we haven’t defined our associations on models. Let’s do it:

Woohoo! Specs are failing with an error we’re trying to reproduce:

Step 1. Setting up an environment for composite_primary_keys gem

In general to submit a patch to the Open Source project hosted by GitHub you have to perform the following steps: fork the repository on GitHub, write tests which fail, write a patch, ensure it works, push your changes to your fork repository, and submit a pull request. Let’s do just that!

To fork the repository I will use a perfect github gem by Dr Nic (BTW, he is the author of composite_primary_keys!)

Now let’s configure our test environment and run tests:

You should see something like this:

If you have any problems, check the test/README_tests.txt file for help.

Step 2. Reproducing failing tests inside composite_primary_keys test suite

We are doing TDD, right? So before any fixes we have to write a failing test first. Gem we’re hacking has a powerful test suite with many database tables created, so all we need is just to add associations to one of models, which will cover our issue.

First, add this to the test/fixtures/membership.rb model:

And this tests set goes to the test/test_associations.rb:

Now rake test_mysql produces following error (there are 4 of them, I will show only the first one):

Well, that are the errors we are working on. Time to fix them!

Step 3. Fixing the bug

I will not explain how I fixed that, you can check my commit for details. Here is the diff:

Run tests to get the following output:

We are done for now!

Step 4. Committing changes and pulling request

It’s time to commit our changes now:

Okay, the next step is a pull request. Last command opened a browser window with your fork. Navigate to the latest commit and press the “Pull Request” button (at the time of writing this article gh pull-request didn’t worked, and you can try to fix by yourself to understand the workflow):

That’s all, we have contributed to the community! Today Darrin Holst merged my commit into the core, and you can find it here. Not all things went smooth (I forgot to add a fixture, so tests were failing on first run), but he helped me a lot to get it working. That’s how Open Source works: we help each other to develop high quality software.

Credits

First of all, thanks to Dr Nic for the great plugin, one of the best piece of functionality I can’t imagine life without. Thanks to Darrin Holst for his patience and great help in debugging tests problem, and also for merging my commit into the composite_primary_keys core. Thanks to GitHub for the great Open Source code hosting solution, which makes working on Open Source projects so exciting.

Do you have comments or suggestions? You are welcome! Also, I will be happy if you follow me in Twitter.

1. Use before :all block carefully

Sometimes it looks like a good idea to create a test data in before :all block. But be careful — these blocks are not wrapped in a transaction, so the data will not be rolled back after the test. In this case you should clear your data in the after :all block manually.

Another option is to move your before :all blocks to before :each to make them rolled back automatically.

2. For each test create exactly what it needs

Fixtures are cool when you start working on a project. But they quickly become painful while project grows: you add a new field to a fixture and break a half of your tests. There are tons of plugins which could simplify test data creation, I personally recommend factory_girl: it’s pretty slick and easy to use.

3. Do not create hundreds of records for a particular spec

Sometimes you want to test a method which operates on large set of records (filtering, trimming, etc). For example, this method returns 50 most popular videos (and no more). The straight approach is to create 51 record and make sure, that the size of the returned array is 50. When I saw a code snippet like this in our project first time, I was surprised. There was a few more pieces sharing this behavior, so here is my advice: add a parameter to the method, which will limit the number of records to return. In this case you can create 3 records, and pass 2 as a parameter.

4. Do not over-mock

Mocking is interesting and sometimes very useful technology. You may mock just everything so you spec will not hit the database. But there is a catch: your model code may be changed some day causing callers to break. Since you mock everything, you will never get failing specs. So now you should update all your mocks to fit a new interface. Also you would not be able to find SQL queries errors if you have mocked them. Instead of this I use integration approach: controller should talk to models, which have to hit the database. Real database with real data (OK, not so real). The practice 2 can help you in test data creation.

Bad:

Good:

But you can use mocks to skip records retrieving from the database (make sure you have specs covering corresponding model code). Let me explain this. For example, you need to render 20 entries in an RSS feed. You could create 21 record in the database using a factory, and then ensure only 20 of them were retrieved, or you could mock your finder method and check its parameter. You may not like magic numbers like 20 in this particular case, and this is a good point. Just move this magic number to the config and ensure it was used to do the retrieval.

5. Use contexts

RSpec spec specifies how particular code should work. Usually, in the beginning you tell what you are going to describe in this spec, and inside describe block you specify what the code should do:

Usually you have more than one it block for each method. To group related specs I recommend to use nested describe blocks. Since describe is aliased to context when placed inside another describe, I think it’s a good idea to use it for specs grouping. Each context may have its own before and after blocks (in this case parent blocks will be called right before child ones).

6. Create several test suites to speed up your workflow

There are many things you can do to make your BDD more efficient. We will take a look at two of them: creating a separate test suites and running recently modified specs.

There are several standard test suites configured in RSpec by default: spec:controllers, spec:views, spec:helpers, spec:lib. Check the rake -T spec output to get a list of available RSpec tasks. Let’s create a simple Rake tasks generator for spec suites:

Check what tasks are available now:

It was easy! And now let’s take a look at the Rake task for running the recently touched specs (last 10 minutes).

And don’t forget to check autospec and watchr gems.

7. Stop spec_helper from being loaded multiple times

Just don’t do that. If you got a big project, there is a chance that the spec_helper will be required in a many different ways: File.expand_path, File.join, etc.,— which results in it being loaded several times and it slows down your test suite!

To avoid this, add the following code at the top of your spec_helper.rb:

It will show where you’ve tried to load spec_helper from so you will be able to fix it immediately. Also there is an interesting snippet of code here, which will find and replace all wrong includes.

Conclusion

RSpec is not a silver bullet. You can have 100% coverage and fine-grained specs, but it does not mean your application is completely bug-free. Refactor your specs, increase your programming level, and refactor again. Write specs for any issue that you, your QAs or users have faced. And remember: do not over-mock.

Credits

This is my first article written completely in Google Wave in collaboration with several good Russian rubyists. Thank you all, guys. I want to acknowledge the editorial help of Roman Dmytrenko and Alexey Kovyrin. Robby Russell created a great picture illustrating how sexy is RSpec. And thank you all my readers for your attention.

Did you like this article? You should follow me in Twitter here.

Here is the list of questions we will talk about:

1. You have some logic in your view, which uses your models extensively. There are no places in other views with such logic. The classic recommendation is to move this code into a model, but after a short time your models become bloated with stupid one-off helper methods. The solution: pattern Presenter.

2. Your constructor contains a lot of code to retrieve some values for your views from the database or another storage. You have a lot of fragment_exist? calls to ensure no of your data is loaded when corresponding fragment is already in cache. It’s really hard to test a particular action because of it’s size. The solution: pattern Presenter.

3. You have a partial, used everywhere on the site. It accepts a lot of parameters to configure how rendered code should look like. The header of this partial, which initializes default values of parameters, becomes larger and larger. The solution: cells plugin.

Please note: sample application is available on GitHub.

Presenter Pattern

Okay, you have an idea when to use this patterns. Let’s look at the example:

And the view:

Note: this code is available in the first commit of my presenter example project.

Scary code, isn’t it? So let’s refactor it using Presenter pattern. I prefer to put presenters into a separate folder app/presenters, so first we should add it to Rails load path. Add this line to your config/environment.rb:

Now we are ready to write our presenter (app/presenters/home_presenters/show_presenter.rb):

Sometimes presenters depend on parameters, so feel free to add an initialize method. It could accept particular params or whole params collection:

Now let’s refactor our controller:

Whoa, that’s nice! View now is little different:

Presenters testing is much easier than testing of bloated controllers:

Please note: this code is available in the second commit of my presenter example project.

Please note: you should not do any manipulations on models in presenters. They only decorate models with helper methods to be used inside controllers or views, nothing else. There are several articles describing a Conductor pattern as a presenter, do not repeat their mistakes. See the first link in the list below to get an idea about the differences.

Related links:

• Presenters & Conductors on Rails
• Presenters in Ruby on Rails Applications
• Rails Model View Controller + Presenter?
• RailsConf Europe 07: Presenter Links

Cells Plugin

Okay, now we have a clean controller. But what about views? Let’s take a look at the videos partial:

So, what the heck? Is this a view or a controller? Remember old PHP days, with all this spaghetti code? That is it. It’s hard to test, it looks scary, it bad. So here cells plugin comes to the stage.

First, we need to install the plugin:

Now let’s generate a cell:

And write some code (app/cells/video.rb):

app/cells/video/videos_section.html.erb:

app/cells/video/videos_list.html.erb:

app/cells/video/videos_div.html.erb:

app/cells/video/_video.html.erb:

And the view:

Wow! That’s pretty easy to read and modify. All the logic is in the code now, all the views are easy to read, and moreover: it’s more than easy to test now! I have a little plugin called rspec-cells, and I have committed a patch yesterday to get it working with the latest RSpec. Here is how you spec could look like:

So it looks almost like a classic Ruby on Rails controller spec. I hope to review the code in nearest feature and will send a pull request to the cells plugin author. Of course, if you
found a bug, feel free to contact me.

Please note: this code is available in the third commit of my presenter example project.

Related links:

• Cells project home page
• Cells project on GitHub
• rspec-cells project on GitHub

That’s all I wanted to show you today. I think a Presenter Example project will be updated periodically, so follow it on GitHub, follow me in Twitter or on GitHub to get instant updates. Also take a look at the rspec-cells plugin, maybe you will have some time to make it better.

First of all, we need to decide what features we need from this tool. Don’t know about you, but all I need is to measure execution time of particular ruby code block. Here is what I mean:

So what do we see from this output? There are two slow blocks: top videos retrieving and tag cloud rendering. Now we just know what to do to make this page faster.

Let’s write the code:

We have defined two class attributes: EasyProfiler::Profile.enable_profiling (to be able to disable or enable profiler globally) and EasyProfiler::Profile.print_limit (to filter out from log code blocks that are fast enough).

Then we defined two methods, which accept name of profile session (for example, “home#index”), and hash of options. Possible options are :enabled (to enable profiling of particular block) and :limit (limit in seconds to filter out fast code fragments).

Method start returns an instance of profiler, which will be used to print check points. It contains two useful methods: debug (to display custom message) and progress (to display a message along with time spent since last checkpoint). Both methods define a new checkpoint.

To simplify usage, let’s create a helper:

And now example:

and view:

As you can see from this example, profiler will be enabled only when you pass a _with_profiling parameter with value yes: http://example.com/home?_with_profiling=yes.

That’s all. If you have any question, feel free to post a comment or contact me.

Update: I have created a Rails plugin called easy-prof, which is hosted on GitHub. It’s more powerful and feature complete, so feel free to grab sources and play with it by yourself (check the RDoc documentation at rdoc.info). Do not forget to drop me a line about your feelings.

In Ruby on Rails we are using routes to generate any sort of links. So let’s include routing mechanism into our own script or class.

First, you need to ensure that Rails core is loaded (if you haven’t done this earlier; for example, in script/console you should not do this). I’m assuming you’re creating a script under /script folder:

Now you need to include ActionController::UrlWriter module, which allows to write URLs from arbitrary places in your codebase, and configure default_url_options[:host]:

Easy and helpful technique. Enjoy!

Check the first command result. Very disappointing, right? So what happen? Quick looking through the Ruby code completely explained it. Here is how this method works internally (this is just prototype in Ruby, original code is in C, but works in the same way):

Let’s test it:

And just to make sure our conclusions are correct:

So the idea behind the Hash.uniq method is the method Hash.hash, which produces different results for hashes in the first example. Be careful when doing obj.uniq on complex objects.

Update: There is a good research on Hash.hash method here.

First thing I want to mention is gems documentation. Every gem in system has powerful documentation based on rdoc, built during an installation procedure. The easiest way to view these docs is to run gem server and open http://localhost:8808/ in browser:

You will be able to review which gems has been installed, and check their documentation.

I’m pretty sure you have lots of gems with the same name, but different versions. You could use the following command to remove all gem versions except the latest one:

That’s all. You have only latest gems installed. But wait, what about Max OS X users, who have pre-installed Ruby? They (actually we) are in trouble, because…

Fixing problem with gem cleanup on Mac OS X 10.5.x (Leopard)

On Mac OS X when you will try to use this command, you will hit following error:

That’s common problem and I will show how to fix it. First, run following command:

You may now uninstall the offending gem with:

So, in general

will get you the location of the gem, and

will uninstall the gem (found here.)

Take a look at the following example.

Here we have an array of fruits and we need to calculate a frequency of each fruit. As you can see, in the line 3 we are initializing frequency value to 0 if there are was no fruit with this name before. We can simplify this code:

In line 2 we are creating a new hash, which default value is 0. This means that if we would try to retrieve value for a non-existing key, 0 would be returned.

Let’s check another example:

Here we are collecting indexes of each fruit in the source array. But now we can’t just create a new hash and pass [] as the default value, because all keys in this hash will refer to the same array, so in result we will get an array [1, 2, 3] for each fruit. So let’s try the following:

In this case we are creating a new array object for any non-existing key, that was accessed. So

will return [] and save it in the hash. When you will hit this key next time, previously created array will be returned.

You can pass a block to Array constructor too. For example, you need an array with 10 random numbers:

You can simplify it using map method:

But you can do it even easier:

Next Memo will cover managing Ruby Gems, so stay tuned.

And then I’m googling on how to install these gems. It’s time simplify my life and post commands here.

Installing the ruby mysql gem on Mac OS X and MacPorts

Installing mysql5 from MacPorts:

Now we can install mysql gem:

Installing the ruby memcached gem on Mac OS X and MacPorts

First you need to install memcached and libmemcached:

And then memcached gem:

If you have any questions that could be covered in this series — ask me in comments.

Table of contents

• Multi-query support
• Extended engine V2
• 64-bit document and word IDs support
• Multiple-valued attributes
• Geodistance feature
• Download

Multi-query support

What does it mean? Multi-query support means sending multiple search queries to Sphinx at once. It’s saving network connection overheads and other round-trip costs. But what’s much more important, it unlocks possibilities to optimize “related” queries internally. Here is quote from the Sphinx home page:

One typical Sphinx usage pattern is to return several different “views” on the search results. For instance, one might need to display per-category match counts along with product search results, or maybe a graph of matches over time. Yes, that could be easily done earlier using the grouping features. However, one had to run the same query multiple times, but with different settings.

From now on, if you submit such queries through newly added multi-query interface (as a side note, ye good olde Query() interface is not going anywhere, and compatibility with older clients should also be in place), Sphinx notices that the full-text search query is the same and it is just sorting/grouping settings which are different. In this case it only performs expensive full-text search once, but builds several different (differently sorted and/or grouped) result sets from retrieved matches. I’ve seen speedups of 1.5-2 times on my simple synthetic queries; depending on different factors, the speedup could be even greater in practice.

To perform multi-query you should add several queries using AddQuery method (parameters are exactly the same as in Query call), and then call RunQueries. Please note, that all parameters, filters, query settings are stored between AddQuery calls. It means that if you have specified sort mode using SetSortMode before first AddQuery call, then sort mode will be the same for the second AddQuery call. Currently you can reset only filters (using ResetFilters) and group by (ResetGroupBy) settings. BTW, you can use Query as usually to perform single query, but don’t try to make this call after you have added query into the batch using AddQuery.

Stop speaking, let’s look the example:

As the result we will get array of 2 hashes:

Each hash contains the same data as result of Query method call. Also they have additional fields error and warning which contains error and warning message respectively when not empty.

Note: I have added ResetFilters call before creating second query. Without this call our query will have two filters with conflicting conditions, so there will be no results at all.

Extended engine V2

New querying engine (codenamed “extended engine V2”) is going to gradually replace all the currently existing matching modes. At the moment, it is fully identical to extended mode in functionality, but is much less CPU intensive for some queries. Here are notes from Sphinx author:

I have already seen improvements of up to 3-5 times in extreme cases. The only currently known case when it’s slower is processing complex extended queries with tens to thousands keywords; but forthcoming optimizations will fix that.

V2 engine is currently in alpha state and does not affect any other matching mode yet. Temporary SPH_MATCH_EXTENDED2 mode was added to provide a way to test it easily. We are in the middle of extensive internal testing process (under simulated production load, and then actual production load) right now. Your independent testing results would be appreciated, too!

So, to use new matching mode we should use SPH_MATCH_EXTENDED2 mode. Let’s do it!

Easy enough, right? You should try it by yourself to feel power of new engine. Please note, that this mode is temporary and it will be removed after release.

64-bit document and word IDs support

Before version 0.9.8 the Sphinx was limited to index up to 4 billion documents because of using 32-bit keys. From here on it has ability to use 64-bit IDs, and new feature does not impact on 32-bit keys performance. Let’s look at the example. First we will make query to DB with 32-bit keys:

As you can see, class of the id field is Fixnum. Let’s try to make call to index with 64-bit keys. You will get Bignum as the result, and it means that you can have more than 4 billion documents!

Multiple-valued attributes

Plain attributes only allow to attach 1 value per each document. However, there are cases (such as tags or categories) when it is necessary to attach multiple values of the same attribute and be able to apply filtering to value lists. In these cases we can use multiple-valued attributes now.

In case of using miltiple-valued attribute tag you will get result like:

As you can see, multiple-valued attributes returned as array of integers.

Geodistance feature

Sphinx now is able to compute geographical distance between two points specified by latitude and longitude pairs (in radians). So you now can specify per-query “anchor point” (and attribute names to fetch per-entry latitude and longitude from), and then use “@geodist” virtual attribute both in the filters and in the sorting clause. In this case distance (in meters) from anchor point to each match will be computed, used for filtering and/or sorting, and returned as a virtual attribute too.

Download

As always, you can download Sphinx Client API from project home page. Take into account that version 0.3.1 of the client API intended to use with Sphinx 0.9.7, and Sphinx Client API 0.4.0 r909 requires Sphinx 0.9.8 r909 development snapshot. You could download Sphinx from the Download section of the Sphinx home page.