opensource | Dmytro Shteflyuk's Home

Submitting a patch to the Open Source project: composite_primary_keys

Dmytro Shteflyuk — Thu, 17 Dec 2009 10:28:12 +0000

Not so far ago I have found a weird bug in the Open Source Ruby gem called composite_primary_keys, occurred when you specify :primary_key option for has_one or has_many association. There are two ways to get it fixed: submit an issue and wait till someone will work out this problem or fix it by yourself and then pull request to get the patch merged into the core. This is a great library and I use it in almost all my project, so I decided to help the author and fix this bug by myself. Here I will show you how to do that.

I have discovered, that composite_primary_keys breaks my SQL queries when :primary_key option specified both for has_many and has_one associations.

Step 0. Reproducing the bug

First of all we need to reproduce a bug. Please note: if you know where the problem is, you can skip this step. Let’s start from a simple example:

1	rails cpk_bug && cd cpk_bug

Now we will add dependencies to the config/environment.rb:

1	config.gem 'composite_primary_keys'

and to the config/environments/test.rb:

1
2
3

config.gem 'rspec', :lib => 'spec'
config.gem 'rspec-rails', :lib => false
config.gem 'factory_girl'

Okay. Now we are ready to start. Let’s generate some migrations:

1
2
3
4
5

script/generate rspec_model document upload_request_id:integer title:string description:text
script/generate rspec_model upload_request filename:string state:integer
script/generate rspec_model copyright_request upload_request_id:integer explanation:text
rm -rf spec/fixtures
rake db:migrate && rake db:test:clone

Let’s create our factories:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

Factory.define :copyright_request do |cr|
cr.explanation "This document is copyrighted by O'Reilly"
end

Factory.define :document do |d|
d.sequence(:title) { |n| "Document #{n}" }
d.description { |a| "The perfect description for the document '#{a.title}'" }
d.association :upload_request
end

Factory.define :upload_request do |ur|
ur.sequence(:filename) { |n| "file#{'%03d' % n}.pdf" }
ur.state 0
end

And our spec which should fail:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

require 'spec/spec_helper'

describe Document do
context 'when has copyright requests' do
before :each do
# Fake upload request used to desynchronise document and
# upload request IDs
Factory(:upload_request)
end

it 'should not have any copyright requests when just created' do
@document = Factory(:document)
@document.copyright_requests.should == []
end

it 'should return a list of copyright requests from #copyright_requests' do
@document = Factory(:document)
@request1 = Factory(:copyright_request, :upload_request => @document.upload_request)

@document.copyright_requests.should == [@request1]
end
end
end

Ok, it fails because we haven’t defined our associations on models. Let’s do it:

1
2
3
4
5
6
7
8
9
10
11
12

class Document < ActiveRecord::Base
belongs_to :upload_request

has_many :copyright_requests, :primary_key => :upload_request_id, :foreign_key => :upload_request_id
end

class UploadRequest < ActiveRecord::Base
end

class CopyrightRequest < ActiveRecord::Base
belongs_to :upload_request
end

Woohoo! Specs are failing with an error we’re trying to reproduce:

1
2
3
4
5
6
7
8
9
10
11
12

~/cpk_bug$ spec spec
..F.

1)
'Document when has copyright requests should return a list of copyright requests from #copyright_requests' FAILED
expected: [#],
got: [] (using ==)
/Users/kpumuk/cpk_bug/spec/models/document_spec.rb:20:

Finished in 0.155708 seconds

4 examples, 1 failure

Step 1. Setting up an environment for composite_primary_keys gem

In general to submit a patch to the Open Source project hosted by GitHub you have to perform the following steps: fork the repository on GitHub, write tests which fail, write a patch, ensure it works, push your changes to your fork repository, and submit a pull request. Let’s do just that!

To fork the repository I will use a perfect github gem by Dr Nic (BTW, he is the author of composite_primary_keys!)

1
2
3
4

sudo gem install github
gh clone drnic/composite_primary_keys
cd composite_primary_keys
gh fork

Now let’s configure our test environment and run tests:

1
2
3

rake local:setup
rake mysql:build_databases
rake test_mysql

You should see something like this:

1
2
3
4
5
6

Using native MySQL
Started
...................................................................................
Finished in 0.487679 seconds.

83 tests, 262 assertions, 0 failures, 0 errors

If you have any problems, check the test/README_tests.txt file for help.

Step 2. Reproducing failing tests inside composite_primary_keys test suite

We are doing TDD, right? So before any fixes we have to write a failing test first. Gem we’re hacking has a powerful test suite with many database tables created, so all we need is just to add associations to one of models, which will cover our issue.

First, add this to the test/fixtures/membership.rb model:

1 2	has_many :readings, :primary_key => :user_id, :foreign_key => :user_id has_one :reading, :primary_key => :user_id, :foreign_key => :user_id, :order => 'id DESC'

And this tests set goes to the test/test_associations.rb:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

def test_has_many_with_primary_key
@membership = Membership.find([1, 1])

assert_equal 2, @membership.readings.size
end

def test_has_one_with_primary_key
@membership = Membership.find([1, 1])

assert_equal 2, @membership.reading.id
end

def test_joins_has_many_with_primary_key
@membership = Membership.find(:first, :joins => :readings, :conditions => { :readings => { :id => 1 } })

assert_equal [1, 1], @membership.id
end

def test_joins_has_one_with_primary_key
@membership = Membership.find(:first, :joins => :reading, :conditions => { :readings => { :id => 2 } })

assert_equal [1, 1], @membership.id
end

Now rake test_mysql produces following error (there are 4 of them, I will show only the first one):

1
2
3
4

1) Error:
test_has_many_with_primary_key(TestAssociations):
ActiveRecord::StatementInvalid: Mysql::Error: Operand should contain 1 column(s): SELECT * FROM `readings` WHERE (`readings`.`user_id` = 1,1)
...

Well, that are the errors we are working on. Time to fix them!

Step 3. Fixing the bug

I will not explain how I fixed that, you can check my commit for details. Here is the diff:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

diff --git a/lib/composite_primary_keys/associations.rb b/lib/composite_primary_keys/associations.rb
index 6b63664..9a9e173 100644
--- a/lib/composite_primary_keys/associations.rb
+++ b/lib/composite_primary_keys/associations.rb
@@ -180,11 +180,12 @@ module ActiveRecord::Associations::ClassMethods
raise AssociationNotSupported, "Polymorphic joins not supported for composite keys"
else
foreign_key = options[:foreign_key] || reflection.active_record.name.foreign_key
+ primary_key = options[:primary_key] || parent.primary_key
" LEFT OUTER JOIN %s ON %s " % [
table_name_and_alias,
composite_join_clause(
full_keys(aliased_table_name, foreign_key),
- full_keys(parent.aliased_table_name, parent.primary_key)),
+ full_keys(parent.aliased_table_name, primary_key)),
]
end
when :belongs_to
@@ -338,7 +339,7 @@ module ActiveRecord::Associations
@finder_sql << " AND (#{conditions})" if conditions

else
- @finder_sql = full_columns_equals(@reflection.klass.table_name, @reflection.primary_key_name, @owner.quoted_id)
+ @finder_sql = full_columns_equals(@reflection.klass.table_name, @reflection.primary_key_name, owner_quoted_id)
@finder_sql << " AND (#{conditions})" if conditions
end

@@ -386,7 +387,7 @@ module ActiveRecord::Associations
"#{@reflection.klass.quoted_table_name}.#{@reflection.options[:as]}_id = #{@owner.quoted_id} AND " +
"#{@reflection.klass.quoted_table_name}.#{@reflection.options[:as]}_type = #{@owner.class.quote_value(@owner.class.base_class.name.to_s)}"
else
- @finder_sql = full_columns_equals(@reflection.klass.table_name, @reflection.primary_key_name, @owner.quoted_id)
+ @finder_sql = full_columns_equals(@reflection.klass.table_name, @reflection.primary_key_name, owner_quoted_id)
end

@finder_sql << " AND (#{conditions})" if conditions

Run tests to get the following output:

1
2
3
4
5
6
7

~/cpk_bug/composite_primary_keys (master)$ rake test_mysql
Using native MySQL
Started
.......................................................................................
Finished in 0.511129 seconds.

87 tests, 266 assertions, 0 failures, 0 errors

We are done for now!

Step 4. Committing changes and pulling request

It’s time to commit our changes now:

1
2
3
4
5
6
7
8
9
10
11
12
13

~/cpk_bug/composite_primary_keys (master)$ git add .
~/cpk_bug/composite_primary_keys (master)$ git commit -m 'Fixed several bugs in has_one and has_many associations when :primary_key specified'
[master 3e29891] Fixed several bugs in has_one and has_many associations when :primary_key specified
3 files changed, 31 insertions(+), 3 deletions(-)
~/cpk_bug/composite_primary_keys (master)$ git push
Counting objects: 16, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (9/9), 2.14 KiB, done.
Total 9 (delta 7), reused 0 (delta 0)
To git@github.com:kpumuk/composite_primary_keys.git
050d832..3e29891 HEAD -> master
~/cpk_bug/composite_primary_keys (master)$ gh home

Okay, the next step is a pull request. Last command opened a browser window with your fork. Navigate to the latest commit and press the “Pull Request” button (at the time of writing this article gh pull-request didn’t worked, and you can try to fix by yourself to understand the workflow):

That’s all, we have contributed to the community! Today Darrin Holst merged my commit into the core, and you can find it here. Not all things went smooth (I forgot to add a fixture, so tests were failing on first run), but he helped me a lot to get it working. That’s how Open Source works: we help each other to develop high quality software.

Credits

First of all, thanks to Dr Nic for the great plugin, one of the best piece of functionality I can’t imagine life without. Thanks to Darrin Holst for his patience and great help in debugging tests problem, and also for merging my commit into the composite_primary_keys core. Thanks to GitHub for the great Open Source code hosting solution, which makes working on Open Source projects so exciting.

Do you have comments or suggestions? You are welcome! Also, I will be happy if you follow me in Twitter.

The post Submitting a patch to the Open Source project: composite_primary_keys first appeared on Dmytro Shteflyuk's Home.

Scribd open source projects

Dmytro Shteflyuk — Tue, 08 Sep 2009 02:06:05 +0000

It’s time to summarize what we have done for the Open Source community. Scribd is pretty open company, we release a lot of code into the public after a time (sometimes it is short, sometimes it is not). Here I want to mention all the code we have opensourced. Please take into account that time is moving on, so we are publishing more and more code. I will update this post periodically, so stay tuned. Follow me on Twitter to get instant updates.

Here is the list of our projects in alphabetical order:

bounces-handler — Email Bounces Processing System with Rails plugin to prevent Rails mailers from sending any messages to a blocked addresses.
db-charmer — ActiveRecord Connections Magic (slaves, multiple connections, etc).
easy-prof — Simple and easy to use Ruby code profiler, which could be used as a Rails plugin.
Fast Sessions — Sessions class for ActiveRecord sessions store created to work fast (really fast).
loops — Simple background loops framework for Ruby on Rails and Merb.
magic-enum — Method used to define ENUM-like attributes in your model (int fields actually).
rlibsphinxclient — A Ruby wrapper for pure C searchd client API library.
rscribd — Ruby client library for the Scribd API.
Rspec Cells — A library for testing applications that are using Cells in RSpec.
Scribd Desktop Uploader — A fully native Cocoa Macintosh uploader app for the Scribd.com website.

bounces-handler

Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP’s feedback loops emails, maintain your mailing blacklists and a Ruby on Rails plugin to use those blacklists in your RoR applications.

This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.

Links: Project Home Page on GitHub | Introduction Blog Post | RDoc Documentation.

db-charmer

DbCharmer is a simple yet powerful plugin for ActiveRecord that does a few things:

Allows you to easily manage AR models’ connections (switch_connection_to method)
Allows you to switch AR models’ default connections to a separate servers/databases
Allows you to easily choose where your query should go (Model.on_db methods)
Allows you to automatically send read queries to your slaves while masters would handle all the updates.
Adds multiple databases migrations to ActiveRecord

It requires Ruby on Rails version 2.3 or later. The main purpose of this plugin is to put all the databases-related code we have been using in Scribd for a while into a single easy-to use package.

Links: Project Home Page on GitHub | Test Rails Application on GitHub | RDoc Documentation.

easy-prof

Simple and easy to use Ruby code profiler, which could be used as a Rails plugin. The main idea behind the easy-prof is creating check points and your code and measuring time needed to execute code blocks. Here is the example of easy-prof output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

[home#index] Benchmark results:
[home#index] debug: Logged in user home page
[home#index] progress: 0.7002 s [find top videos]
[home#index] progress: 0.0452 s [build categories list]
[home#index] progress: 0.0019 s [build tag cloud]
[home#index] progress: 0.0032 s [find featured videos]
[home#index] progress: 0.0324 s [find latest videos]
[home#index] debug: VIEW STARTED
[home#index] progress: 0.0649 s [top videos render]
[home#index] progress: 0.0014 s [categories render]
[home#index] progress: 2.5887 s [tag cloud render]
[home#index] progress: 0.0488 s [latest videos render]
[home#index] progress: 0.1053 s [featured video render]
[home#index] results: 3.592 s

From this output you can see what checkpoints takes longer to reach, and what code fragments are pretty fast.

Links: Project Home Page on GitHub | Introduction Blog Post | RDoc Documentation.

Fast Sessions

FastSessions is a sessions class for ActiveRecord sessions store created to work fast (really fast). It uses some techniques which are not so widely known in developers’ community and only when they cause huge problems, performance consultants are trying to help with them.

FastSessions plugin was born as a hack created for Scribd.com (large RoR-based web project), which was suffering from InnoDB auto-increment table-level locks on sessions table.

So, first of all, we removed id field from the table. Next step was to make lookups faster and we’ve used a following technique: instead of using (session_id) as a lookup key, we started using (CRC32(session_id), session_id) — two-columns key which really helps MySQL to find sessions faster because key cardinality is higher (so, mysql is able to find a record earlier w/o checking a lots of index rows). We’ve benchmarked this approach and it shows 10–15% performance gain on large sessions tables.

And last, but most powerful change we’ve tried to make was to not create database records for empty sessions and to not save sessions data back to database if this data has not been changed during current request processing. With this change we basically reduce inserts number by 50-90% (depends 0n application).

All of these changes were implemented and you can use them automatically after a simple plugin installation.

There is a fork patched by mudge for full compatibility with Ruby on Rails version 2.3 or later.

Links: Project Home Page on Google Code | Introduction Blog Post | Fork on GitHub Compatible with Rails 2.3 and Later.

loops

loops is a small and lightweight library for Ruby on Rails, Merb and other frameworks created to support simple background loops in your application which are usually used to do some background data processing on your servers (queue workers, batch tasks processors, etc).

Originally loops plugin was created to make our own loops code more organized. We used to have tens of different modules with methods that were called with script/runner and then used with nohup and other not so convenient backgrounding techniques. When you have such a number of loops/workers to run in background it becomes a nightmare to manage them on a regular basis (restarts, code upgrades, status/health checking, etc).

After a short time of writing our loops in more organized ways we were able to generalize most of the loops code so now our loops look like a classes with a single mandatory public method called run. Everything else (spawning many workers, managing them, logging, backgrounding, pid-files management, etc) is handled by the plugin itself.

Links: Project Home Page on GitHub | Introduction Blog Post | RDoc Documentation.

magic-enum

Method used to define ENUM-like attributes in your model (int fields actually). It’s easier to show what it does in code rather than to explain in plain English:

1
2
3
4
5
6
7

Statuses = {
:unknown => 0,
:draft => 1,
:published => 2,
:approved => 3
}
define_enum :status, :default => 1, :raise_on_invalid => true, :simple_accessors => true

is identical to

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

Statuses = {
:unknown => 0,
:draft => 1,
:published => 2,
:approved => 3
}
StatusesInverted = Statuses.invert

def status
StatusesInverted[self[:status].to_i] || StatusesInverted[1]
end

def status=(value)
raise ArgumentError, "Invalid value "#{value}" for :status attribute of the #{self.class} model" if
Statuses[value].nil?
self[:status] = Statuses[value]
end

def unknown?
status == :unknown
end

def draft?
status == :draft
end

def published?
status == :published
end

def approved?
status == :approved
end

This plugin was originally developed for Best Tech Videosand later was cleaned up in Scribd repository and released to the public.

Links: Project Home Page on GitHub | RDoc Documentation.

rlibsphinxclient

A Ruby wrapper for pure C searchd client API library. It works much faster than any Ruby client for Sphinx, so you can check it to ensure you application works as fast as possible.

Please note: this is *highly experimental* library so use it at your own risk.

Links: Project Home Page on GitHub | RDoc Documentation.

rscribd

Ruby client library for the Scribd API. This gem provides a simple and powerful library for the Scribd API, allowing you to write Ruby applications or Ruby on Rails websites that upload, convert, display, search, and control documents in many formats. For more information on the Scribd platform, visit the Scribd Platform Documentation page.

The main features are:

Upload your documents to Scribd’s servers and access them using the gem
Upload local files or from remote web sites
Search, tag, and organize documents
Associate documents with your users’ accounts

Links: Project Home Page on GitHub | Scribd Platform Documentation | RDoc Documentation.

Rspec Cells

This plugin allows you to test your cells easily using RSpec. Basically, it adds an example group especially for cells, with several helpers to perform cells rendering.

If you are not sure what is cells, please visit its home page.

Spec for a regular cell could look like:

1
2
3
4
5
6
7
8
9
10
11
12
13

describe VideoCell do
integrate_views

context '.videos' do
it 'should initialize :videos variable' do
params[:id] = 10
session[:user_id] = 20
opts[:opt] = 'value'
result = render_cell :videos, { :videos => [] }, :slug => 'hello'
result.should have_tag('div', :class => :videos)
end
end
end

Links: Project Home Page on GitHub | Cells Home Page | Cells Home Page on GitHub.

Scribd Desktop Uploader

A fully native Cocoa Macintosh uploader app for the Scribd.com website. Supports following features:

Upload many files at once from your desktop.
Edit titles, tags, and other metadata before uploading.
Quickly and easily manage bulk uploads, straight from your desktop.
Right-click to start uploading files directly to Scribd (Windows only).

Links: Project Home Page on GitHub | Home Page on Scribd.com.

Changelog

September 9, 2009
- Fixed a link to GitHub homepage of the rspec-cells plugin.
September 8, 2009
- Added rspec-cells plugin.

The post Scribd open source projects first appeared on Dmytro Shteflyuk's Home.

opensource | Dmytro Shteflyuk's Home

Submitting a patch to the Open Source project: composite_primary_keys

Step 0. Reproducing the bug

Step 1. Setting up an environment for composite_primary_keys gem

Step 2. Reproducing failing tests inside composite_primary_keys test suite

Step 3. Fixing the bug

Step 4. Committing changes and pulling request

Credits

Scribd open source projects

Table of Contents

Changelog