Mac OS X | Dmytro Shteflyuk's Home

Installing and Using Scribe with Ruby on Mac OS

Dmytro Shteflyuk — Wed, 13 May 2009 18:46:24 +0000

In Scribd we have tons of analytics data generated daily that should be somehow processed. Currently we use MySQL to store all this stuff, but that is not the best option for logging lots of data. So we’ve decided to try some more specialized tools, which could help us to store and process our data. The most interesting thing which could simplify analytics data collecting was Scribe. As it turned out, installation process is not so simple as expected so here you will find a few steps manual on how to install Scribe on a developer machine.

0. Prerequisites

First thing you’ll need is to install thrift library that is used by Scribe to do all the networking communication. To build it you need to have boost C++ library installed:

1	sudo port install boost

1. Installing Thrift

Now we are ready to download and build thrift. I prefer to keep all manually built tools in /opt:

1
2
3
4
5
6

git clone git://git.thrift-rpc.org/thrift.git
cd thrift
./bootstrap.sh
./configure --prefix=/opt/thrift
sudo make
sudo make install

Please note, that make command needs root privileges (it installs Ruby bindings to the system folder).

You will also need to install fb303 library (it is used in all facebook/thrift related tools to do status/health monitoring calls):

1
2
3
4
5

cd contrib/fb303
./bootstrap.sh
./configure --prefix=/opt/fb303 --with-thriftpath=/opt/thrift
make
sudo make install

2. Installing Scribe

Download Scribe from Sourceforge. Of course, you can use current development version from SVN.

1
2
3
4
5

cd scribe
./bootstrap.sh
./configure --prefix=/opt/scribe --with-thriftpath=/opt/thrift --with-fb303path=/opt/fb303
make
sudo make install

3. Configuring Scribe

I’ve created the /opt/scribe/conf directory and copied examples/example1.conf configuration there:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

##
## Sample Scribe configuration
##

# This file configures Scribe to listen for messages on port 1463 and write
# them to /tmp/scribetest

port=1463
max_msg_per_second=2000000
check_interval=3

# DEFAULT
>
category=default
type=buffer

target_write_size=20480
max_write_interval=1
buffer_send_rate=2
retry_interval=30
retry_interval_range=10

>
type=file
fs_type=std
file_path=/tmp/scribetest
base_filename=thisisoverwritten
max_size=1000000
add_newlines=1
>

>
type=file
fs_type=std
file_path=/tmp
base_filename=thisisoverwritten
max_size=3000000
>
>

Now we are ready to start scribe server:

1	sudo /opt/scribe/bin/scribed -c /opt/scribe/conf/example1.conf

4. Creating a test Ruby client application

First thing you’ll need is to gather all Ruby bindings into your app directory:

1
2
3
4
5

mkdir testapp
cd testapp
/opt/thrift/bin/thrift -o . -I /opt/fb303/share/ --gen rb /path/to/downloaded/scribe/if/scribe.thrift
/opt/thrift/bin/thrift -o . -I /opt/fb303/share/ --gen rb /opt/fb303/share/fb303/if/fb303.thrift
mv gen-rb scribe

Do not forget to replace /path/to/downloaded/scribe with real path where you have extracted Scribe sources to.

And here is the test console Ruby script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

#!/usr/bin/env ruby

$LOAD_PATH.unshift(File.dirname(__FILE__) + '/scribe')
require 'scribe'

begin
socket = Thrift::Socket.new('localhost', 1463)
transport = Thrift::FramedTransport.new(socket)
protocol = Thrift::BinaryProtocol.new(transport, false)
client = Scribe::Client.new(protocol)
transport.open()
log_entry = LogEntry.new(:category => 'test', :message => 'This is a test message')
client.Log([log_entry])
transport.close()
rescue Thrift::Exception => tx
print 'Thrift::Exception: ', tx.message, "\n"
end

Woot, it works! Time to start creating your highly productive logging/data collection system.

P.S. On my machine I got the following output from this script:

1
2
3
4
5
6
7

./scribe/fb303_types.rb:9: warning: already initialized constant DEAD
./scribe/fb303_types.rb:10: warning: already initialized constant STARTING
./scribe/fb303_types.rb:11: warning: already initialized constant ALIVE
./scribe/fb303_types.rb:12: warning: already initialized constant STOPPING
./scribe/fb303_types.rb:13: warning: already initialized constant STOPPED
./scribe/fb303_types.rb:14: warning: already initialized constant WARNING
./scribe/fb303_types.rb:15: warning: already initialized constant VALID_VALUES

To fix this issue, open the generated scribe/scribe_types.rb and replace require 'fb303_types' line with this:

1	require File.dirname(__FILE__) + '/fb303_types'

The post Installing and Using Scribe with Ruby on Mac OS first appeared on Dmytro Shteflyuk's Home.

Customizing iTerm. Creating a display profile with pastel colors

Dmytro Shteflyuk — Mon, 04 May 2009 15:09:40 +0000

Last week I have posted an article on how to write ssh host name on the iTerm’s background. I have been looking for something like this for a while, and this is that killer feature, which forced me to switch to iTerm from classic Terminal app. Here I will show what settings I have tuned to get iTerm more comfortable to use as for me.

First thing necessary to do is enabling UTF-8 support. If you create folders in Finder with extended characters (e.g. Russian, Japanese, etc.) iTerm will only show question marks instead of the real characters. To fix this you just have to set the LANG environment variable of the shell (put this into the ~/.bash_profile):

1	export LANG=ru_RU.UTF-8

The next time you will start an iTerm session the variable will be set and unicode characters will be shown correctly.

A list of all supported UTF-8 locales could be retrieved with the command:

1	locale -a \| grep UTF-8

Next thing is to fix an input problem (from iTerm FAQ). If you can’t input Chinese/Japanese/Umlauts/Accents, put this into your ~/.bash_profile:

1
2
3
4

set meta-flag on
set input-meta on
set output-meta on
set convert-meta off

Ok, everything works now and all I want is to migrate my color scheme options from Terminal. I love dark color scheme with pastel ANSI colors theme:

There is a great feature of iTerm called bookmarks. You can create display, keyboard, and terminal profiles, and then add bookmarks with these profiles associated to them. Each bookmark opens a terminal window with specified command in it. By default iTerm creates a default profile with something like login -fp kpumuk (for my machine).

I’ve created a new display profile called “Pastel” with nice pastel colors which replace default ANSI colors in terminal (click to enlarge):

I think I’m not alone in my pastel wishes, so here is the script, which will import Pastel display profile to your iTerm, and will assign it to the Default bookmark. Please note: you should run this script from Terminal.app, and do not forget to quit from iTerm before running, because settings will not be applied in this case!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138

#!/bin/bash

PASTEL='{
"Ansi 0 Color" = {
"Blue Component" = 0.3097887;
"Green Component" = 0.3097887;
"Red Component" = 0.3097887;
};
"Ansi 1 Color" = {
"Blue Component" = 0.3764706;
"Green Component" = 0.4235294;
"Red Component" = 1;
};
"Ansi 10 Color" = {
"Blue Component" = 0.6727703;
"Green Component" = 1;
"Red Component" = 0.8094148;
};
"Ansi 11 Color" = {
"Blue Component" = 0.7996491;
"Green Component" = 1;
"Red Component" = 1;
};
"Ansi 12 Color" = {
"Blue Component" = 0.9982605;
"Green Component" = 0.8627756;
"Red Component" = 0.7116503;
};
"Ansi 13 Color" = {
"Blue Component" = 0.9965209;
"Green Component" = 0.6133059;
"Red Component" = 1;
};
"Ansi 14 Color" = {
"Blue Component" = 0.9970397;
"Green Component" = 0.8763103;
"Red Component" = 0.8759136;
};
"Ansi 15 Color" = {
"Blue Component" = 1;
"Green Component" = 1;
"Red Component" = 1;
};
"Ansi 2 Color" = {
"Blue Component" = 0.3764706;
"Green Component" = 1;
"Red Component" = 0.6588235;
};
"Ansi 3 Color" = {
"Blue Component" = 0.7137255;
"Green Component" = 1;
"Red Component" = 1;
};
"Ansi 4 Color" = {
"Blue Component" = 0.9960784;
"Green Component" = 0.7960784;
"Red Component" = 0.5882353;
};
"Ansi 5 Color" = {
"Blue Component" = 0.9921569;
"Green Component" = 0.4509804;
"Red Component" = 1;
};
"Ansi 6 Color" = {
"Blue Component" = 0.9960784;
"Green Component" = 0.772549;
"Red Component" = 0.7764706;
};
"Ansi 7 Color" = {
"Blue Component" = 0.9335317;
"Green Component" = 0.9335317;
"Red Component" = 0.9335317;
};
"Ansi 8 Color" = {
"Blue Component" = 0.4862745;
"Green Component" = 0.4862745;
"Red Component" = 0.4862745;
};
"Ansi 9 Color" = {
"Blue Component" = 0.6901961;
"Green Component" = 0.7137255;
"Red Component" = 1;
};
"Anti Alias" = 1;
"Background Color" = {
"Blue Component" = 0;
"Green Component" = 0;
"Red Component" = 0;
};
Blur = 1;
"Bold Color" = {
"Blue Component" = 0.5067359;
"Green Component" = 0.5067359;
"Red Component" = 0.9909502;
};
Columns = 120;
"Cursor Color" = {
"Blue Component" = 0.3764706;
"Green Component" = 0.6470588;
"Red Component" = 1;
};
"Cursor Text Color" = {
"Blue Component" = 1;
"Green Component" = 1;
"Red Component" = 1;
};
"Disable Bold" = 0;
Font = "Monaco 14";
"Foreground Color" = {
"Blue Component" = 1;
"Green Component" = 1;
"Red Component" = 1;
};
"Horizontal Character Spacing" = 1;
NAFont = "Monaco 14";
Rows = 24;
"Selected Text Color" = {
"Blue Component" = 0.9476005;
"Green Component" = 0.9476005;
"Red Component" = 0.9476005;
};
"Selection Color" = {
"Blue Component" = 0.5153061;
"Green Component" = 0.2224857;
"Red Component" = 0.2099074;
};
Transparency = 0.1;
"Vertical Character Spacing" = 1;
}'

# Add display profile
defaults write net.sourceforge.iTerm Displays -dict-add Pastel "$PASTEL"
echo "Pastel display profile added"

# Set the default display profile
BOOKMARKS=`defaults read net.sourceforge.iTerm Bookmarks | sed 's/$"Display Profile" = $"[^"]*";/\1"Pastel";/'`
defaults write net.sourceforge.iTerm Bookmarks "$BOOKMARKS"
echo "Pastel display profile installed as default"

You can download this script from GitHub here.

Hope, you will find this post useful. Fill free to post your comments and suggestions!

Changelog

2011-08-09 — updated gist URL and install command.

2010-07-13 — fixed bug which was making iTerm totally unusable in some cases.

The post Customizing iTerm. Creating a display profile with pastel colors first appeared on Dmytro Shteflyuk's Home.

How to show SSH host name on the iTerm’s background

Dmytro Shteflyuk — Fri, 01 May 2009 09:30:50 +0000

How many ssh session do you open usually? In Scribd we have about 50 machines, and most of the time I have to connect to several of them to do my work. But there is a big problem — it’s hard to distinguish among different tabs in iTerm. Of course, you can see the host where you connected to in the tab title, but the title is really small, low contrast, etc. So I had a dream since I’ve started using Mac — to get current host written with large letters on the background.

iTerm has a great support of AppleScript. Looking through the commands list I’ve found a really useful one: background image path. The last thing to figure out is how to create an image with the current host name, and I decided to use ImageMagick. So, let’s do it!

First, you need to install ImageMagick. You can use binary distribution, but I’d prefer the MacPorts version:

1	port install ImageMagick

Let’s create a test image:

1
2
3

convert -size "570x342" xc:black -gravity "NorthEast" -fill "#662020" \
-family "Georgia" -style "Normal" -pointsize "60" -antialias \
-draw "text 20,10 'kpumuk.info'" "/tmp/iTermBG.png"

Here is the result and it looks pretty good!

Ok, now the most interesting part. I’ve created a bash script in ~/bin/ssh, which sets the background before calling ssh and clears it on exit. I suppose you have this folder specified before /usr/bin in your .bash_profile:

1	export PATH=$HOME/bin:/opt/local/bin:$PATH

Here is the script (the latest version could be found in a GitHub repository here):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

#!/bin/bash

# SSH with host name and IP address in background (only in iTerm.app)

# First, check to see if we have the correct terminal!
if [ "$(tty)" == 'not a tty' ] || [ "$TERM_PROGRAM" != "iTerm.app" ] ; then
/usr/bin/ssh "$@"
exit $?
fi

function __calculate_iterm_window_dimensions {
local size=( $(osascript -e "tell application "iTerm"
get bounds of window 1
end tell" | tr ',' ' ') )

local x1=${size[0]} y1=${size[1]} x2=${size[2]} y2=${size[3]}
# 15px - scrollbar width
local w=$(( $x2 - $x1 - 15 ))
# 44px - titlebar + tabs height
local h=$(( $y2 - $y1 - 44))
echo "${w}x${h}"
}

# Console dimensions
DIMENSIONS=$(__calculate_iterm_window_dimensions)

BG_COLOR="#000000" # Background color
FG_COLOR="#662020" # Foreground color
GRAVITY="NorthEast" # Text gravity (NorthWest, North, NorthEast,
# West, Center, East, SouthWest, South, SouthEast)
OFFSET1="20,10" # Text offset (host name)
OFFSET2="20,80" # Text offset (ip address)
FONT_SIZE="60" # Font size in points
FONT_STYLE="Normal" # Font style (Any, Italic, Normal, Oblique)
# Font path
FONT="$HOME/.bash/resources/SimpleLife.ttf"

HOSTNAME=`echo $@ | sed -e "s/.*@//" -e "s/ .*//"`

output=`dscacheutil -q host -a name $HOSTNAME`
RESOLVED_HOSTNAME=`echo -e "$output"|grep '^name:'|awk '{print $2}'`
RESOLVED_IP=`echo -e "$output"|grep '^ip_address:'|awk '{print $2}'`

function set_bg {
local tty=$(tty)
osascript -e "
tell application "iTerm"
repeat with theTerminal in terminals
tell theTerminal
try
tell session id "$tty"
set background image path to "$1"
end tell
on error errmesg number errn
end try
end tell
end repeat
end tell"
}

on_exit () {
if [ ! -f /tmp/iTermBG.empty.png ]; then
convert -size "$DIMENSIONS" xc:"$BG_COLOR" "/tmp/iTermBG.empty.png"
fi
set_bg "/tmp/iTermBG.empty.png"
rm "/tmp/iTermBG.$$.png"
}
trap on_exit EXIT

convert \
-size "$DIMENSIONS" xc:"$BG_COLOR" -gravity "$GRAVITY" -fill "$FG_COLOR" \
-font "$FONT" -style "$FONT_STYLE" -pointsize "$FONT_SIZE" -antialias \
-draw "text $OFFSET1 '${RESOLVED_HOSTNAME:-$HOSTNAME}'" \
-pointsize 30 -draw "text $OFFSET2 '${RESOLVED_IP:-}'" \
"/tmp/iTermBG.$$.png"
set_bg "/tmp/iTermBG.$$.png"

/usr/bin/ssh "$@"

The script is big, but very simple. In lines 6-9 script detects the current terminal application, and if it is not iTerm — skips the background generation. Also it checks if the SSH connection was requested from an interactive command line or from an another script (e.g., from git). If you want to see what host git tries to connect to — remove the tty check. Function on lines 11-22 calculates the iTerm window size, so generated background will not be stretched. If you have disabled scrollbars — remove - 15 on line 18. Configuration options determining how host name will be printed on your background are on lines 27-36. Lines 38-42 normalize host name and resolve IP address. Function on lines 44-59 changes background of the iTerm tab where ssh command was invoked from using AppleScript. It’s tricky a little, but still easy to understand. Lines 61-68 trap script exit. They handle special case when you kill ssh session, and get background image for non-ssh console. Lines 70-76 generate background image using ImageMagick.

Let’s see it in action:

So now I can stop shaking with fear when doing some destructive things in console — I know which host exactly I’m trying to kill. I hope you will find this post useful.

Changelog

05/01/2009: Updated on_exit function to fix bug with scrolling after ssh session being closed.

05/03/2009: Added some variables to customize how host name will look.

06/23/2010: Better console size detection, added normalization of host name, and IP address below the hostname, iTerm detection (to prevent errors in Terminal.app).

The post How to show SSH host name on the iTerm’s background first appeared on Dmytro Shteflyuk's Home.