thrift | Dmytro Shteflyuk's Home https://kpumuk.info In my blog I'll try to describe about interesting technologies, my discovery in IT and some useful things about programming. Mon, 07 Sep 2015 23:10:58 +0000 en-US hourly 1 https://wordpress.org/?v=6.7.1 Installing and Using Scribe with Ruby on Mac OS https://kpumuk.info/mac-os-x/installing-and-using-scribe-with-ruby-on-mac-os/ https://kpumuk.info/mac-os-x/installing-and-using-scribe-with-ruby-on-mac-os/#comments Wed, 13 May 2009 18:46:24 +0000 http://kpumuk.info/?p=668 In Scribd we have tons of analytics data generated daily that should be somehow processed. Currently we use MySQL to store all this stuff, but that is not the best option for logging lots of data. So we’ve decided to try some more specialized tools, which could help us to store and process our data. […]

The post Installing and Using Scribe with Ruby on Mac OS first appeared on Dmytro Shteflyuk's Home.]]> Scribe In Scribd we have tons of analytics data generated daily that should be somehow processed. Currently we use MySQL to store all this stuff, but that is not the best option for logging lots of data. So we’ve decided to try some more specialized tools, which could help us to store and process our data. The most interesting thing which could simplify analytics data collecting was Scribe. As it turned out, installation process is not so simple as expected so here you will find a few steps manual on how to install Scribe on a developer machine.

0. Prerequisites

First thing you’ll need is to install thrift library that is used by Scribe to do all the networking communication. To build it you need to have boost C++ library installed:

1
sudo port install boost

1. Installing Thrift

Now we are ready to download and build thrift. I prefer to keep all manually built tools in /opt:

1
2
3
4
5
6
git clone git://git.thrift-rpc.org/thrift.git
cd thrift
./bootstrap.sh
./configure --prefix=/opt/thrift
sudo make
sudo make install

Please note, that make command needs root privileges (it installs Ruby bindings to the system folder).

You will also need to install fb303 library (it is used in all facebook/thrift related tools to do status/health monitoring calls):

1
2
3
4
5
cd contrib/fb303
./bootstrap.sh
./configure --prefix=/opt/fb303 --with-thriftpath=/opt/thrift
make
sudo make install

2. Installing Scribe

Download Scribe from Sourceforge. Of course, you can use current development version from SVN.

1
2
3
4
5
cd scribe
./bootstrap.sh
./configure --prefix=/opt/scribe --with-thriftpath=/opt/thrift --with-fb303path=/opt/fb303
make
sudo make install

3. Configuring Scribe

I’ve created the /opt/scribe/conf directory and copied examples/example1.conf configuration there:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
##
## Sample Scribe configuration
##

# This file configures Scribe to listen for messages on port 1463 and write
# them to /tmp/scribetest

port=1463
max_msg_per_second=2000000
check_interval=3

# DEFAULT
<store>
category=default
type=buffer

target_write_size=20480
max_write_interval=1
buffer_send_rate=2
retry_interval=30
retry_interval_range=10

<primary>
type=file
fs_type=std
file_path=/tmp/scribetest
base_filename=thisisoverwritten
max_size=1000000
add_newlines=1
</primary>

<secondary>
type=file
fs_type=std
file_path=/tmp
base_filename=thisisoverwritten
max_size=3000000
</secondary>
</store>

Now we are ready to start scribe server:

1
sudo /opt/scribe/bin/scribed -c /opt/scribe/conf/example1.conf

4. Creating a test Ruby client application

First thing you’ll need is to gather all Ruby bindings into your app directory:

1
2
3
4
5
mkdir testapp
cd testapp
/opt/thrift/bin/thrift -o . -I /opt/fb303/share/ --gen rb /path/to/downloaded/scribe/if/scribe.thrift
/opt/thrift/bin/thrift -o . -I /opt/fb303/share/ --gen rb /opt/fb303/share/fb303/if/fb303.thrift
mv gen-rb scribe

Do not forget to replace /path/to/downloaded/scribe with real path where you have extracted Scribe sources to.

And here is the test console Ruby script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/usr/bin/env ruby

$LOAD_PATH.unshift(File.dirname(__FILE__) + '/scribe')
require 'scribe'

begin
  socket = Thrift::Socket.new('localhost', 1463)
  transport = Thrift::FramedTransport.new(socket)
  protocol = Thrift::BinaryProtocol.new(transport, false)
  client = Scribe::Client.new(protocol)
  transport.open()
  log_entry = LogEntry.new(:category => 'test', :message => 'This is a test message')
  client.Log([log_entry])
  transport.close()
rescue Thrift::Exception => tx
  print 'Thrift::Exception: ', tx.message, "\n"
end

Woot, it works! Time to start creating your highly productive logging/data collection system.

P.S. On my machine I got the following output from this script:

1
2
3
4
5
6
7
./scribe/fb303_types.rb:9: warning: already initialized constant DEAD
./scribe/fb303_types.rb:10: warning: already initialized constant STARTING
./scribe/fb303_types.rb:11: warning: already initialized constant ALIVE
./scribe/fb303_types.rb:12: warning: already initialized constant STOPPING
./scribe/fb303_types.rb:13: warning: already initialized constant STOPPED
./scribe/fb303_types.rb:14: warning: already initialized constant WARNING
./scribe/fb303_types.rb:15: warning: already initialized constant VALID_VALUES

To fix this issue, open the generated scribe/scribe_types.rb and replace require 'fb303_types' line with this:

1
require File.dirname(__FILE__) + '/fb303_types'
The post Installing and Using Scribe with Ruby on Mac OS first appeared on Dmytro Shteflyuk's Home.]]> https://kpumuk.info/mac-os-x/installing-and-using-scribe-with-ruby-on-mac-os/feed/ 5