Darian Shimy

WebSockets that Reconnect

Darian Shimy — Sat, 01 Oct 2011 06:06:15 +0000

Here is a JavaScript sample of a WebSocket that will try to reconnect every second if the connection is lost to the server. This is completely transparent to the end-user.

function WS() {
  var _self = this;
  this.start = function() {
    var ws;
    ws = new WebSocket("ws://example.com/");					
    ws.onmessage = function(e) {	
      // Do something
    };
    ws.onclose = function(e) {
      clearTimeout(_self.refresh);
      setTimeout(_self.start, 1000);
    }            
  }
  _self.start();
} 
new WS();

Passenger and ruby-oci8 (Oracle)

Darian Shimy — Sat, 01 Oct 2011 05:06:22 +0000

Oracle is going to drive a man to drink. After getting our rake migrations to run with this solution, our application had issues. We run our Rails app in Passenger + Apache2. After we deployed, we got this error:

ERROR: ActiveRecord oracle_enhanced adapter could not load ruby-oci8 library. Please install ruby-oci8 gem.

As it turns out, Passenger does not pick up the LD_LIBRARY_PATH and the only way I’m aware of set it is to create a wrapper for the ruby binary. To fix this, create the file /usr/local/bin/passenger-ruby:

#!/bin/sh
 
export LD_LIBRARY_PATH=/opt/oracle/instantclient_11_2
exec "/usr/bin/ruby" "$@"

Then update passenger.conf and change PassengerRuby to:

PassengerRuby /usr/local/bin/passenger-ruby

Don’t forget to restart Apache. Good luck.

RabbitMQ on OS X via MacPorts

Darian Shimy — Thu, 22 Sep 2011 16:05:53 +0000

MacPorts installs an old version of RabbitMQ. To get the latest, you need to add the RabbitMQ repository as described here:

http://www.rabbitmq.com/macports.html

However, when you try and install it, it throws the following error:

$ sudo port install rabbitmq-server
Error: Unable to open port: tar: +CONTENTS: Not found in archive
tar: Error exit delayed from previous errors.
To report a bug, see

As it turns out, there is a bug in MacPorts 2.0.1.

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-September/014855.html

Upgrading to 2.0.3 resolved it.

ruby-oci8 library problems

Darian Shimy — Thu, 08 Sep 2011 03:25:52 +0000

We started using Oracle with a Ruby on Rails 2.3.x project. I will say, it has been nothing but a pain. After getting the Oracle Instant Client installed and the ruby-oci library compiled, I kept hitting this error during the deployment (BTW: this was on Ubuntu 10.04):

ERROR: ActiveRecord oracle_enhanced adapter could not load ruby-oci8 library. Please install ruby-oci8 gem.

But it was installed. Testing from irb:

irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'oci8'
LoadError: libaio.so.1: cannot open shared object file: No such file or directory - /usr/lib/ruby/gems/1.8/gems/ruby-oci8-2.0.6/lib/oci8lib_18.so
	from /usr/lib/ruby/gems/1.8/gems/ruby-oci8-2.0.6/lib/oci8lib_18.so
	from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
	from /usr/lib/ruby/gems/1.8/gems/ruby-oci8-2.0.6/lib/oci8.rb:38
	from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
	from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
	from (irb):2
	from :0

After some snooping around, I found the libaio library missing. Sigh.

$ apt-get install libaio-dev

Now it works.

Hot Potato – A Real-time Processing Framework

Darian Shimy — Wed, 27 Jul 2011 16:00:22 +0000

Today, I am happy to announce the availability of HotPotato. Hot Potato is an open source real-time processing framework written in Ruby. Originally designed to process the Twitter firehose at 3,000+ tweets per second, it has been extended to support any type of streaming data as input or output to the framework. The framework excels with applications such as, social media analysis, log processing, fraud prevention, spam detection, instant messaging, and many others that include the processing of streaming data.

Signal Handling

Darian Shimy — Tue, 19 Apr 2011 04:33:46 +0000

A signal is a way to notify a process of an event from another process. A signal handler allows one execute code when a signal is received. For example, the most common signal sent from the command-line is SIGINT via CTRL-C. The default behavior is to terminate the application. We can change that to print diagnostic information:

#!/usr/bin/env ruby
 
Signal.trap('INT') { puts 'DEBUG: ...' }
sleep 60

Start the above program and press CTRL-C or kill -2 pid. You can optionally add an exit statement to the code block as well to exit.

Why do I care?

If you just write Rails apps then you probably won’t, however if you write a daemon or long running process, there is a good chance you’ll want to handle the signal appropriately. For example, I had a long running perfomance test script and didn’t want to loose the data if I decided to stop the process. In this case (similar to the example), I had the application print the progress to STDOUT before exiting.

References:

EventMachine Process Control

Darian Shimy — Tue, 22 Mar 2011 16:26:58 +0000

A while ago I wrote a pilot project for a real-time twitter analysis tool using eventmachine and websockets. It worked perfectly, but there were questions how to deploy it to production. Specifically, how to handle process management, logging, configuration, deployment, and monitoring. This post describes how I handled process management.

Process management is a facility to support starting and stopping of long running processes as well as inquire if a process is running. Generally, only one instance of the process should be running, so support for checking for a running process is needed.

Starting and stopping the process can easily be handled by the Daemons Gem. From their webpage: “Daemons provides an easy way to wrap existing ruby scripts (for example a self-written server) to be run as a daemon and to be controlled by simple start/stop/restart commands.”

In the code example below, we wrap our websocket.rb file located in the ./lib/websockets directory.

#!/usr/bin/env ruby
 
APP_ROOT = File.expand_path(File.join(File.dirname(__FILE__) + '/..'))
APP_ENV =  ENV["APP_ENV"] || "development"
$LOAD_PATH.unshift(APP_ROOT + '/lib') unless $LOAD_PATH.include?(APP_ROOT + '/lib')
 
require 'rubygems'
require 'daemons'
require 'websockets'
 
def stop_and_exit
  # Do something useful here....
  exit 0
end
 
Signal.trap('INT') { stop_and_exit }
Signal.trap('TERM'){ stop_and_exit }
 
options = {
  :app_name => 'Websocket',
  :multiple => false,
  :log_output => true,
  :dir_mode => :normal,
  :dir => File.join(APP_ROOT, 'log')
}
Daemons.run(File.join(APP_ROOT, 'lib', 'websockets', 'websocket.rb'), options)

Running ./websocket -h yields:

websocket -h
Usage: Websocket   -- 
 
* where  is one of:
  start         start an instance of the application
  stop          stop all instances of the application
  restart       stop all instances and restart them afterwards
  reload        send a SIGHUP to all instances of the application
  run           start the application and stay on top
  zap           set the application to a stopped state
  status        show status (PID) of application instances
 
* and where  may contain several of the following:
 
    -t, --ontop                      Stay on top (does not daemonize)
    -f, --force                      Force operation
    -n, --no_wait                    Do not wait for processes to stop
 
Common options:
    -h, --help                       Show this message
        --version                    Show version

References

Efficient Real-Time Memory De-duplication

Darian Shimy — Fri, 11 Feb 2011 06:33:31 +0000

De-duplication is the process of removing duplicates from a collection. Hashes and bloom filters are common tools to use when implementing de-duplication, however, there are times when these are not fast enough. It is fairly simple to implement a memory efficient real-time de-duplication system using two hashes. The secret is in adding the objects to both hashes and every n objects (or time-based) making the current hash become the next and clearing the current once the number of objects reaches a certain threshold.

This is easier explained by a simple reference implementation in Ruby:

# A memory efficient deduplication class used in cases where a
# duplicate object can only occur within n objects of each other.
class MemoryDedup
 
  # Returns a new deduper.
  #
  # == Options
  # * :size - the number of objects to store in the cache
  def initialize(options = {})
    @size = options{:size} || 1000
    @current = {}
    @next = {}
  end
 
  # Returns true if the object has been seen in the past :size unique
  # objects, false otherwise.
  def exists?(obj)
    if (2 * @current.size) > @size
      @current = @next
      @next = {}
    end
    @next[obj] = true
    if @current.has_key?(obj)
      return true
    else
      @current[obj] = true
      return false
    end
  end
 
end

Sinaglo – A Sinatra-based Blog backed by MongoDB

Darian Shimy — Sun, 23 Jan 2011 00:59:17 +0000

I will say, I love the simplicity of Sinatra and MongoDB and Sinaglo is a great way to demonstrate Sinatra and MongoDB working together for a semi-useful application.

Sinaglo is everything you need and nothing you don’t. This started out as a project to learn more about Sinatra and turned out to be a useful blog software. No, it does not do what WordPress does, but it does everything I needed it to do. If anything, this should be a good starting point for a Sinatra based application. Find it here:

https://github.com/dshimy/sinaglo

Rails 2.3.8 Deprecation Warning

Darian Shimy — Tue, 25 May 2010 23:00:34 +0000

If you happen to come across the following deprecation warning:

DEPRECATION WARNING: Giving :session_key to SessionStore is deprecated, please use :key instead. (called from new at /Users/dshimy/.rvm/gems/ruby-1.8.7-p249/gems/actionpack-2.3.8/lib/action_controller/middleware_stack.rb:72)

Take a look at your config/environment.rb file for the following:

config.action_controller.session = {
    :secret_key    => '_sample_session',
    :secret        => '00000000000000000000'
}

and change it to:

config.action_controller.session = {
    :key    => '_sample_session',
    :secret => '00000000000000000000'
}