Darian Shimy » realtime

Hot Potato – A Real-time Processing Framework

Darian Shimy — Wed, 27 Jul 2011 16:00:22 +0000

Today, I am happy to announce the availability of HotPotato. Hot Potato is an open source real-time processing framework written in Ruby. Originally designed to process the Twitter firehose at 3,000+ tweets per second, it has been extended to support any type of streaming data as input or output to the framework. The framework excels with applications such as, social media analysis, log processing, fraud prevention, spam detection, instant messaging, and many others that include the processing of streaming data.

Efficient Real-Time Memory De-duplication

Darian Shimy — Fri, 11 Feb 2011 06:33:31 +0000

De-duplication is the process of removing duplicates from a collection. Hashes and bloom filters are common tools to use when implementing de-duplication, however, there are times when these are not fast enough. It is fairly simple to implement a memory efficient real-time de-duplication system using two hashes. The secret is in adding the objects to both hashes and every n objects (or time-based) making the current hash become the next and clearing the current once the number of objects reaches a certain threshold.

This is easier explained by a simple reference implementation in Ruby:

# A memory efficient deduplication class used in cases where a
# duplicate object can only occur within n objects of each other.
class MemoryDedup
 
  # Returns a new deduper.
  #
  # == Options
  # * :size - the number of objects to store in the cache
  def initialize(options = {})
    @size = options{:size} || 1000
    @current = {}
    @next = {}
  end
 
  # Returns true if the object has been seen in the past :size unique
  # objects, false otherwise.
  def exists?(obj)
    if (2 * @current.size) > @size
      @current = @next
      @next = {}
    end
    @next[obj] = true
    if @current.has_key?(obj)
      return true
    else
      @current[obj] = true
      return false
    end
  end
 
end

Realtime Log Visualization

Darian Shimy — Thu, 10 Dec 2009 22:00:16 +0000

Erlend Simonsen put out a program called gltail that creates a graphical representation of your log files.

I saw this a long time ago, but last night I finally started playing with it. It works quite well. I needed to make a small change to support an SSH gateway which you can get here: http://github.com/dshimy/gltail. On OS X, getting started was simple:

$ git clone git://github.com/dshimy/gltail.git
$ sudo gem install ruby-opengl file-tail

If you want the dots to bump each other, you need to install the Chipmunk physics library:

$ cd vendor/Chipmunk-4.1.0/ruby
$ ruby extconf.rb
$ sudo make install

Beyond that, tweak the configuration file and enjoy.

Yahoo Search Index Speed

Darian Shimy — Thu, 10 Sep 2009 06:19:00 +0000

I submitted this blog to Yahoo a while ago. Why? It was more of a test of how fast they spider sites rather than an attempt to drive traffic. After the submission, they stated:

Thank you! Your URL has been added to our list of URLs to crawl. Please expect a delay of several weeks before your URL is crawled.

Several weeks? Really!

Since the submission, it appears Yahoo is doing their best to meet their goal of several weeks.