betweenGo

Twitter has become the poster child of why not to use Ruby on Rails.  Recently a friend of mine started a company and when I noticed he used PHP I asked why he didn’t use Rails.  He responded “Twitter uses Rails and they keep going down.  Facebook uses PHP.  We’ll use PHP.”

There is an interesting interview w/ Twitter developer Alex Payne about the issues Twitter has had with Ruby on Rails.

The common wisdom in the Rails community at this time is that scaling Rails is a matter of cost: just throw more CPUs at it. The problem is that more instances of Rails (running as part of a Mongrel cluster, in our case) means more requests to your database. At this point in time there’s no facility in Rails to talk to more than one database at a time. The solutions to this are caching the hell out of everything and setting up multiple read-only slave databases..

All the convenience methods and syntactical sugar that makes Rails such a pleasure for coders ends up being absolutely punishing, performance-wise. Once you hit a certain threshold of
traffic, either you need to strip out all the costly neat stuff that Rails does for you (RJS, ActiveRecord, ActiveSupport, etc.) or move the slow parts of your application out of Rails, or both.

It’s also worth mentioning that there shouldn’t be doubt in anybody’s mind at this point that Ruby itself is slow. It’s great that people are hard at work on faster implementations of the language, but right now, it’s tough. If you’re looking to deploy a big web application and you’re language-agnostic, realize that the same operation in Ruby will take less time in Python. All of us working on Twitter are big Ruby fans, but I think it’s worth being frank that this isn’t one of those relativistic language issues. Ruby is slow.

In April 2007, former Chief Architect Blaine Cook made a presentation called Scaling Twitter.  It’s a well done presentation, even if everything is black.

In May 2008 there were rumors that Twitter was abandoning Rails.  SitePoints believes it is not a framework issue but rather an architectural issue that is causing Twitter’s problems.

There are many large sites that run Rails well and many smaller ones that do not.  It seems like another case of the “leaky abstraction” where we have such a nice framework that we think we can abstract away issues like scaling but in the end when the site becomes big enough we have to look under the covers.

Ruby has been documented to have I/O issues and general performance issues.  Like PHP before it looks like Ruby needs to mature more and deal better with scaling and I/O issues.  I am confident though this will happen because of the tremendous momentum of Ruby on Rails.  However I am keeping on eye on Django, especially now that Google has thrown its large hat into Django’s ring.

This post about Monit and Mongrel is based on this fantastic post, Monit makes Mongrel play nice!

Install Monit from source:

$ wget http://www.tildeslash.com/monit/dist/monit-4.10.1.tar.gz
$ tar xvfz monit-4.10.1.tar.gz
$ cd monit-4.10.1
$ ./configure -prefix=/usr  (the default prefix is /usr/local)
$ make
$ make install

Configure Monit:

In the monit source you will find a sample monitrc which is completely commented out. You can then copy it to wherever you like and then edit it. Here is my monitrc.

# Start monit in the background (run as a daemon) and check services at
# 1-minute intervals.
set daemon  60

# Set logging.
set logfile /tmp/monit.log

# Set the list of mail servers for alert delivery.
set mailserver localhost

# Set alert recipients.
set alert foo@example.com                # receive all alerts

# Start Monit's embedded web server.
set httpd port 2812 and
     allow admin:monit      # require user 'admin' with password 'monit'

# Mongrel - Development
# Check that Mongrel is running and that it responds to HTTP
# requests. Check its resource usage such as cpu and memory. If the
# process is not running, monit will restart it by default. In case
# the service was restarted very often and the problem remains, it is
# possible to disable monitoring using the TIMEOUT statement.
check process mongrel-dev with pidfile /rails/log/mongrel.pid
    start program = "/usr/bin/mongrel_rails start -d  -c /rails
                          -p 3000 -P /rails/log/mongrel.pid"
    stop program  = "/usr/bin/mongrel_rails stop
                           -P /rails/log/mongrel.pid"
    if cpu > 50% for 2 cycles then alert
    if cpu > 80% for 5 cycles then restart
    if totalmem > 60.0 MB for 5 cycles then restart
    if loadavg(5min) greater than 10 for 8 cycles then restart
    if failed port 3000 protocol http
       with timeout 15 seconds
       for 2 cycles
       then restart
    if 3 restarts within 5 cycles then timeout
    group mongrel-dev

After you are done configuring monitrc copy it to /etc.

$ sudo cp monitrc /etc

To test the configuration:

$ sudo monit -t

Run Monit:

To start monit:

$ sudo monit

To restart monit:

$ sudo monit reload

To stop monit:

$ sudo monit quit

To see monit’s log (assuming you are using my configuration):

$ tail -f /tmp/monit.log

Learn More:

Monit Getting Started
Monit Manual

Today I took the plunge and installed Rails 2.0.2. Notes on the Rails 2.0 release can be found here.

Here are the steps I took.

1. Upgrade Ruby Gems to the latest version (in this case 1.0.1).
   gem update --system
2. Install Ruby on Rails 2.0.2.
   gem install rails -v 2.0.2
3. Install MySQL gem (as of March 5, 2008, it is version 2.7.3) and add MySQL executable to path.
   gem install mysql
4. Create prayer application.
   rails prayer -d mysql
5. Install Restful Authentication plugin.
   cd prayer
ruby script/plugin install http://svn.techno-weenie.net/projects/plugins/restful_authentication/
6. Create scaffold for prayer model.
   ruby script/generate scaffold Prayer title:string body:text

   The first thing I noticed different about Rails 2.0 is that the scaffold now also generates the model and data migration file with title and body as columns in the prayer table. Also the format of the data migration file is slightly more compact.
7. Create user model with authentication. The --include-activation flag is to create the mailers for activation and signup notification.
   ruby script/generate authenticated user sessions --include-activation
8. Configure config/database.yml and then create prayer and user tables in DB.
   rake db:migrate
9. Configure SMTP settings.
10. Modify app/models/user_mailer.rb, replacing the place holders with constants which you can configure with different values for different environments using config/environments/development.rb, config/environments/production.rb.
11. Modify user_observer.rb to incorporate the reset and forgotten password features.
12. Create scaffold for role model. Add an admin user.
    ruby script/generate scaffold Role rolename:string
13. Create permission model. Modify role and permission models to know about each other.
    ruby script/generate model Permission
14. Modify user model to use permissions.

Much of this post is based on this excellent post, Restful Authentication with all the bells and whistles.  At this point I did not continue with much of the tutorial, especially with parts like password forgotten, separate account controller, etc.

When I run rake db:migrate I get this error at the end.

Error in my_thread_global_end(): 1 threads didn't exit

I pinned it down to this code in one of my migration files.

# create admin user
user = User.new
user.login = 'admin'
user.password = 'password'
user.save(false)

If I don’t run user.save(false) I don’t get the error. I am not sure why, the user does get saved properly to the database.

I saw this post on the MySQL forums that seemed to indicate it was an issue with libmySQL.dll. So I upgraded my MySQL instance from 5.0.27 to 5.0.51a. Of course this did not go smoothly, I got this error when trying to reconfigure the MySQL server instance “MySQL service could not be started error 0″. Fortunately another post on the MySQL forums, Could not start service : Error 2003, solved this problem for me. I just had to remove the following files from the mysql/data directory.

• ib_logfile0
• ib_logfile1
• ibdata1

Unfortunately when I then did a rake db:migrate I saw this error.

Mysql::Error: Table 'prayer.schema_info' doesn't exist:
SELECT version FROM schema_info

After deleting and recreating the database I was finally able to run rake db:migrate. Unfortunately I still got the same error that inspired this post.

Googling some more I saw a MySQL bug report, MySQL Bugs: #25621: Error in my_thread_global_end(): 1 threads didn’t exit. Apparently this is a client side issue and I think I can safely ignore it though it is quite annoying.

Strictly speaking, this is not MySQL bug. This is a client bug - a client application (PHP, probably) linked with libmysqlclient calls my_thread_init() but does not call my_thread_end() as appropriate (doesn’t call at all or calls too late). MySQL client library detects this and issues a warning.

On the other hand, we can just remove the check and let buggy applications to fail some other way. Not calling my_thread_end() is a guaranteed memory leak. Calling it too late could easily crash the application.

I tried to update the restful_authentication plugin by doing this.

$ ruby script/plugin update restful_authentication

Unfortunately it didn’t do anything.

I know I don’t have the latest version because in another project I pulled down the latest restful_authentication code and it was different.

To get around this I forced an install with the latest code.

$ ruby script/plugin install -x \
    http://svn.techno-weenie.net/projects/plugins/restful_authentication/

Fortunately I received an answer about this on the rails mailing list. Apparently update only works if the plugin is in svn:externals. To do this you would install it with the -x flag (see list of flags by doing ruby script/plugin install -h’).

However if you try this with the restful_authentication plugin you’ll see this.

$ ruby script/plugin install -f \
    http://svn.techno-weenie.net/projects/plugins/restful_authentication/
Cannot install using externals because this project is not under
subversion.

Previously I talked about configuring a production Ruby on Rails applications using a Mongrel cluster with an Apache front end / load balancer. Now though a new Apache module was released called mod_rails which obviates the need for a Mongrel cluster. Seems promising, both for simplicity of installation and performance.

As mentioned in my post, Configuring Apache to work with a Mongrel Cluster, you need an Apache installation with mod_rewrite and mod_proxy_balancer. Unfortunately on the Red Hat installation I was using the default installation is Apache 2.0 which does not have mod_proxy_balancer. When I tried to upgrade it using up2date I got some errors that seemed to indicate this install was not properly registered. I then tried using the Apache 2.2 install done by the tech guy but this install did not have the modules I needed.

Therefore it was time for me to do the install by myself. Fortunately it was not hard with the help of this article, Building Apache 2.2 plus the manual.

Here is what I did.

$ wget http://apache.mirror99.com/httpd/httpd-2.2.8.tar.gz
$ tar xvfz httpd-2.2.8.tar.gz
$ cd httpd-2.2.8
$ ./configure --enable-rewrite=shared --enable-proxy=shared \
    --enable-proxy_balancer=shared --enable-proxy_http=shared
$ make
$ sudo make install
$ sudo /usr/local/apache2/bin/apachectl start

Previously I wrote about Configuring Capistrano and Mongrel. Now we are going to configure Apache to work with the Mongrel cluster.

For this article I used the chapter Setting Up A Development Environment in Agile Web Development with Rails, version 2.0, plus this article.

• Time For A Grown-Up Server: Rails, Mongrel, Apache, Capistrano and You

Capistrano working together with Mongrel allows you to deploy and restart Mongrel clusters quite nicely.

Configure Apache proxy balancer

Apache has a mod_proxy_balancer module which must be enabled. Once this is done you can add the following to the end of conf/httpd.conf or if you are on Red Hat Linux you can put the proxy balancing section in /etc/httpd/conf.d/myapp.proxy_cluster.conf and the virtual host section in /etc/httpd/conf.d/myapp.conf.

<Proxy balancer://mongrel_cluster>
  BalancerMember http://127.0.0.1:8000
  BalancerMember http://127.0.0.1:8001
  BalancerMember http://127.0.0.1:8002
</Proxy>

<VirtualHost *:80>
  Include conf/myapp.common (or Include conf.d/myapp.common)
  ErrorLog logs/myapp_errors_log
  CustomLog logs/myapp_log combined
</VirtualHost>

Configure Apache Virtual Host

Next you configure the virtual host that represents the Ruby on Rails application in the custom file conf/myapp.common or if you are on Red Hat Linux in /etc/httpd/conf.d/myapp.common.

ServerName myapp.com
DocumentRoot /usr/local/rails/myapp/current/public

<Directory "/usr/local/rails/myapp/current/public">
  Options FollowSymLinks
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>

RewriteEngine On

# Uncomment for rewrite debugging
#RewriteLog logs/myapp_rewrite_log
#RewriteLogLevel 9 

# Check for maintenance file and redirect all requests
RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
RewriteCond %{SCRIPT_FILENAME} !maintenance.html
RewriteRule ^.*$ /system/maintenance.html [L]

# Rewrite index to check for static
RewriteRule ^/$ /index.html [QSA] 

# Rewrite to check for Rails cached page
RewriteRule ^([^.]+)$ $1.html [QSA]

# Redirect all non-static requests to cluster
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ balancer://mongrel_cluster%{REQUEST_URI} [P,QSA,L]

Restart Apache

$ sudo /usr/local/apache2/bin/httpd -k restart

or

$ sudo /etc/init.d/httpd restart

I was wondering what the difference was between require and require_dependency.

According to this post in comp.lang.ruby the difference is not significant but interesting. Looks like I’ll be using require_dependency from now on.

require loads a file (shared object or ruby source) once from the load path.

require_dependency isn’t a ruby core feature, it’s part of rails. It remembers the given file and loads it on each new server request (like load “path/file.rb” does), if development mode is enabled:

http://wiki.rubyonrails.com/rails/pages/RequireDependency

–
Florian Frank

For this article I used the chapter Setting Up A Development Environment in Agile Web Development with Rails, version 2.0, plus these two articles.

• Time For A Grown-Up Server: Rails, Mongrel, Apache, Capistrano and You
• Using Capistrano with Rails

Capistrano working together with Mongrel allows you to deploy and restart Mongrel clusters quite nicely.

Configure Mongrel Cluster

First you need to configure a Mongrel cluster. Here is an example that creates three Mongrel instances starting at port 8000, listening on the local interface, 127.0.0.1.

$  mongrel_rails cluster::configure -N 3 -p 8000 -e production -a 127.0.0.1 \
   -c /usr/local/rails/production/current \
   -C /usr/local/rail/production/current/config/mongrel_cluster.yml

Here is how the created file looks like.

---
cwd: /usr/local/rails/production/current
log_file: log/mongrel.log
port: "8000"
environment: production
address: 127.0.0.1
pid_file: tmp/pids/mongrel.pid
servers: 3

Note that for testing purposes you should comment out the address line.

#address: 127.0.0.1

We specified listening only on the local interface for security purposes but for testing the cluster we need to access it directly via its remote IP address.

Setup Capistrano

Next you create a stub Capfile and config/deploy.rb.

$ capify .

Now you can get a list of all the tasks that are available and there quite a few.

$ cap -T

Next modify config/deploy.rb like in this example.

require 'mongrel_cluster/recipes'

set :application, "foobook"
set :repository, "http://example.com/svn/foo/trunk/foobook"

role :web, "192.168.3.17"
role :app, "192.168.3.17"
role :db,  "192.168.3.117", :primary => true

set :deploy_to, "/usr/local/rails/production"
set :mongrel_conf, "#{current_path}/config/mongrel_cluster.yml"
set :svn_user, "fkim"
set :svn_password, "fkim"
#set :user, "root"            # defaults to the currently logged in user
set :scm, :subversion         # defaults to :subversion
set :svn, "/usr/bin/svn"      # defaults to searching the PATH

Deploy using Capistrano

Now run setup which will create the directories remotely.

$ cap deploy:setup

Check dependencies.

$ cap -q deploy:check

Deploy for the first time.

$ cap deploy:cold

Everytime after you can deploy like this.

$ cap deploy

Starting and Stopping Mongrel using Capistrano

If you want to just start the mongrel cluster.

$ cap mongrel:cluster:start

If you want to just stop the mongrel cluster.

$ cap mongrel:cluster:stop

And that’s it.

Not that bad at all. Starting and stopping mongrel was the biggest issue for me. I realized that I did not need to create the spin script as suggested in the Using Capistrano with Rails article. If the mongrel configuration is correct then Capistrano will correctly start and stop it and you can use the above cap commands to test it.