about.me/v1nc3ntlaw

Sheng-Zhe Lin 林聖哲

Rails developer and linux system administrator at Techbang.

Rails Production Server

• Ubuntu (LTS) / Debian (Stable)
• Ruby Enterprise Edition (1.8.7)
• Nginx + Passenger
  passenger-install-nginx-module

Rails and Server Monitoring

New Relic

Monitor, Troubleshoot and Tune Your Production Application

Scout

Monitoring Server Cluster

Open Source Solution

Nagios + Cacti

Nagios

services current status, unusual situation or crash alert

Cacti

devices data graphing solution

• CPU Usage
  
• Disk IO
  
• Memory Usage
  

God

Process Monitoring Framework in Ruby

Nginx and Passenger Tuning

Break Limits, Unleashed Power

/opt/nginx/conf/nginx.conf

worker_processes 16;        # cat /proc/cpuinfo
worker_rlimit_nofile 32768;

events {                   # max_clients =
  worker_connections 8192; # worker_processes * worker_connections
  use epoll;
}

/etc/security/limits.conf

rails soft nofile 32768
rails hard nofile 32768

Passenger Configuration

$ passenger-memory-stats
----- Passenger processes ------
PID    VMSize    Private   Name
--------------------------------
3054   365.8 MB  150.2 MB  Rack: /home/rails/project/current

passenger_max_pool_size 20; # Private 150.2 * 20 = 3624 MB

Nginx upload module with Paperclip

moved hard work from Ruby to C

Load Balancing on Rails

HAProxy F5 BIG-IP

Rails Asset Cache

• merge multiple css or js files into one file
• decrease http requests
• render view generate when it not exist
• with cdn prefix
  http://cdn0.techbang.com.tw/all.css?1314077705

Problem with Rails Asset Cache

Preheat Rails Asset Cache: Jammit

An industrial strength asset packaging library for Rails

• support compress by YUI, Closure, or UglifyJS
• capistrano deploy hook

task :preheat_assets, :roles => [:web] do
  run "cd #{release_path} && bundle exec jammit --base-url http://www.techbang.com.tw"
end

# before symblo link to current and touch tmp/restart.txt
before "deploy:symlink", "my_tasks:preheat_assets"

Memcached Distribute and Failover

config/enviroments/production.rb

MEMCACHE_SERVERS = ["192.168.100.3:11211", "192.168.100.4:11211"]
config.cache_store = :mem_cache_store, MEMCACHE_SERVERS, {
  :namespace => "techbang"
}

Rails Backup Solution

Whenever + Backup Gems

Whenever

Cron jobs in Ruby

Linux Crontab Format
# min hour dom mon dow command

Whenever

job_type: command, rake task

# whenever config/schedule.rb
every 1.day, :at => "23:45" do
  command "echo 'Say Hello!'"
  rake "mail:notification"
end
# generated crontab
45 23 * * * /bin/bash -l -c 'echo '\''Say Hello!'\'''
45 23 * * * /bin/bash -l -c 'cd project && \
RAILS_ENV=production bundle exec rake mail:notification'

Whenever

Capistrano integration, update crontab with deploy

# config/deploy.rb
set :whenever_command, "bundle exec whenever"
require "whenever/capistrano"

Backup (RubyGem)

• Database: MySQL, PostgreSQL, MongoDB, Redis
• Backup Type: RSync, Tar archive (Gzip, Bzip2)
• Cloud Storage: S3, Dropbox, Rackspace Cloud
• Remote Servers (FTP, SFTP, SCP and RSync)
• Notifiers: Mail, Twitter, Campfire, Presently

Backup (RubyGem)

backup perform -t local_backup

Backup::Model.new(:local_backup, "Backup!") do
  backup_path = "/mnt/backup_copy/project"
  backup_list = []
  backup_list << "/home/rails/project/shared/system"
  backup_list << "/home/rails/project/shared/config"

  if File.directory? backup_path
    system("rsync -ravh --delete #{backup_list.join(" ")} #{backup_path}")
  end
end

Backup has a great wiki, explains each component in detail with many examples.

Backup Target

project/
├── current -> /home/rails/project/releases/20110819032217
├── releases
│   ├── 20110728073852
│   ├── ...
│   └── 20110819032217
└── shared
    ├── bundle
    ├── cached-copy
    ├── config => current/config/database.yml
    ├── log
    ├── pids
    └── system => current/public/system

Configuration Stay with Project

backup run at one main server only

# config/schedule.rb
host = `hostname`.chomp
if host == "main_server"
  every 6.hours do
    command "cd /home/rails/project/current && \
    AWS_CALLING_FORMAT=SUBDOMAIN backup perform -t local_backup \
    --config_file 'config/backup.rb'"
  end
end

Q & A

Thanks!