Remotely upgrading a kernel without some sort of remote KVM solution is not for the faint hearted. Realizing you forgot to include some module, or build the initrd image after remotely rebooting a kernel is even worse

Luckily there are some nice tricks you can use to make a remote kernel upgrade as safe as possible.

First, make sure you have a valid kernel. Copy the .config of an old kernel and ‘make oldconfig’ to make it work on your new-to-be-compiled kernel.

Second, make the kernel using the ‘make-kpkg’ package, don’t forget the –initrd parameter if you need it.

Third, after installing the kernel package, don’t forget to check if the initrd image got generated. For some strange reason Debian won’t generate this even after building with make-kpkg –image, you can read more about that here. You basically need to copy a file from ‘/usr/share/kernel-package/examples/etc/kernel/postinst.d/initramfs’ to ‘/etc/kernel/postinst.d/initramfs’

Fourth, you need to configure Grub2 so it uses a kernel boot option which makes the kernel restart if it can’t boot for some reason. To do this with Grub2, open /etc/default/grub and add “panic=5″ to GRUB_CMDLINE_LINUX_DEFAULT, like this:

GRUB_CMDLINE_LINUX_DEFAULT=”quiet panic=5″

Fifth, Grub2 needs to be told to only boot the new kernel once, so in case the new kernel panics, it automatically restarts using a kernel you know works (your current one perhaps). Else the ‘panic=5′ we set earlier will only lead to a reboot loop.

To do this open /etc/default/grub and change GRUB_DEFAULT to saved, like this:

GRUB_DEFAULT=saved

To configure ‘reboot-once-into-new-kernel’, we need to do two more things, first we need to tell Grub2 what your working, safe kernel is. Check /boot/grub/grub.cfg for the name of this kernel, but don’t change anything in this file.

My safe kernel is called: ”Debian GNU/Linux, with Linux 2.6.32-2-686″, which I can set as the default, safe, kernel with this command:

grub-set-default ”Debian GNU/Linux, with Linux 2.6.32-2-686″

Now we have to tell Grub to reboot once into the new kernel, again we need the name of the new kernel, mine is ”Debian GNU/Linux, with Linux 2.6.32-example”, use this command.

grub-reboot ”Debian GNU/Linux, with Linux 2.6.32-example”

Don’t worry, this won’t reboot your machine.

The final step is to generate a new /boot/grub/grub.cfg, else all the things we just did won’t have an effect

update-grub

Now you’re ready to reboot, good luck and fingers crossed. After a succesful boot, you can set your new kernel as the default using the grub-set-default command

grub-set-default ”Debian GNU/Linux, with Linux 2.6.32-example”

A week ago I got a new phone, the HTC Hero.

It’s an Android based phone so it’s easy to customize and there are a lot of great apps available for it.

However, HTC’s default firmware is a bit sluggish. Luckily there’s a solution. It takes 5 minutes and besides speeding it up, it also ‘roots’ it. ‘Rooting’ an Android phone allows a user to do basically anything he wants with the phone.

Here are the steps (by Paul of MoDaCo)

- Download the custom ROM, and copy it to your SD card (modaco 2.2 core)
- download this file (http://content.modaco.net…copatchedrecovery-1.0.zip) and extract it to your PC or mac
- open a command prompt to the directory containing the extracted files
- type the following commands (prefix each command with ./ if you’re on a mac)
- adb shell reboot bootloader (wait for device to reboot into bootloader)
- fastboot boot cm-hero-recovery.img
- Select ‘nandroid backup’ from the menu to do a backup
- If you copied the custom rom as update.zip, select ‘apply sdcard:update.zip’, else select the option below.
- all done

I’ve been struggling for quite some time trying to get 1080p content to play smoothly on my MythTV pc. I managed to get good playback of 720p files by compiling mplayer myself and using a recent version of the closed source nvidia driver.

Playing back 1080p content however, was still a problem. I was never able to get smooth playback with mplayer. Only the xbmc videoplayer could play 1080p smoothly on my system. XBMC is not an option due to the WAF, it has to be something that runs from MythTV.

Fooling around with coreavc-for-linux (I love CoreAVC on Windows) got me nowhere. Recompiling, tweaking and updating mplayer/nvidia/xorg, nothing seemed to work.

Until I tried a new version of mplayer from the Debian Multimedia repositories. The mplayer version in that repository has support for a new feature in the linux nvidia drivers that (finally) allows video decoding on the GPU.

The easy steps to get this to work:

• Download and install the new nvidia linux drivers (180+)
• Add the Debian Multimedia repository to your sources.list
• Install mplayer from the new Debian Multimedia repository
• Start mplayer with ‘-vo vdpau -vc ffh264vdpau’

More than a year of tweaking and tinkering made redundant by these easy steps, excellent!

Just a quick post to maybe help someone who has a similar problem:

I had the following error whenever I tried to use ’script/console’ (script/server and rake worked fine)

require ‘mysql’

MissingSourceFile: no such file to load — mysql
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:27:in `gem_original_require’
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:27:in `require’

Turns out I had multiple versions of RubyGems installed on my Macbook. The error message already spoils it by saying rubygems in /System/Library/Framework/ is being called, while I actually installed rubygems manually in a different folder.

To prevent these kind of problems in the future I simply deleted the entire Ruby.framework directory and use the manually installed versions instead.

I’ve been using my new Macbook for a whopping two days now, so I think it’s time to share some of my early impressions.

First of all, I’m not a typical computer user. I’m picky, opinionated and totally unreasonable when it comes to my requirements for an Operating System. Before my Macbook my setup consisted of:

• Desktop PC, Windows XP: Used for gaming mainly, also editting photo’s and as a ‘fat client‘ for Putty
• Laptop, Ubuntu: Used for work and midnight-coding. Purely a development tool for developing Ruby on Rails webapplications, accessing icanhazcheeseburger.com and other important work stuff.

Read the rest of this entry »

Online (previously Wanadoo/Orange) is currently testing a new modem to replace the ‘Livebox’. The new modem is a Speedtouch 706 (WL) and to prevent people from accidently breaking it or use the modem on different DSL networks nearly all administrative options are disabled.

Unlike the old Livebox, you only get limited access to forward ports and change the wifi SSID and password. Even DynDNS support, which was perfectly usable in the Livebox, is disabled.

Looking for a way to disable the built-in DHCP server and change the default IP address of the Speedtouch 706 I tried flashing the device with the original firmware instead of the locked one. However, the firmware updater refuses to flash because of incompatibility.

This didn’t stop me and using TFTP I tried forcing the flash, but again, this didn’t work.

After some googling I came across a vulnerability in the Speedtouch 780, that allows you to access any page of the webinterface, even the ones you shouldn’t have access to. Using this vulnerability it’s possible to download the modem’s configuration, change it in a text editor and upload it again.

Thanks to this vulnerability I was able to disable the DHCP server and change the symmetric NAT implementation to cone NAT. The default configuration makes it nearly impossible to play online games based on a player-to-player architecture. C&C3 and Supreme Commander are two games that use this technique to allow players to play against eachother. With the original configuration I was unable to connect to about 80% of players in Supreme Commander, now with the cone NAT configuration I no longer have any issues.

You can find instructions to download/upload the Speedtouch configuration here and you can find/replace these bits in the user.ini to get cone NAT instead of symmetric NAT:

connection bind application=CONE(UDP) port=0

connection appconfig application=CONE(UDP) timeout=0

ids config state disabled

With the recent release of mod_rails, I thought it was time for yet another benchmark. Mod_rails aims to make deployment of RubyOnRails applications comparable to deploying a PHP application on a shared host.

The contestants

mod_rails (Apache 2.2.8)

Thin 0.8.0 (Cluster of 4 behind nginx 0.5.33, both TCP and Unix socket)

Ebb 0.1.0 (Cluster of 4 behind nginx 0.5.33)

Mongrel 1.1.4 (Cluster of 4 behind nginx 0.5.33)

The benchmark is the index page of Leef je doel, the hardware used is a Dell D830 laptop with a T7300 (speedstep disabled), 2GB of RAM and a 7200RPM harddisk. The page runs a few queries and also serves a lot of fragments from memcached.

The results

Performance

Like my previous test, Ebb comes out on top, closely followed by Thin.

Using the default Apache2 config supplied by Ubuntu, mod_rails shows performance equal to mongrel_cluster behind nginx. I’m sure that by tweaking the Apache config files, some performance can be gained.

Compared to setting up nginx and thin/ebb/mongrel, setting up mod_rails is a lot simpler. Have a look at their screencast to see how easy it is, if you have a working Apache2, you’ll have mod_rails up and running within minutes. There are some small issues with mod_rails at the moment, for a list, check mod_rails’ Google Code page.

Like any web 2.0 site, leefjedoel.nl is currently in beta. During this phase we’re trying to find bottlenecks, fix the last few bugs and optimize heavy parts of the site.

During development we already prepared caching for all pages, mostly fragment caching. To expire cache that’s no longer current, we use sweepers that get called when something relevant is updated. These sweepers only sweep the caches that get outdated.

Because we were unfamiliar with caching and needed to see the result of our fragment caching, we used the file_store to store the generated caches. These files are stored on disk and this way you can easily see how many cache gets generated and what they contain.

Regex and file_store == FAIL

To sweep caches we used regular expressions, this way we could easily sweep all relevant caches at once. This was a bad idea, as you can read here and here

During the beta phase the size of the site steadily increased, more users, more goals, more groups. There was a noticeable delay whenever you updated/created something. A short investigation pointed to the cache sweepers as the culprit.

The file_store for cache isn’t exactly the fastest solution to store your cache, but when you combine it with regex sweepers, things really slow down. Whenever you do a regex sweep, all files in the cache directory get returned (not that surprising if you think of it), and the regex is run against it. So even if you do a sweep on /goals, it will also return files in /users and /groups. As a result, updating your profile could take 15 seconds.

MemCached

We’d been planning on moving the cache to memcached all along, so this seemed a good opportunity to do it. In the next few paragraphs I’ll describe how to install memcached, get the correct Rails plugin to make memcached play nice with fragment caching and how to configure your Rails application so it uses your memcache server.

Installation of MemCached on GNU/Linux

First of all, you’ll need to memcache daemon, assuming you have a nice linux distro you can:

sudo apt-get install memcached

This will work on Debian, Ubuntu and other Debian-based distro’s, on Gentoo you can

emerge memcached

The great thing about memcached is its simplicity, it requires no configuration after installation, just run it.

All about the gems baby

Now we’ll get the gem to allow Ruby to talk to memcache. There are two gems that do this, Ruby-MemCache and memcache-client. memcache-client is supposed to be faster, so I used that.

sudo gem install memcache-client

Plugin to play nice with rails

Rails’ fragment caching doesn’t work with memcached out of the box, you’ll need a plugin. This plugin also adds a nice bonus to the cache method in views.

script/plugin install svn://rubyforge.org/var/svn/zventstools/projects/extended_fragment_cache

Now we need to configure your Rails app to use the memcached server. You’ll need to edit your config/environments/production.rb

memcache_options = {
:c_threshold = 10_00,
:compression = true,
:debug = false,
:namespace = 'yourappname_or_anything_you_like',
:readonly = false,
:urlencode = false
}
CACHE = MemCache.new(memcache_options)
CACHE.servers = '127.0.0.1:11211'
config.action_controller.fragment_cache_store = CACHE, {}

That’s all folks!

That’s it, you’re all done, the ‘cache’ method in your views will now use the memcache server.

Oh wait, there’s an encore

But there’s more, using memcached you can set expiry times for caches. I edited the plugin for a default expiry time of 1 day. In vendor/plugins/extended_fragment_cache/lib/extended_fragment_cache.rb look for def write(key,content,options=nil)

def write(key,content,options=nil)
  expiry = options && options[:expire] || 1.day
  begin
    set(key,content,expiry)
    rescue ActiveRecord::Base.logger.error("MemCache Error: #{$!}")
    rescue MemCache::MemCacheError = err
    ActiveRecord::Base.logger.error("MemCache Error: #{$!}")
  end
end

You can change the 1.day to anything you want. To override this default behaviour, you can use the following code in your views

cache('goals/large_cloud', {:expire = 30.minutes.to_i})  do

This will make the cache called ‘goals/large_cloud’ to expire 30 minutes after it got created.

There are two important things to consider when you move to memcached

1. MemCached doesn’t support regex based expiry of caches. You need to manually enter every cache you want to expire. You can do this in some nice methods of course. Here’s ours for expiring the cache when a user gets updated.

def expire_user_fragments(user)
  fragments = %w[author_icon author_link side_block_friends ..snip...]
  fragments.each do |f|
    expire_fragment("user/#{user.id}/#{f}")
  end
end

2. Your application will fail when the MemCache server becomes unavailable. If you ever restart MemCache, or if it crashes (haven’t seen that happen yet), you need to restart your mongrel-cluster/thin/ebb.

3. When you restart MemCache, all cache is cleared, and you need to restart your mongrel-cluster/thin/ebb.

This guide only talks about fragment caching, over at Ben Curtis’ blog, you can read all about action caching.

Ruby Inside just posted a pretty impressive performance graph of a new Ruby server called Ebb. The graph was taken from the homepage of Ebb

Now what is Ebb?

The design is similar to the Evented Mongrel web server; except instead of using EventMachine (a ruby binding to libevent), the Ebb web server is written in C and uses the libev event loop library.

Connections are processed as follows:

1. libev loops and waits for incoming connections.
2. When Ebb receives a connection, it passes the request into the mongrel state machine which securely parses the headers.
3. When the request is complete, Ebb passes the information to a user supplied callback.
4. The Ruby binding supplying this callback transforms the request into a Rack compatible env hash and passes it on a Rack adapter.

The graph describes performance of a simple rack application and compares Ebb to mongrel, evented mongrel and thin. I’m more interested in performance with a Rails application and decided to do a benchmark for that.

In my benchmark I used the same application I used for my previous benchmark, only this time I benchmarked some extra pages.

Page 1 is a heavily cached page with few dynamic elements

Page 2 is a less cached page with a bit more dynamic elements

Page 3 is a non-cached page with an N+1 performance issue.

Ebb was tested using version 0.0.3, while Thin was on version 0.7.0. Both were run in a cluster of 4 behing nginx as a load balancer.

Interestingly Ebb managed to outperform Thin by about 10% on every page.

I’m currently helping a colleague to build a rather large community website using Ruby on Rails. As most Rails developers are well aware, Ruby isn’t exactly the quickest language you can use to build web applications. To get a rough idea of how much users our current code would handle, I decided to run some performance tests.

First, here’s a picture of the page that was used.

The page has a tagcloud, site statistics and a conditional menu in the upper right corner.

For the benchmark I’m using the following tools

ApacheBench 2.0.40. To fire off a large amount of requests into the webserver.

Mongrel 1.1.3. One of the Ruby web servers

Thin 0.6.4. An alternative to Mongrel.

nginx 0.5.26. A lightweight HTTP server, which will act as a proxy/balancer in front of the Ruby application servers

For the benchmarks I settled on the following scenario’s.

Single instance of Mongrel in development mode

Single instance of Mongrel in production mode

Cluster of 4 Mongrel instances behind nginx

Cluster of 4 Thin instances behind nginx

In a real production environment, you’d never see the single Mongrel instance, and certainly not in development mode. The graph will show you why.

The red bar is for 50 concurrent requests, with a total of 10000. The red blue is for 10 concurrent requests, with a total of 10000.

The raw numbers:

I’m pleased with the results, we still have a lot of stuff we could cache and database queries that could be optimized, so there’s room for improvement. The server that will eventually host the application runs an Apache proxy going to mongrel_cluster. I’m probably going to run some benchmarks later, to see how Apache proxy holds up against nginx.