Remotely upgrading a kernel without some sort of remote KVM solution is not for the faint hearted. Realizing you forgot to include some module, or build the initrd image after remotely rebooting a kernel is even worse

Luckily there are some nice tricks you can use to make a remote kernel upgrade as safe as possible.

First, make sure you have a valid kernel. Copy the .config of an old kernel and ‘make oldconfig’ to make it work on your new-to-be-compiled kernel.

Second, make the kernel using the ‘make-kpkg’ package, don’t forget the –initrd parameter if you need it.

Third, after installing the kernel package, don’t forget to check if the initrd image got generated. For some strange reason Debian won’t generate this even after building with make-kpkg –image, you can read more about that here. You basically need to copy a file from ‘/usr/share/kernel-package/examples/etc/kernel/postinst.d/initramfs’ to ‘/etc/kernel/postinst.d/initramfs’

Fourth, you need to configure Grub2 so it uses a kernel boot option which makes the kernel restart if it can’t boot for some reason. To do this with Grub2, open /etc/default/grub and add “panic=5″ to GRUB_CMDLINE_LINUX_DEFAULT, like this:

GRUB_CMDLINE_LINUX_DEFAULT=”quiet panic=5″

Fifth, Grub2 needs to be told to only boot the new kernel once, so in case the new kernel panics, it automatically restarts using a kernel you know works (your current one perhaps). Else the ‘panic=5′ we set earlier will only lead to a reboot loop.

To do this open /etc/default/grub and change GRUB_DEFAULT to saved, like this:

GRUB_DEFAULT=saved

To configure ‘reboot-once-into-new-kernel’, we need to do two more things, first we need to tell Grub2 what your working, safe kernel is. Check /boot/grub/grub.cfg for the name of this kernel, but don’t change anything in this file.

My safe kernel is called: ”Debian GNU/Linux, with Linux 2.6.32-2-686″, which I can set as the default, safe, kernel with this command:

grub-set-default ”Debian GNU/Linux, with Linux 2.6.32-2-686″

Now we have to tell Grub to reboot once into the new kernel, again we need the name of the new kernel, mine is ”Debian GNU/Linux, with Linux 2.6.32-example”, use this command.

grub-reboot ”Debian GNU/Linux, with Linux 2.6.32-example”

Don’t worry, this won’t reboot your machine.

The final step is to generate a new /boot/grub/grub.cfg, else all the things we just did won’t have an effect

update-grub

Now you’re ready to reboot, good luck and fingers crossed. After a succesful boot, you can set your new kernel as the default using the grub-set-default command

grub-set-default ”Debian GNU/Linux, with Linux 2.6.32-example”

I’ve been struggling for quite some time trying to get 1080p content to play smoothly on my MythTV pc. I managed to get good playback of 720p files by compiling mplayer myself and using a recent version of the closed source nvidia driver.

Playing back 1080p content however, was still a problem. I was never able to get smooth playback with mplayer. Only the xbmc videoplayer could play 1080p smoothly on my system. XBMC is not an option due to the WAF, it has to be something that runs from MythTV.

Fooling around with coreavc-for-linux (I love CoreAVC on Windows) got me nowhere. Recompiling, tweaking and updating mplayer/nvidia/xorg, nothing seemed to work.

Until I tried a new version of mplayer from the Debian Multimedia repositories. The mplayer version in that repository has support for a new feature in the linux nvidia drivers that (finally) allows video decoding on the GPU.

The easy steps to get this to work:

• Download and install the new nvidia linux drivers (180+)
• Add the Debian Multimedia repository to your sources.list
• Install mplayer from the new Debian Multimedia repository
• Start mplayer with ‘-vo vdpau -vc ffh264vdpau’

More than a year of tweaking and tinkering made redundant by these easy steps, excellent!

I’ve been using my new Macbook for a whopping two days now, so I think it’s time to share some of my early impressions.

First of all, I’m not a typical computer user. I’m picky, opinionated and totally unreasonable when it comes to my requirements for an Operating System. Before my Macbook my setup consisted of:

• Desktop PC, Windows XP: Used for gaming mainly, also editting photo’s and as a ‘fat client‘ for Putty
• Laptop, Ubuntu: Used for work and midnight-coding. Purely a development tool for developing Ruby on Rails webapplications, accessing icanhazcheeseburger.com and other important work stuff.

Read the rest of this entry »

With the recent release of mod_rails, I thought it was time for yet another benchmark. Mod_rails aims to make deployment of RubyOnRails applications comparable to deploying a PHP application on a shared host.

The contestants

mod_rails (Apache 2.2.8)

Thin 0.8.0 (Cluster of 4 behind nginx 0.5.33, both TCP and Unix socket)

Ebb 0.1.0 (Cluster of 4 behind nginx 0.5.33)

Mongrel 1.1.4 (Cluster of 4 behind nginx 0.5.33)

The benchmark is the index page of Leef je doel, the hardware used is a Dell D830 laptop with a T7300 (speedstep disabled), 2GB of RAM and a 7200RPM harddisk. The page runs a few queries and also serves a lot of fragments from memcached.

The results

Performance

Like my previous test, Ebb comes out on top, closely followed by Thin.

Using the default Apache2 config supplied by Ubuntu, mod_rails shows performance equal to mongrel_cluster behind nginx. I’m sure that by tweaking the Apache config files, some performance can be gained.

Compared to setting up nginx and thin/ebb/mongrel, setting up mod_rails is a lot simpler. Have a look at their screencast to see how easy it is, if you have a working Apache2, you’ll have mod_rails up and running within minutes. There are some small issues with mod_rails at the moment, for a list, check mod_rails’ Google Code page.

Like any web 2.0 site, leefjedoel.nl is currently in beta. During this phase we’re trying to find bottlenecks, fix the last few bugs and optimize heavy parts of the site.

During development we already prepared caching for all pages, mostly fragment caching. To expire cache that’s no longer current, we use sweepers that get called when something relevant is updated. These sweepers only sweep the caches that get outdated.

Because we were unfamiliar with caching and needed to see the result of our fragment caching, we used the file_store to store the generated caches. These files are stored on disk and this way you can easily see how many cache gets generated and what they contain.

Regex and file_store == FAIL

To sweep caches we used regular expressions, this way we could easily sweep all relevant caches at once. This was a bad idea, as you can read here and here

During the beta phase the size of the site steadily increased, more users, more goals, more groups. There was a noticeable delay whenever you updated/created something. A short investigation pointed to the cache sweepers as the culprit.

The file_store for cache isn’t exactly the fastest solution to store your cache, but when you combine it with regex sweepers, things really slow down. Whenever you do a regex sweep, all files in the cache directory get returned (not that surprising if you think of it), and the regex is run against it. So even if you do a sweep on /goals, it will also return files in /users and /groups. As a result, updating your profile could take 15 seconds.

MemCached

We’d been planning on moving the cache to memcached all along, so this seemed a good opportunity to do it. In the next few paragraphs I’ll describe how to install memcached, get the correct Rails plugin to make memcached play nice with fragment caching and how to configure your Rails application so it uses your memcache server.

Installation of MemCached on GNU/Linux

First of all, you’ll need to memcache daemon, assuming you have a nice linux distro you can:

sudo apt-get install memcached

This will work on Debian, Ubuntu and other Debian-based distro’s, on Gentoo you can

emerge memcached

The great thing about memcached is its simplicity, it requires no configuration after installation, just run it.

All about the gems baby

Now we’ll get the gem to allow Ruby to talk to memcache. There are two gems that do this, Ruby-MemCache and memcache-client. memcache-client is supposed to be faster, so I used that.

sudo gem install memcache-client

Plugin to play nice with rails

Rails’ fragment caching doesn’t work with memcached out of the box, you’ll need a plugin. This plugin also adds a nice bonus to the cache method in views.

script/plugin install svn://rubyforge.org/var/svn/zventstools/projects/extended_fragment_cache

Now we need to configure your Rails app to use the memcached server. You’ll need to edit your config/environments/production.rb

memcache_options = {
:c_threshold = 10_00,
:compression = true,
:debug = false,
:namespace = 'yourappname_or_anything_you_like',
:readonly = false,
:urlencode = false
}
CACHE = MemCache.new(memcache_options)
CACHE.servers = '127.0.0.1:11211'
config.action_controller.fragment_cache_store = CACHE, {}

That’s all folks!

That’s it, you’re all done, the ‘cache’ method in your views will now use the memcache server.

Oh wait, there’s an encore

But there’s more, using memcached you can set expiry times for caches. I edited the plugin for a default expiry time of 1 day. In vendor/plugins/extended_fragment_cache/lib/extended_fragment_cache.rb look for def write(key,content,options=nil)

def write(key,content,options=nil)
  expiry = options && options[:expire] || 1.day
  begin
    set(key,content,expiry)
    rescue ActiveRecord::Base.logger.error("MemCache Error: #{$!}")
    rescue MemCache::MemCacheError = err
    ActiveRecord::Base.logger.error("MemCache Error: #{$!}")
  end
end

You can change the 1.day to anything you want. To override this default behaviour, you can use the following code in your views

cache('goals/large_cloud', {:expire = 30.minutes.to_i})  do

This will make the cache called ‘goals/large_cloud’ to expire 30 minutes after it got created.

There are two important things to consider when you move to memcached

1. MemCached doesn’t support regex based expiry of caches. You need to manually enter every cache you want to expire. You can do this in some nice methods of course. Here’s ours for expiring the cache when a user gets updated.

def expire_user_fragments(user)
  fragments = %w[author_icon author_link side_block_friends ..snip...]
  fragments.each do |f|
    expire_fragment("user/#{user.id}/#{f}")
  end
end

2. Your application will fail when the MemCache server becomes unavailable. If you ever restart MemCache, or if it crashes (haven’t seen that happen yet), you need to restart your mongrel-cluster/thin/ebb.

3. When you restart MemCache, all cache is cleared, and you need to restart your mongrel-cluster/thin/ebb.

This guide only talks about fragment caching, over at Ben Curtis’ blog, you can read all about action caching.

I’ve been running torrents on my server for quite a while now. I used to run Azureus on my desktop, but because it’s Java-based, it was quite heavy on memory.

To run torrents on my server I used Torrentflux. It’s a PHP-webinterface to BitTornado and it has a nice community and forum where they regularly post hacks to add new features. After a while I found out there was a different version (fork) of Torrentflux with more features and I switched.

TF-B4rt

This fork of Torrentflux is called Torrentflux-b4rt. It adds a bunch of new features, including some improvements to the interface and support for multiple bittorrent clients. TF-b4rt supports BitTornado, Mainline, Transmission and Azureus.
Azureus support is provided through a piece of middleman software called Fluazu, which talks in XML to Azureus. This (and being written in Python) causes Fluazu to use quite a bit of CPU cycles. About 10-30% on my 2.13Ghz Core2Duo server. It’s also a rather delicate system, if you manually delete a torrent through the Azureus-GUI/console, or disk, the webinterface starts to bug. It will show torrents running that aren’t, and you can’t delete them because it’s convinced they’re still active. But even if I didn’t do something like that, Fluazu got confused with my torrents.

The main reason I wanted to use Azureus (and thus, Fluazu), is the support for global download/upload rate-limits. However, the extra CPU usage for Fluazu was bothering me and I looked for alternatives.

Azureus Web Interface

Azureus has it’s own web interface. Two actually. The HTML Web-UI, and the Swing Web-UI. I tried the Swing Web-UI and it works rather well. You upload torrents, Azureus downloads them. Because it talks directly to Azureus it’s more reliable than Torrentflux-Fluazu.

Installing the Swing Web-UI breaks support for TF-B4rt/Fluazu.

Clutch

Another webinterface is called Clutch. It’s a webinterface that talks to the Tranmisson daemon. Transmission is a fast light-weight torrent client. Using the daemon version, you can have multiple uploads with a global upload/download limit.

After loading about 20 torrents in Transmission, I was using about 1% memory, while Azureus used 9.5%. CPU usage for Azureus was 2-4%, while Transmission rarely got above 1%.

Torrentflux-b4rt provides the most functuonality. It has some features you can’t find in the other two. For example you can enter RSS-feeds for torrents that get downloaded automatically and you can upload multiple files at a time. Also you can stop/start/delete all torrents and data at once. It also supports multiple users, multiple torrent-clients and has more extensive statistics. Because of all these extra feature, Torrentflux is also used in hosted solutions.

Unfortunately it doesn’t support global upload/download limits on its own. It needs the Fluazu/Azureus combination for this. And using this combination is rather CPU intensive. So all in all, Torrentflux-b4rt is a great solution if you don’t need the global upload/download limits.

The Azureus Web Interface is very nice if you’re a single user. The Azureus bittorrent client has a lot of features so having a web interface to this powerful client is great. It’s quite a bit more resource intensive compared to BitTornado or Transmission, but it does feature advanced features like the distributed tracker, which allows you to download torrents of which the original tracker is down.

Transmission+Clutch has a very clean interface and turned out to be the best choice for me. It doesn’t have all the advanced features of Torrentflux, but it does the basics just right. It requires few resources, allows you to set global upload/download rates and it has a feature to start/stop all torrents at once.

Today I rediscovered screenlets for Ubuntu. If you don’t know what screenlets are, click on the thumbnail on the right.

On my old laptop I tried using them before, but the poor thing couldn’t handle it.

Now with my new laptop I decided to give it another go.

Using compiz-fusion you can put these screenlets on a special ‘widget layer’, which you can summon by pressing a keyboard key. Whenever you need some information or need to make a small note, you press that keyboard key and the widget layer is displayed. Pressing the same key again and the widgets disappear.

There’s a nice screenlet that displays some sensor information, CPU and HDD temperatures. However, my laptop has another sensor on the graphics card that shows the GPU temperature. I just had to have this added to the screenlet.

I made a modified version of the Temperature2 script to generate a screenlet for nvidia GPU temperature. You’ll need the proprietary nvidia drivers installed and access to the ‘nvidia-settings’ utility. Download the file and place it in the ~/.screenlets/Temperature2/ directory, overwriting the old file.

Update: My modifications are now included into version 0.4 of Temperature2. You can get it at screenlets.org

In a couple of days/weeks/months two friends and I will be giving a workshop at our school about Ruby On Rails. For this workshop we were looking for a bootable CD that provides a fully functioning Ruby on Rails dev environment.

We looked at RailsLiveCD.org but it wasn’t exactly the way we wanted. And whenever something isn’t exactly as you want in the world of OSS, you can fix it.

It took a great tutorial, some trial/error and a couple of failed CD’s, but finally I managed to get it just the way I wanted.

The failed CD’s were due to the fact that ‘qemu’, an emulation program that you can use to test bootable .iso files, ran very slowly on my laptop.

The live CD is based on Ubuntu Feisty. It includes MySQL, vim-ruby, Aptana, ruby, rails and the sample project we’ll be using at the workshop.

No other live cd would have our own project on the disk

I’ll post the ISO once we’ve given the workshop.

Update

You can now download the Ubuntu on Rails iso.

My previous post about getting the temperatures of disks connected to an Areca raid array already noted that MRTG can only handle two datasources in a single graph.

With 5 disks in a system, this would force me to create 3 graphs, or 1 graph with the temperatures of the 4 disks connected to the RAID array averaged into a single line.

I’ve always wanted to move from MRTG to RRDtool, because RRDtool can handle multiple datasources per graph, and has more options.

However, I found RRDtool very hard to configure compared to MRTG. So I went looking for an alternative.

Cacti is a front-end for RRDtool, which makes it easier to create nice graphs. If you just want to graph routers, it’s even easier to configure than MRTG. But if you like to have custom stats, like the Areca HDD temperatures, it takes a little more work.
Cacti has some nice guides on how to create graphs and new input methods on their site, but even with those it took me quite some time to create my first graph.

By now I’ve migrated all my graphs from my personal MRTG page to Cacti

My Ubuntu laptop was running quite hot because of the built-in Mobility Radeon 9700. After a while I found out about a little utility called ‘aticonfig’ that you can use to change the powerstate of your videocard.
Using this utility I was able to clock down the videocard and enjoy a cooler laptop.

Recently I re-enabled all desktop eyecandy by using the ‘compiz-fusion‘ project. I use compiz-fusion in combination with the closed source aTi fglrx driver. However, my laptop began running hot again. At first I thought this was because of all the 3D eyecandy that uses your videocard.

After a while I decided to check the powerstate of my videocard using the ‘aticonfig –lsp’ command and got the following error:

Error: Unable to obtain POWERplay information.

After some investigation I found out that the ‘aticonfig’ utility cannot be run from X display :1. Thanks to aTi’s craptastic linux drivers you’re forced to run 3D-eyecandy desktops on display :1 instead of display :0. So the solution was simple, kind of. I wrote the following snippet that can be run from display :1

DISPLAY=:0

aticonfig –set-powerstate=1

DISPLAY=:1

Put this in a small script and make it execute everytime you start your X session ( System, Prefences, Sessions). Hopefully aTi manages to catch up with Nvidia’s linux driver support and keep their promise about better drivers.