Skip to main content.

How to install and boot Linux Debian on Flash storage using the JFFS2 file system.

There are already projects on that matter, like RootSync. But these currently seem to be in an early stage. So I tried out what is possible with existing Linux distributions. Here are the results:

Introduction

Compact Flash drives are, compared to a hard disk, extremely low power devices. Of course, the price for a certain amount of storage is still much more expensive on a Compact Flash drive than on a hard disk. But if you don't need the capacity of a hard disk, you can save some energy (and money[1]) by putting all data including the operating system on a Compact Flash. The idea of the following project is to build a really low power workstation or server which would boot a standard Linux off a Compact Flash.

With an adapter, a Compact Flash can be used as a replacement for an IDE disk. However, the Flash technology allows only a limited number of write cycles for each memory cell, and most file systems have "hot spots" which are written very frequently, which could render a flash drive unusable within short time. Therefore a technique called wear levelling is required, to distribute write access accross the medium. Rumors say that this is done in the controller of the Compact Flash Cards, but I have seen this only explicitely noted in the product description for industrial grade Compact Flash cards. I am really not sure whether cheap consumer grade Compact Flash Cards also perform wear levelling. The Compact Flash Association does not mention wear levelling. There is only a "defect managment and error correction", which, i.m.o. does not prevent defects, but merely replace bad areas after they have become faulty. When all spare sectors are exhausted, the capability of the drive to replace bad blocks eventually comes to an end, in which case the drive is defect despite of "defect management and error correction"[2]. In order to use consumer grade cards safely, the wear levelling task can be done in software. This is where a file system like JFFS2 comes into play.

There are a couple of problems, though.

  • JFFS2 is designed to work only with Memory Technology Devices.

  • Using JFFS2 on top of Compact Flash drives is generally not recommended.

  • There is no obvious way to install Linux directly into a JFFS2 partition.

To overcome the first issue, a block2mtd driver is available.

The second point is something to think about. Adding yet another layer on top of an IDE emulation on top of a flash memory storage does not increase performance. If performance is an issue (and money is not), stop reading here. Buy an industrial grade Compact Flash instead and proceed normally.

The third issue is what the remainder of this article is about. This might also be useful for really embedded solutions where flash memory is directly addressable by the CPU and does not have its own logic.

So if you want to go ahead with me, this is what you can expect:

  • JFFS2 adds the benefit of implicit data compression: expect the card to hold roughly double its nominal capacity.

  • You can use Compact Flash drives off the shelf. We will leave them formatted as FAT drives. The Linux file system will be inside one or more container files. If you like, you can attach the Compact Flash to any Windows box and copy these files for backup.

  • We will have to customize the Linux boot procedure. This will give you some insights of what is going on at a stage where most users see only a progress bar.

My favourite Linux distribution is Debian, and the following description applies to Debian Etch. However the general principle on making Linux boot a JFFS2 drive can certainly be adapted for any other Linux distribution.

Installing Debian

As mentioned above, there is no obvious way to install Linux directly into a JFFS2 partition. So we don't.

Instead, we install Linux into a separate drive, which we attach temporarly to the target. An USB disk is fine, even if the target does not know how to boot USB disks. We won't boot that installation directly. So just go ahead and install Linux onto that drive. I prefer the netinst CD, but this requires a fast internet connection.

[Note] Note

Running the install CD with install priority=medium gives you more choices. Especially, you can choose a more recent kernel. However, you can update your kernel later anyway.

[Note] Note

If your low-power system happens to be VIA C3 based, don't use a i686 kernel. The C3 will not run i686 code.

Install the Compact Flash in the location where you will later boot it. This might be an adapter which makes it an IDE drive. Compact Flash drives come preformatted as FAT drives. This is OK. Copy the kernel and the initial ram disk from your fresh Linux installation to the root directory of the compact flash. Name them vmlinuz and initrd.img, respectively.

Preparing the flash drive for booting

We will boot the flash using Syslinux. If you can't boot your newly installed Linux yet, you can use an already working Linux (a Knoppix system running from CD is fine) to install Syslinux on the Compact Flash. Create a configuration file syslinux.cfg on the root of the Compact Flash as follows:

default vmlinuz
      append  root=/dev/sda1 initrd=initrd.img

Replace /dev/sda1 with the path to your (temporary) installation disk.

Install a master boot record on the flash. This can be done with the command install-mbr. If you are using Knoppix to prepare your flash disk, this command will be already available. On a vanilla Debian, you will probably have to install the package mbr first.

You should now be able to boot your new Linux installation, using the Compact Flash merely as an initial boot medium.

[Note] Note

If you update your kernel, don't forget to update the copies of the kernel and the initial ram disk on the Compact Flash too.

We will later extend the boot configuration to allow to boot the system we are now installing on the flash itself.

Create the Flash image(s)

Install the package mtd-tools. This will provide mkfs.jffs2.

Create an image of a size that suits your Compact Flash (and the expected final size of your Linux system). Use a command like the following:

mkfs.jffs2 -d empty --eraseblock=65536 --pad=nnnnnnnnnn -o rootfs.img

Where empty is the name of an empty directory. We will add files later. See below. The eraseblock parameter should match the actual technology used on the drive - which we usually don't know in case of a Compact Flash drive. 64k might be a useful guess, but probably the setting will not be optimal. You have been warned. The pad parameter specifies the actual size of the image. In bytes. For example a 800MB container (fitting nicely in a 1GB drive) has a size of 838860800. The size should be a multiple of the eraseblock size (otherwise mkfs.jffs2 will round it to an apropriate value). The output parameter should point to your mounted Compact Flash drive.

If you want to keep parts of the file name space like /usr and /var on separate containers, you are free to to so. Just repeat the step above with other file names and sizes for the additional containers. I.m.o this only complicates things, but it is possible.

Mount the Flash image

Setup loop device(s)

losetup /dev/loop0 /media/hda1/rootfs.img

Where /media/hda1/rootfs.img is the path where you mounted your Compact Flash drive.

Create the block2mtd device(s)

mknod /dev/mtdblock0 b 31 0

If your setup requires more than one image, repeat with increasing index numbers (minor number)

Load the block2mtd driver

modprobe block2mtd block2mtd=/dev/loop0,65536
[Note] Note

The size parameter (65536) must match the eraseblock parameter in mkfs.jffs2 above.

If your setup requires more than one image, specifiy the remaining loop devices with additional block2mtd=xxx parameters:

modprobe block2mtd block2mtd=/dev/loop0,65536 block2mtd=/dev/loop1,65536 block2mtd=/dev/loop2 ...

Mount the image(s)

mount -t jffs2 /dev/mtdblock0 /mnt

... or where ever you want the flash image to appear in your file system.

[Note] Note

This might take a while, especially when the size of the drive is in the gigabyte range.

Populate the flash file system

Preparation

Before populating the target flash system, install the following package:

unionfs

This will be required to make the Debian package manager (apt) work on the flash file system

Copy files

cd /
tar cf - etc home lib root sbin usr var | (cd /mnt; tar xf - )
cd /mnt
mkdir dev mnt opt proc sys tmp .aptcache
[Note] Note

Do not populate the special directories dev, proc, and sys now. Just create empty directories.

Fine tuning the directory structure

Replace /var/tmp with a link to /tmp. We will mount tmpfs into /tmp later:

cd /mnt/var
rm -r tmp
ln -s /tmp

Remove files which will later be placed on tmpfs:

rm cache/apt/*.bin run/* lock/* log/* 
mv cache/apt cache/.apt

Create /etc/fstab

Edit /mnt/etc/fstab to look like this:

/dev/mtdblock0  /               jffs2   defaults        0       0
proc            /proc           proc    defaults        0       0
tmpfs           /tmp            tmpfs   defaults        0       0
tmpfs           /var/run        tmpfs   defaults        0       0
tmpfs           /var/log        tmpfs   defaults        0       0
tmpfs           /var/lock       tmpfs   defaults        0       0
tmpfs           /.aptcache      tmpfs   defaults        0       0
unionfs         /var/cache/apt  unionfs dirs=/.aptcache:/var/cache/.apt 0 0

Unmount the Flash image

Unmount the image

umount /mnt

Unload block2mtd driver

rmmod block2mtd

Release loop device

losetup -d /dev/loop0

It is now safe to unmount the Compact Flash. But before we do, we first

Create the Initial RAM Disk

When Linux first boots, there is just the kernel, a boot command line, and an embryonal file system called the Initial RAM Disk. The latter is normally created by the installation procedure, and updated by system updates, but in general, we don't have to worry about much. Booting a JFFS is different. We have to teach the Initial RAM Disk new tricks. Here is how. The following requires the package initramfs-tools. It usually comes with the installation. If you don't have it, install it now.

Add required modules

Create a file in /etc/initramfs-tools/hooks with the following content. You can choose any file name, say mtdsetup.

#!/bin/sh
PREREQ=""

prereqs()
{
      echo "$PREREQ"
}

case $1 in
# get pre-requisites
prereqs)
      prereqs
      exit 0
      ;;
esac

# Hooks for loading extra kernel bits into the initramfs
. /usr/share/initramfs-tools/hook-functions
manual_add_modules block2mtd
manual_add_modules loop
manual_add_modules vfat
manual_add_modules nls_cp437
manual_add_modules nls_iso8859-1
manual_add_modules jffs2
copy_exec /sbin/losetup /sbin

Make the file executable (chmod +x mtdsetup).

Create initialization scripts

The hook above just ensures that the mentioned modules and programs are included in the RAM disk. It does not cause them to be called. This must be done in init scripts. We need two of them.

Pre mount script

Create a file in /etc/initramfs-tools/scripts/local-top with the following content. You can choose any file name consisting of alphanumeric characters. Don't put a dash (minus) in the name! [3]

#!/bin/sh -e
PREREQ=""
prereqs()
{
        echo "$PREREQ"
}

case $1 in
# get pre-requisites
prereqs)
        prereqs
        exit 0
        ;;
esac

domtd=no

for x in $(cat /proc/cmdline); do
        case $x in
        mtd)
                domtd=yes
                ;;
        esac
done

if [ "$domtd" = "yes" ]
then
        mkdir /flash
        mount -t vfat /dev/hda1 /flash

        for i in 0 1 2 3 4 5 6 7; do
                mknod /dev/mtdblock${i} b 31 ${i}
                mknod /dev/loop${i} b 7 ${i}
        done

        losetup /dev/loop0 /flash/rootfs.img
        modprobe block2mtd block2mtd=/dev/loop0,65536
fi
exit 0

Make the script executable. It will be called during the boot process just before the root file system is mounted.

[Note] Note

The MTD emulation stuff is made conditional with a boot command line parameter mtd. This makes the same RAM disk usable for both a flash disk and a conventional boot. If you don't need this, you can leave that out.

[Note] Note

If you want to put /usr and /var into separate containers, just add the required losetup calls and add more block2mtd parameters to the modprobe call as decribed above. The setup provides up to eight loop and mtdblock devices for that purpose. The additional file systems will not be mounted until a later phase in the boot process, but the block2mtd setup must be done here.

Post mount script

Create a file in /etc/initramfs-tools/scripts/local-bottom with the following content. You can choose any file name consisting of alphanumeric characters. Don't put a dash (minus) in the name![3]

#!/bin/sh -e
PREREQ=""
prereqs()
{
        echo "$PREREQ"
}

case $1 in
# get pre-requisites
prereqs)
        prereqs
        exit 0
        ;;
esac

domtd=no

for x in $(cat /proc/cmdline); do
        case $x in
        mtd)
                domtd=yes
                ;;
        esac
done

if [ "$domtd" = "yes" ]
then
        umount -l /flash
        mount -t tmpfs tmpfs ${rootmnt}/var/log
        touch ${rootmnt}/var/log/dmesg
fi

exit 0

Make the script executable. It will be called during the boot process just after the root file system has been mounted, but before it is made the root of the file system.

[Note] Note

The original mount point (/flash) of the flash disk will disappear when the embryonal root file system is replaced by the real root. If we do nothing, the device will still be marked as mounted, and the shutdown procedure will try to unmount it, but cannot find a mount point. There are two possibilities to solve this: either move the mount point to a path in the new root (mount --move). This will cause the flash disk to remain mounted and this will be visible. The alternative used here is a lazy dismount. This will cause the device to be dismounted when the last file access is gone.

The last commands create an empty dmesg file to keep the boot procedure happy which tries to chown it and complains if it is not there.

Create RAM Disk and setup Syslinux boot loader

We are almost done. We now recreate the initial RAM disk using the command

update-initramfs -u

Copy the RAM disk image to the root of the flash disk. It should be safe to replace the RAM disk image created above.

Now update syslinux.cfg:

default vmlinuz
        append root=/dev/mtdblock0 rootfstype=jffs2 initrd=initrd.img mtd

label orig
      append  root=/dev/sda1 initrd=initrd.img
[Note] Note

Note the mtd parameter at the end of the kernel command line. It triggers the jffs2 handling in the scripts.

[Note] Note

You will find messages like

block2mtd: Overrun end of disk in cache readahead

in the system logs. This is harmless.

System maintenance

After running the system for a while, you might encounter the following problem:

Locale woes

After a system update, the defined locales might have disappeared. The reason for this is that the program locale-gen, which is supposed to generate new or updated locales, uses file mapping to map the file /usr/lib/locale/locale-archive directly into memory. However file mapping does not work with a compressed file system such as JFFS2[4], and generating the locales fails.

Solution: Generate the locales manually, after replacing the directory /usr/lib/locale with a piece of RAM disk as follows:

cd /usr/lib
rmdir locale
mkdir /tmp/locale
ln -s /tmp/locale .
locale-gen
rm locale
mkdir locale
mv /tmp/locale/* locale


[1] The energy consumed by a hard disk (rated at 10W) during one year pays for a discount 2GB Flash Drive, or for a 1GB brand model.

[2] My personal experience on that matter is a memory stick which became faulty after only one year of use.

[3] Thanks to Uwe Holz for the tip!

[4] For the same reason, apt will not work when the index files are located on a JFFS2 volume.