HOWTO – Make a Raspberry Pi truly read-only, reliable and trouble-free

The Raspberry Pi is a cost effective SBC, which can be used for various internet-of-things purposes. I have been using 5 RPi’s as a camera solution with motion detection and post-processing. This application requires high reliability and it was a pretty long journey until all off the issues were solved. I am sharing the facts I have learned here, in the hope it will benefit other people in the attempt to attain stability on the Raspberry Pi.

1. Connectivity

The first thing you need is proper connectivity. At first this seemed easy, just use a WiFi USB-Key, set up wpa_supplicant, and you’re done. Guess not. There are many WiFi USB-Key’s on the market, but not all of them are supported under Linux. To be able to choose a working adapter, it’s advisable to take a look at the wireless wiki of kernel.org. Not all chipset manufacturers have open firmware for their products, limiting their usage in open source systems. So make sure to choose a listed chipset, for which there is open firmware, and a compatible driver, based on that, select an appropriate adapter.

For places where signal quality is low, it can be beneficial to connect an external antenna, this will amplify the reception of your wireless adapter by a few db, which sometimes makes a difference in good, bad, or ugly communication.

In my case, I was suffering from intermittent disconnections. To overcome this, I wrote a small script which would bring the connection back up when it failed.

#!/bin/bash
 
/sbin/iwconfig wlan0 power off
 
[ -f /tmp/wifi.lock ] && exit
touch /tmp/wifi.lock
 
ping -c4 8.8.8.8 &> /dev/null
 
if [ $? != 0 ]; then
        logger -t $0 "Wifi seems down, restarting.."
        killall dhclient
        sleep 1
        /sbin/ifdown --force wlan0
        sleep 10
        /sbin/ifup wlan0
        sleep 10
        /usr/sbin/service sshtunnel stop
        /usr/sbin/service sshtunnel start
fi
 
rm /tmp/wifi.lock

2. Remote shell

The Raspberry Pi’s which I was configuring were not going to have a public IP address, nor was there any port forwarding installed on the remote end firewall for direct SSH access. Therefore I configured a reverse shell script which would perform a dial out to another SSH server under my control, and set up an SSH tunnel (reverse port forwarding).

$ sudo vi /usr/local/sbin/sshtunnel.sh
#!/bin/bash
 
if [ "$#" -ne 4 ]; then
        echo "Usage: $0 user@host ssh_port local_port remote_port"
        exit 1
fi;
 
while true; do
        ssh -o ConnectTimeout=30 -o BatchMode=yes -o TCPKeepAlive=yes -o ServerAliveInterval=240 -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=no -N -R $4:localhost:$3 -p$2 $1 &> /dev/null
        sleep 15
done;

Once the script was in place I still needed to install a service file to make it start automatically.

$ sudo vi /etc/init.d/sshtunnel
#! /bin/sh
 
### BEGIN INIT INFO
# Provides:             sshtunnel
# Required-Start:       $remote_fs $syslog
# Required-Stop:        $remote_fs $syslog
# Default-Start:        2 3 4 5
# Default-Stop:
# Short-Description:    SSH Tunnel
### END INIT INFO
 
set -e
 
export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
 
. /lib/lsb/init-functions
 
case "$1" in
  start)
        log_daemon_msg "Starting SSH Tunnel" "sshtunnel" || true
        iwconfig wlan0 power off
        if start-stop-daemon --background --start --quiet --oknodo --pidfile /var/run/sshtunnel.pid --exec /usr/local/bin/sshtunnel.sh -- rpiuser\@yourhost 23 33 3302 ; then
            log_end_msg 0 || true
        else
            log_end_msg 1 || true
        fi
        ;;
  stop)
        log_daemon_msg "Stopping SSH Tunnel" "sshtunnel" || true
        if start-stop-daemon --stop --quiet --oknodo --pidfile /var/run/sshtunnel.pid; then
            log_end_msg 0 || true
        else
            log_end_msg 1 || true
        fi
        killall sshtunnel.sh &> /dev/null
        killall ssh &> /dev/null
        sleep 5
        ;;
 
  status)
        status_of_proc -p /var/run/sshtunnel.pid /usr/local/bin/sshtunnel sshtunnel && exit 0 || exit $?
        ;;
  *)
        log_action_msg "Unknown action!" || true
        exit 1
esac
 
exit 0

A startup script needs to be writeable, so make sure to change the permissions to make the startup script executable:

$ sudo chmod 755 /etc/init.d/sshtunnel

It’s possible that you will need to amend the script to fit for your set-up. Basically after each boot or WiFi re-connection, the Raspberry Pi connects to my own SSH server and starts a reverse port forwarding so I can access the Raspberry Pi from remote. The login from the Raspberry Pi to my own SSH server is configured for single-sign-on, or key based authentication, so the login process is automated and doesn’t require any user attention.

3. Bad SD card

To survive an unexpected power off, you need to have all file systems mounted read-only.

Make sure to purchase a decent quality SD card for your Raspberry Pi. These cards are not made for heavy write operations, that will break them after time.

4. Read-only filesystem

This is one of the most complicated things, but when done properly, it’s not too much work and it won’t cause any problems.

Unless you have special requirements, there are only a few paths which needs to be writable. So let’s go through all steps required to make a truly, trouble-free read-only RPi:

4.1 Update your RPi so you have the recent software

It’s a bad idea to set it up with an old software. That may complicate the update later…

4.2 Remove unnecessary services and files

Here is the command. Of course, remove only what you don’t need. But this is what you probably won’t need on a headless RPi. You definitely don’t want cron on your read-only RPi unless you have external hardware clock source, more about it later.

apt-get remove --purge wolfram-engine triggerhappy cron anacron logrotate dbus dphys-swapfile xserver-common lightdm fake-hwclock
insserv -r x11-common
apt-get autoremove --purge

4.3 Install buysbox syslog

You won’t need normal syslog text files on a read-only filesystem, either. Install busybox syslog instad. It logs into memory and is very lightweight. You can then use logread command to read syslog ringbuffer from the memory when needed.

apt-get install busybox-syslogd
dpkg --purge rsyslog

4.4 Disable filesystem check and swap

Because the filesystem will be mounted read-only, there is nothing to be corrupted so filesystem check must be disabled. I say MUST because it MUST. If you don’t have an external HW clock and use NTP time sync only and you do a change to the filesystem and reboot, filesystem check will see it as an update from the future, denying further boot, requiring manual action on the site. To disable filesystem checks, specifying ‘fastboot’ to the kernel cmdline should be enough. You also don’t want any swapfiles. You can disable them by specifying ‘noswap’ to the cmdline. So edit /boot/cmdline.txt and append the following two at the end of the line:

fastboot noswap

4.5 Set up clock sync.

Because you uninstalled fake-hwclock (it won’t be able to store clock on a readonly filesystem), you need to install and set up NTP sync. Also clock keeping is poor on a standard RPi so you may consider updating time regularly (every hour or two should be enough).

I used ntpdate for this. apt-get install ntpdate. I added it to /etc/rc.local:

/usr/sbin/ntpdate -b cz.pool.ntp.org # change the ntp server according to your location

4.6 Update some writable paths

Now you need to update a few services which writes something. wpa_supplicant for WiFi is ok as it already writes to /tmp. DHCP lease is the major problem. Simple solution is to delete the old directory and make it as a symlink to tmp like this:

rm -rf /var/lib/dhcp/
ln -s /tmp /var/lib/dhcp

You can consider adding more symlinks from some /var subdirectories, especially run,spool and lock
rm -rf /var/run /var/spool /var/lock
ln -s /tmp /var/run 
ln -s /tmp /var/spool
ln -s /tmp /var/lock

4.7 Consider disabling some other startup scripts

insserv -r bootlogs
insserv -r sudo # if you plan to be root all the time
insserv -r alsa-utils # if you don't need alsa stuff (sound output)
insserv -r console-setup
insserv -r fake-hwclock # probably already removed at this point..

If you use alsamixer to set up volume level, make sure to do so in read-write filesystem. If won’t be able to store it on a readonly filesystem. It normally uses this path /var/lib/alsa/asound.state .

4.8 Tell the kernel to mount root filesystem read-only!

Finally getting there.. Add ” ro” at the end of your  /boot/cmdline.txt line.

4.9. Add “,ro” flag to both block devices in /etc/fstab

…to let the system know you want to mount them read-only:

proc              /proc           proc    defaults     0       0
/dev/mmcblk0p1    /boot           vfat    defaults,ro  0       2
/dev/mmcblk0p2    /               ext4    defaults,ro  0       1
tmpfs             /tmp            tmpfs   defaults     0       0

4. Watchdog

It is useful to set up a watchdog which can reboot your RPi in case it renders unresponsive, we can use a watchdog kernel module for this purpose.

Load the watchdog kernel module:

$ sudo modprobe bcm2708_wdog 

Add bcm2708_wdog into the /etc/modules so it gets loaded on boot.

$ sudo echo "bcm2708_wdog" >> /etc/modules

In addition to the kernel module, there is also a userspace daemon that we need:

$ sudo apt-get install watchdog 

Examine the configuration file: /etc/watchdog.conf and configure it as appropriate for your situation.

Uncomment the line watchdog-device = /dev/watchdog

Uncomment the line with max-load-1

Setting a minimum free RAM amount is a good idea. Before starting the watchdog service, be prepared that you might have configured it incorrectly and it will reboot immediately when you start it and may continuously reboot after each boot. So be prepared to modify your SD card on a different device if that happens.

Enable  the watchdog to start at boot and start it now:

$ sudo insserv watchdog  
$ sudo /etc/init.d/watchdog start

During some modifications to your system (in read-write mode) later, you can consider disabling watchdog first. It rebooted my box once while I was doing some filesystem changes. Fortunately it booted fine for me, but it may not for you and may require manual, local fix.

In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing /etc/sysctl.conf. Add this line:

 kernel.panic = 10 

This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.

5. Add some cool utilities

It is helpful to receive syslog over the network. I think you can somehow enable sending busybox syslog over the network. I made my own script for this purpose, though. It reads output from ‘logread’ and sends it simply over the UDP to my server where it is saved to a logfile. Simple and helped a few times. Make sure to set up correct hostnames by editing /etc/hostname.

5.1 Cron?

Normal cron can’t be used unless you have a real external HW clock source. Because normally you can’t be sure that the clock got updated by NTP. If you can ensure it somehow, then fine, use cron. If not and using relative time is enough, you can make a fake cron using bash script, while loop and wait commands. Here is mine: (to be placed as /usr/local/fakecron.sh) https://paste.k3a.me/view/4515b0a4 . I use this startup script for it (to be placed to /etc/fakecron) https://paste.k3a.me/view/0a995efd.

6. Reboot!

Now it’s the best part. If you did everything correctly, it will boot just fine. If not, look at syslog and try to find out why. You can fix the SD card in a different computer.

7. What to do next

Enjoy your reliable RPi. Good work! If you ever want to update the software, just remount the root filesystem as read-write temporarily:

 mount -o remount,rw / 

You may want to stop watchdog temporarily. Now run your apt-get etc stuff, modify what you need.. then mount read-only again:

mount -o remount,ro / 

Camera

I don’t know why and if it is still true, but raspicam leaked memory for me, causing RPi to reboot after some time. I managed to improve it by disabling preview (-n) but still I set up RPi to reboot daily just to be sure……

Done

And that should be it. Hopefully it helped. If it all works, back up your SD card using dd (google it https://www.google.com/search?q=backup+using+dd).

If you have some tips, write them in comments. I can also update the article to include more info.

Sources

Published by

Ronny Van den Broeck

I'm a network and system engineer for more than 20 years now. During this period I became a pro in hunting down one's and zero's, with an eager mindset to help people accomplish the same or abstract them away from the matrix.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s