The Raspberry Pi is a cost effective SBC, which can be used for various internet-of-things purposes. I have been using 5 RPi’s as a camera solution with motion detection and post-processing. This application requires high reliability and it was a pretty long journey until all off the issues were solved. I am sharing the facts I have learned here, in the hope it will benefit other people in the attempt to attain stability on the Raspberry Pi.
1. Connectivity
The first thing you need is proper connectivity. At first this seemed easy, just use a WiFi USB-Key, set up wpa_supplicant, and you’re done. Guess not. There are many WiFi USB-Key’s on the market, but not all of them are supported under Linux. To be able to choose a working adapter, it’s advisable to take a look at the wireless wiki of kernel.org. Not all chipset manufacturers have open firmware for their products, limiting their usage in open source systems. So make sure to choose a listed chipset, for which there is open firmware, and a compatible driver, based on that, select an appropriate adapter.
For places where signal quality is low, it can be beneficial to connect an external antenna, this will amplify the reception of your wireless adapter by a few db, which sometimes makes a difference in good, bad, or ugly communication.
In my case, I was suffering from intermittent disconnections. To overcome this, I wrote a small script which would bring the connection back up when it failed.
#!/bin/bash
/sbin/iwconfig wlan0 power off
[ -f /tmp/wifi.lock ] && exit
touch /tmp/wifi.lock
ping -c4 8.8.8.8 &> /dev/null
if [ $? != 0 ]; then
logger -t $0 "Wifi seems down, restarting.."
killall dhclient
sleep 1
/sbin/ifdown --force wlan0
sleep 10
/sbin/ifup wlan0
sleep 10
/usr/sbin/service sshtunnel stop
/usr/sbin/service sshtunnel start
fi
rm /tmp/wifi.lock
2. Remote shell
The Raspberry Pi’s which I was configuring were not going to have a public IP address, nor was there any port forwarding installed on the remote end firewall for direct SSH access. Therefore I configured a reverse shell script which would perform a dial out to another SSH server under my control, and set up an SSH tunnel (reverse port forwarding).
$ sudo vi /usr/local/sbin/sshtunnel.sh
#!/bin/bash
if [ "$#" -ne 4 ]; then
echo "Usage: $0 user@host ssh_port local_port remote_port"
exit 1
fi;
while true; do
ssh -o ConnectTimeout=30 -o BatchMode=yes -o TCPKeepAlive=yes -o ServerAliveInterval=240 -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=no -N -R $4:localhost:$3 -p$2 $1 &> /dev/null
sleep 15
done;
Once the script was in place I still needed to install a service file to make it start automatically.
$ sudo vi /etc/init.d/sshtunnel
#! /bin/sh
### BEGIN INIT INFO
# Provides: sshtunnel
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop:
# Short-Description: SSH Tunnel
### END INIT INFO
set -e
export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"
. /lib/lsb/init-functions
case "$1" in
start)
log_daemon_msg "Starting SSH Tunnel" "sshtunnel" || true
iwconfig wlan0 power off
if start-stop-daemon --background --start --quiet --oknodo --pidfile /var/run/sshtunnel.pid --exec /usr/local/bin/sshtunnel.sh -- rpiuser\@yourhost 23 33 3302 ; then
log_end_msg 0 || true
else
log_end_msg 1 || true
fi
;;
stop)
log_daemon_msg "Stopping SSH Tunnel" "sshtunnel" || true
if start-stop-daemon --stop --quiet --oknodo --pidfile /var/run/sshtunnel.pid; then
log_end_msg 0 || true
else
log_end_msg 1 || true
fi
killall sshtunnel.sh &> /dev/null
killall ssh &> /dev/null
sleep 5
;;
status)
status_of_proc -p /var/run/sshtunnel.pid /usr/local/bin/sshtunnel sshtunnel && exit 0 || exit $?
;;
*)
log_action_msg "Unknown action!" || true
exit 1
esac
exit 0
A startup script needs to be writeable, so make sure to change the permissions to make the startup script executable:
$ sudo chmod 755 /etc/init.d/sshtunnel
It’s possible that you will need to amend the script to fit for your set-up. Basically after each boot or WiFi re-connection, the Raspberry Pi connects to my own SSH server and starts a reverse port forwarding so I can access the Raspberry Pi from remote. The login from the Raspberry Pi to my own SSH server is configured for single-sign-on, or key based authentication, so the login process is automated and doesn’t require any user attention.
3. Bad SD card
To survive an unexpected power off, you need to have all file systems mounted read-only.
Make sure to purchase a decent quality SD card for your Raspberry Pi. These cards are not made for heavy write operations, that will break them after time.
4. Read-only filesystem
This is one of the most complicated things, but when done properly, it’s not too much work and it won’t cause any problems.
Unless you have special requirements, there are only a few paths which needs to be writable. So let’s go through all steps required to make a truly, trouble-free read-only RPi:
4.1 Update your RPi so you have the recent software
It’s a bad idea to set it up with an old software. That may complicate the update later…
4.2 Remove unnecessary services and files
Here is the command. Of course, remove only what you don’t need. But this is what you probably won’t need on a headless RPi. You definitely don’t want cron on your read-only RPi unless you have external hardware clock source, more about it later.
apt-get remove --purge wolfram-engine triggerhappy cron anacron logrotate dbus dphys-swapfile xserver-common lightdm fake-hwclock
insserv -r x11-common apt-get autoremove --purge
4.3 Install buysbox syslog
You won’t need normal syslog text files on a read-only filesystem, either. Install busybox syslog instad. It logs into memory and is very lightweight. You can then use logread command to read syslog ringbuffer from the memory when needed.
apt-get install busybox-syslogd dpkg --purge rsyslog
4.4 Disable filesystem check and swap
Because the filesystem will be mounted read-only, there is nothing to be corrupted so filesystem check must be disabled. I say MUST because it MUST. If you don’t have an external HW clock and use NTP time sync only and you do a change to the filesystem and reboot, filesystem check will see it as an update from the future, denying further boot, requiring manual action on the site. To disable filesystem checks, specifying ‘fastboot’ to the kernel cmdline should be enough. You also don’t want any swapfiles. You can disable them by specifying ‘noswap’ to the cmdline. So edit /boot/cmdline.txt and append the following two at the end of the line:
fastboot noswap
4.5 Set up clock sync.
Because you uninstalled fake-hwclock (it won’t be able to store clock on a readonly filesystem), you need to install and set up NTP sync. Also clock keeping is poor on a standard RPi so you may consider updating time regularly (every hour or two should be enough).
I used ntpdate for this. apt-get install ntpdate. I added it to /etc/rc.local:
/usr/sbin/ntpdate -b cz.pool.ntp.org # change the ntp server according to your location
4.6 Update some writable paths
Now you need to update a few services which writes something. wpa_supplicant for WiFi is ok as it already writes to /tmp. DHCP lease is the major problem. Simple solution is to delete the old directory and make it as a symlink to tmp like this:
rm -rf /var/lib/dhcp/ ln -s /tmp /var/lib/dhcp You can consider adding more symlinks from some /var subdirectories, especially run,spool and lock rm -rf /var/run /var/spool /var/lock ln -s /tmp /var/run ln -s /tmp /var/spool ln -s /tmp /var/lock
4.7 Consider disabling some other startup scripts
insserv -r bootlogs insserv -r sudo # if you plan to be root all the time insserv -r alsa-utils # if you don't need alsa stuff (sound output) insserv -r console-setup insserv -r fake-hwclock # probably already removed at this point..
If you use alsamixer to set up volume level, make sure to do so in read-write filesystem. If won’t be able to store it on a readonly filesystem. It normally uses this path /var/lib/alsa/asound.state .
4.8 Tell the kernel to mount root filesystem read-only!
Finally getting there.. Add ” ro” at the end of your /boot/cmdline.txt line.
4.9. Add “,ro” flag to both block devices in /etc/fstab
…to let the system know you want to mount them read-only:
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults,ro 0 2 /dev/mmcblk0p2 / ext4 defaults,ro 0 1 tmpfs /tmp tmpfs defaults 0 0
4. Watchdog
It is useful to set up a watchdog which can reboot your RPi in case it renders unresponsive, we can use a watchdog kernel module for this purpose.
Load the watchdog kernel module:
$ sudo modprobe bcm2708_wdog
Add bcm2708_wdog into the /etc/modules so it gets loaded on boot.
$ sudo echo "bcm2708_wdog" >> /etc/modules
In addition to the kernel module, there is also a userspace daemon that we need:
$ sudo apt-get install watchdog
Examine the configuration file: /etc/watchdog.conf and configure it as appropriate for your situation.
Uncomment the line watchdog-device = /dev/watchdog
Uncomment the line with max-load-1
Setting a minimum free RAM amount is a good idea. Before starting the watchdog service, be prepared that you might have configured it incorrectly and it will reboot immediately when you start it and may continuously reboot after each boot. So be prepared to modify your SD card on a different device if that happens.
Enable the watchdog to start at boot and start it now:
$ sudo insserv watchdog
$ sudo /etc/init.d/watchdog start
During some modifications to your system (in read-write mode) later, you can consider disabling watchdog first. It rebooted my box once while I was doing some filesystem changes. Fortunately it booted fine for me, but it may not for you and may require manual, local fix.
In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing /etc/sysctl.conf. Add this line:
kernel.panic = 10
This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.
5. Add some cool utilities
It is helpful to receive syslog over the network. I think you can somehow enable sending busybox syslog over the network. I made my own script for this purpose, though. It reads output from ‘logread’ and sends it simply over the UDP to my server where it is saved to a logfile. Simple and helped a few times. Make sure to set up correct hostnames by editing /etc/hostname.
5.1 Cron?
Normal cron can’t be used unless you have a real external HW clock source. Because normally you can’t be sure that the clock got updated by NTP. If you can ensure it somehow, then fine, use cron. If not and using relative time is enough, you can make a fake cron using bash script, while loop and wait commands. Here is mine: (to be placed as /usr/local/fakecron.sh) https://paste.k3a.me/view/4515b0a4 . I use this startup script for it (to be placed to /etc/fakecron) https://paste.k3a.me/view/0a995efd.
6. Reboot!
Now it’s the best part. If you did everything correctly, it will boot just fine. If not, look at syslog and try to find out why. You can fix the SD card in a different computer.
7. What to do next
Enjoy your reliable RPi. Good work! If you ever want to update the software, just remount the root filesystem as read-write temporarily:
mount -o remount,rw /
You may want to stop watchdog temporarily. Now run your apt-get etc stuff, modify what you need.. then mount read-only again:
mount -o remount,ro /
Camera
I don’t know why and if it is still true, but raspicam leaked memory for me, causing RPi to reboot after some time. I managed to improve it by disabling preview (-n) but still I set up RPi to reboot daily just to be sure……
Done
And that should be it. Hopefully it helped. If it all works, back up your SD card using dd (google it https://www.google.com/search?q=backup+using+dd).
If you have some tips, write them in comments. I can also update the article to include more info.
Random notes of a SysAdmin 