I recently upgraded my systems to Debian Stretch, which caused GnuPG to stop working within Mutt. I'm not exactly sure what was wrong, but I discovered that GnuPG version 2 changed quite some things and relies more heavily on the gpg-agent, and I discovered that recent SSH version can forward unix domain socket instead of just TCP sockets, which allows forwarding a gpg-agent connection over SSH.

Until now, I had my GPG private keys stored on my server, Tika, where my Mutt mail client also runs. However, storing private keys, even with a passphrase, on permanentely connected multi-user system never felt quite right. So this seemed like a good opportunity to set up proper forwarding for my gpg agent, and keep my private keys confined to my laptop.

I already had some small scripts in place to easily connect to my server through SSH, attach to the remote tmux session (or start it), set up some port forwards (in particular a reverse port forward for SSH so my mail client and IRC client could open links in my browser), and quickly reconnect when the connection fails. However, once annoyance was that when the connection fails, the server might not immediately notice, so reconnecting usually left me with failed port forwards (since the remote listening port was still taken by the old session). This seemed like a good occasion to fix that as wel.

The end result is a reasonably complex script, that is probably worth sharing here. The script can be found in my scripts git repository. On the server, it calls an attach script, but that's not much more than attaching to tmux, or starting a new session with some windows if no session is running yet.

The script is reasonably well-commented, including an introduction on what it can do, so I will not repeat that here.

For the GPG forwarding, I based upon this blogpost. There, they suggest configuring an extra-socket in gpg-agent.conf, but I've found that gpg-agent already created an extra socket (whose path I could query with gpgconf --list-dirs), so I didn't use that extra-socket configuration line. They also talk about setting StreamLocalBindUnlink to clean up a lingering socket when creating a new one, but that is already handled by my script instead.

Furthermore, to prevent a gpg-agent from being autostarted by gnupg serverside (in case the forwarding fails, or when I would connect without this script, etc.), I added no-autostart to ~/.gnupg/gpg.conf. I'm not running systemd user session on my server, but if you are you might need to disable or mask some ssh-agent sockets and/or services to prevent systemd from creating sockets for ssh-agent and starting it on-demand.

My next step is to let gpg-agent also be my ssh-agent (or perhaps just use plain ssh-agent) to enforce confirming each SSH authentication request. I'm currently using gnome-keyring / seahorse as my SSH agent, but that just silently approves everything, which doesn't really feel secure.

On a small embedded system, I wanted to run a simple Rails application and have it automatically start up at system boot. The system is running systemd, so a systemd service file seemed appropriate to start the rails service.

Normally, when you run the ruby-on-rails standalone server, it binds on port 3000. Binding on port 80 normally requires root (or a special capability enabled for all of ruby), but I don't want to run the rails server as root. AFAIU, normal deployments using something like Nginx to open port 80 and let it forward requests to the rails server, but I wanted a minimal setup, with just the rails server.

An elegant way to binding port 80 without running as root is to use systemd's socket activation feature. Using socket activation, systemd (running as root) opens up a network port before starting the daemon. It then starts the daemon, which inherits the open network socket file descriptor, with some environment variables to indicate this. Apart from allowing privileged ports without root, this has other advantages such as on-demand starting, easier parallel startup and seamless restarts and upgrades (none of which is really important for my usecase, but it is still nice :-p).

See more ...

TL;DR: This post describes an easy way to add a power button to a raspberryp pi that:

• Only needs a button and wires, no other hardware components.
• Allows graceful shutdown and powerup.
• Only needs modification of config files and does not need a dedicated daemon to read GPIO pins.

There are two caveats:

• This shuts down in the same way as shutdown -h now or halt does. It does not completely cut the power (like some hardware add-ons do).
• To allow powerup, the I²C SCL pin (aka GPIO3) must be used, conflicting with externally added I²C devices.

If you use Raspbian stretch 2017.08.16 or newer, all that is required is to add a line to /boot/config.txt:

dtoverlay=gpio-shutdown,gpio_pin=3

Make sure to reboot after adding this line. If you need to use a different gpio, or different settings, lookup gpio-shutdown in the docs.

Then, if you connect a pushbutton between GPIO3 and GND (pin 5 and 6 on the 40-pin header), you can let your raspberry shutdown and startup using this button.

If you use an original Pi 1 B (non-plus) revision 1.0 (without mounting holes), pin 5 will be GPIO1 instead of GPIO3 and you will need to specify gpio_pin=1 instead. The newer revision 2.0 (with 2 mounting holes in the board) and all other rpi models do have GPIO3 and work as above.

All this was tested on a Rpi Zero W, Rpi B (rev 1.0 and 2.0) and a Rpi B+.

If you have an older Raspbian version, or want to know how this works, read on below.

See more ...

On my mailserver, I'm using Spamassassin with a Bayes filter to detect spam. Such a filter needs to be trained with samples of spam and ham (non-spam) messages to let it learn what spam and ham looks like, but it also needs to be retrained when the spam or ham changes over time. I have some automatic training set up, but since a while I've seen the bayes filter being completely wrong (showing a confident ham score for something that is very clearly spam), so I decided to retrain the filter from scratch, using the spam and ham messages I collected over the last time (I don't really throw away any e-mail).

Since training with all my e-mail is not productive (more than 5,000 messages aren't really helpful AFAIU, and training with old messages is not representative for current messages), I decided to just take all of my e-mail and take the last 2,000 spam and ham messages and train with that. My spam is neatly collected in 2 mailboxes (Spam for obvious spam and ProbablySpam for messages that need an occasional review to find false positives), but my ham is sorted out in dozens of different mailboxes. Hence, I needed some find magic to get a list of the most recent spam and ham messages. So, I built these commands:

# find Spam ProbablySpam -type f \( -path '*/cur/*' -o -path '*/new/*' \) -printf "%T@ %p\n"
  | sort -n | cut -d' ' -f 2 | tail -n 2000 > spam
# find . -type d \( -path ./Spam -o -path ./ProbablySpam -o -path ./Bulk -o -path ./Sent \) -prune -o \
         -type f \( -path '*/cur/*' -o -path '*/new/*' \) -printf "%T@ %p\n" \
  | sort -n | cut -d' ' -f 2 | tail -n 2000 > ham
# sa-learn --progress --spam -f spam
# sa-learn --progress --ham -f ham

After retraining with recent spam, the results were a lot better, so I'm not longer spending time every day deleting a couple dozens spam e-mails :-D

XBee modules are a range of wireless modules built by Digi, and are typically used to add wireless connectivity to Arduino or other microcontroller based projects. To configure these modules and update their firmware, you can use the XCTU configuration utility. This utility uses a serial port to talk to the XBee module, so you will need some way to connect to the XBee module to a serial port on your computer (using a USB "TTL" serial port, a "real" RS232 port has too high voltage).

The easiest way is to use a dedicated board, like the SparkFun Explorer USB:

However, if you already have an Arduino and an XBee shield for it, you might want to use those to connect XCTU to your XBee module. In theory, this should be a matter of re-arranging some wires, but in practice I've run into some problems attempting this (depending on the hardware used).

In this post, I'll show a few ways to do this using an Arduino and a shield, and explain some of the problems you might run into.

See more ...

My book about Arduino and XBee includes a chapter on battery power and sleeping. When I originally wrote it, it ended up over twice the number of pages originally planned for it, so I had to severely cut down the content. Among the content removed, was a large section talking about interrupts, sleeping and race conditions. Since I am not aware of any other online sources that cover this subject as thoroughly, I decided to publish this content as a blogpost separately, which is what you're looking at now.

In this blogpost, I will first explain interrupts and race conditions using a number of examples. Then sleeping is added into the mix, which again results in some interesting race conditions. All these examples have been written for Arduino boards using the AVR architecture, but the general concepts apply equally well to other platforms.

The basics of interrupts and sleeping on AVR are not covered in detail here. If you have no experience with this, I recommend these excellent articles on interrupts and on sleeping by Nick Gammon, which cover interrupts, sleeping and other powersaving in a lot of detail.

See more ...

Yay! Last night (yes really night, it was 4:30 AM) I finalized and submitted the last chapter of my book. There's still a few details left, but the text of the book is out of my hands now, and the great folks at Packt publishing are now doing their magic with correcting my English, testing my code, layouting the book, and so on.

Great to have the pressure of these (repeatedly postponed) deadlines off my back, so I can finally take some time to do some long-due other stuff. Like cleaning up my desk:

Past summer, I've been writing a book, which is now nearing completion. A few months ago I was approached by Packt, a publisher of technical books, which want to add a book about sensor networks using Arduino and XBee modules to their lineup, and asked me to write it. The result will be a book titled "Building Wireless Sensor Networks Using Arduino".

I've been working on the book since then and am now busy with the last revisions on the text and examples. Turns out that writing a book is a lot more work than I had anticipated, though that might be partly due to my own perfectionism. In any case, I'll be happy when I'm done in a few weeks and can spend some time again on all the stuff I've been postponing in the last few months :-)

I originally wrote this, just under a year ago when I returned from this sailing trip. I found this draft post again a few weeks ago, being nearly finished. Since I'm about to leave on a new sailing trip next week (to France this time, if the wind permits), I thought it would be nice to finish this post before then. Even though it's a year old, the story is bizarre enough to publish it still :-)

I've put up photographs taken by me and Danny van den Akker up on Flickr.

Last year in May, I went on a 11 day sailing trip to the Isle of Wight an island off the southern coast of England. Together with my father, brother and two others on my father's boat as well as 10 other people on two more boats, we started out on this trip from Harderwijk on Thursday, navigating to the port of IJmuiden and going out on the north sea from there on Friday morning.

Initially, we had favourable wind and made good speed up to Dover at noon on Saturday. From then on, it seemed Murphy had climbed aboard our ship and we were in for our fair share of setbacks, partly balanced by some insane luck as well. In any case, here's a short timeline of the bad and lucky stuff:

• :-( Just around Beachy Head, our engine stopped running. We quickly restarted it and carried on, but shortly after the engine stopped again. It didn't want to start again either. In case you're wondering
  • not having a working engine is a problem for a (bigger) sailing ship. Sailing into an (unknown) (sea) port is usually hard and on the open sea your engine is an important safety measure in case of problems.
• :-) Fortunately, Atlantis, one of our companion ships was only a few miles ahead (after having crossed most the North Sea through different routes, many miles apart). We sailed towards the port of New Haven, where they towed us into the port.
• :-( We diagnosed that there was a problem in the fuel supply, but couldn't find the real cause. We spent the sunday in the thrilling harbour town of New Haven, waiting for a mechanic to arrive.
• :-) On monday morning, a friendly mechanic called Neil fixed our fuel tank - it turned out there was a hose in there with a clogged filter as well as a kink near the end that prevented fuel from getting through. Neil cut off the end of the tube, fixing both problems at once and our engine ran again like a charm.

• :-( We continued towards Wight around noon, but shortly after passing No mans land Fort - guess what - our engine stopped running again.
• :-( This time we had no friends around to tow us in, but there was a Dutch ship (the Abel Tasman) nearby. We tried to get their attention using radio, a horn and a hand-held flare, but they misunderstood our intentions and carried on. We kept on sailing, but contacted the very much polite and helpful Solent Coast Guard by radio, to let them know we were coming in by sail and might needed a tow into the harbour.
• :-) After we finished our conversation with the Solent Coast Guard, the Abel Tasman hailed us on the radio and offered to tow us all the way to the harbor. With the wind dropping, we gladly accepted.
• :-) On monday evening We arrived on the Isle of Wight!
• :-) In the harbour, our engine started again (with some initial coughing). Yay for having a working engine, but without having fixed a problem, it could break down any moment. :-( We had some theories about what the problem could be, but no real confidence in any of them.
• :-) In the morning, we called for Tristan, a mechanic from the town Cowes where we were staying. He had a look at our engine, but since it was running again, there wasn't much he could do. He found a connection in the fuel line that wasn't very tight - guessing that that had allowed some air to slip into the fuel line. He tighted the connection and left us again.
• :-( Shortly after he left, we started the engine for a short refueling trip - and it stopped running again. We called for the mechanic to have another look, but he didn't show up all day...

• :-( Near the end of the day, we didn't want to wait anymore and tried to bleed out the air from our engine, thinking that was the last problem. No matter how hard we tried, air nor fuel would come through. Debugging further, we found that the fuel tank would again not supply any fuel at all. We "fixed" this by disconnecting the fuel line, blowing into it and then sucking at it (credits to Jan-Peter for sucking up a mouthful of Diesel...). Together with the diesel there came also a wad of something that looked like mud.
• :-( Previously, we didn't suspect any pollution of our fuel tank, because the looking glass in the fuel filter was squeeky clean. Now, prompted by the gunk coming out of the tank, we opened up the filter to have a more thorough look and it turned out the filter was nearly completely filled with the same muddy stuff (which turned out to be "Diesel Bug" - various (byproducts of) bacteria and other organisms that like liveing in diesel tanks)
• :-) We didn't have a working engine yet, but at least we knew with confidence what the cause of our problems was, and had been. As Danny stated it: At least it is truely broken now. All we needed now was machinery to filter our fuel and rinse the inside of the tank.
• :-( On Tuesday, we found another mechanic who couldn't help us directly, but made a few phone calls. He found someone with the exact machinery we needed. The only problem was that the machinery was in the Swanwick Marina, on the UK mainland.

• :-) On Wednesday, we tried to get into contact with the mechanic in Swanwick, to see if he could come over to Wight by ferry. We weren't hopeful, so in the meantime we also found a small plastic fuel tank (normally used for larger outboard motors) that could serve as a spare diesel tank. After also finding a few meters of diesel hose, getting an extra hose barb fitted and even getting a valve for fast on-the-fly switching between tanks (Thanks to Power Plus Marine Engineering for going out of their way to help us!), we had a working engine again by the end of the day!

  We weren't confident our shiny new tank to sail all the way back to the Netherlands using it, but it would be more than sufficient to get over to Swanwick to get our main tank cleaned. As an extra bonus, it would be a good backup tank in case we still had problems after cleaning.

• :-) Early Thursday morning, we headed over to Swanwick where Geoffrey and Johnny laboured all morning to get a ton of sludge out of our tank. They sprayed the fuel back in using a narrow piece of pipe to clean the stuff off the edges of our tank as good as possible.

• :-( As soon as they were done, we cast off and set course to Brighton Eastbourne Dover Ramsgate Dunkerque, Oostende, Zeebrugge, Dunkerque. Unfortunately, the wind direction was unfavorable (and later also didn't abide by the prediction, in our disadvantage of course), so we ended up sailing for only a small bit and using our engine most of the way.

  We changed our destination a few times along the way, usually because we thought "We might as well go on for a bit more, the further we go now, the easier it will be for the last stretch". Near Dunkerque, we planned to go on to Zeebrugge, but the wind picked up significantly. With the waves and wind against, we were getting a very rough ride, so we stopped at Dunkerque after all.

• :-( We came into port friday after office hours, so there was no personell anymore. In the restaurant we managed to get an access card so we could get to the toilet and back again, but the card had no balance, so we couldn't even take a shower...
• :-( Early saturday morning we carried on, this time with favorable wind direction, but not enough wind to make a decent speed. Since we lost all the slack in our schedule already (we had planned to be home by saturday afternoon), we used our engine again to make some speed.

• :-( Shortly after crossing the Westerschelde, our engine stopped running again (can you believe that?). We quickly found that the engine itself would run fine, but would stop as soon as you put it into gear - there was something blocking up our screw propeller. Using Danny's GoPro underwater camera we confirmed that there was indeed something stuck in the screw. We didn't want to go into the water to fix it on open sea because of the safety risk, as well as the slim chance that we'd actually be able to cut out whatever was stuck.

• :-) We contacted the coast guard again (the dutch one this time), telling them we'd sail towards the Roompot locks and would probably need a tow through the locks and into the harbour. We contacted the Burghsluis harbour and found they had a boat crane and were willing to operate it during the evening, so we had our target.

• :-( Then the wind dropped to nearly zero and we just floated around the North Sea for a bit. After an hour of floating, we contacted the coast guard and asked them to tow us in all the way - we were losing time quickly and with unreliable wind, there was no way we could safely cross the northern exit channel of the Westerschelde.

• :-) While floating around a bit more, waiting for the coast guard to arrive, we slowly floated nearer to a small boat that we had previously seen in the distance already. Now, and this is where the story becomes even more unlikely, this boat was running a "diver flag", meaning there was a diver in the water nearby.

  We slowly approached the boat and shortly after yelling over our problem to the woman on the boat, the diver emerged from the water. She told him of our problem, he swam over and under our boat and not even 30 seconds later he emerged again carrying a fairly short but thick piece of rope that had been stuck in our screw.

  We started our engine and were on our way again, suffering about a 3-4 hour delay in total.

• :-) We contacted the coast guard again to call them off, but they were already nearly there. To resolve some formalities, they continued heading our way. A minute or so later, the Koopmansdansk, a super-speedy boat from the KNRM came alongside. After we gave (read: Tried to yell over the roaring of their motor) them some contact information and the gave us a flyer about sponsoring the KNRM, we both went our separate ways again.

• :-( During the night, a thunderstorm came over us. I missed most of it because I was sleeping, but Jan Peter and Danny were completely drenched and attested that it was both heavy and somewhat scary due to the lightning.
• :-) Early sunday morning we went through the locks in IJmuiden, passed through Amsterdam and the Markermeer pretty uneventful. It even seemed like I would be at home just in time to head over to Enschede for our regular Glee night with some friends.
• :-( We were running low on diesel, so we had to find a place to fuel up in Lelystad. We found a harbour that was documented to have a gas pump, but didn't have one, so wasted some time finding one...
• :-( When we got to the locks of Roggebotsluis, it took a long time for the bridge to open. After floating around for 10 minutes or so, we noticed that one of the boom barriers (slagboom) looked warped and bent. Turns out just before we arrived there, some car got locked in between the barriers, panicked, hit the barrier while backing up and then sped away past the barrier on the opposite side. A mechanic was being summoned, but being a sunday...
• :-) After a short while, the bridge controller fellow came out, pulled the boom somewhat back into shape a bit and announced they'd give it a go. Thankfully for us, everything still worked and we were on our way!
• :-) No more delays after that. Brenda picked me up in the harbour, where I took a very quick shower before we headed to Enschede. Just in time for Glee, though I started falling asleep before we were even halfway. Apparently I was exhausted...

Overall, this was a crazy journey. All the problems we had have been a bit stressful and scary at times, but it was nice nonetheless. I especially enjoyed trying get things working again together, it was really a team effort.

Also interesting, especially for me, is that we had installed an AIS receiver on-board, which allows receiving information from nearby ships about their name, position, course and speed. This is especially useful in busy areas with lots of big cargo ships and to safely make your way through the big shipping lanes at sea.

This AIS receiver needed to be wired up to the existing instruments and to my laptop, so we could view the information on our maps. So I spent most of the first day belowdecks figuring this stuff out. I continued fiddling with these connections and settings to improve the setup, running into some bugs and limitations. Knowing I might have to improvise, I had packed a few Arduinos, some basic electronics and an RS232 transceiver, which allowed me to essentially build a NMEA multiplexer that can forward some data and drop other data. Yay for building your own hardware and software while at sea. Also yay for not getting seasick easily :-p

Now, I'm getting ready for our next trip. We start, and possibly end, in Vlissingen this time (to buy a bit more time) and intend to go to Cherbourg. Let's hope this journey is a bit less "interesting" than the previous one!

I don't really have a well-defined plan for my professional carreer, but things keep popping up. A theme for this month seems to be "education".

I just returned from a day at Saxion hogeschool in Enschede, where I gave a lecture/workshop about programming Arduino boards without using the Arduino IDE or Arduino core code.

It was nice to be in front of students again and things went reasonably well. It's a bit different to engage non-academic (HBO) students and I didn't get around to telling and doing everything I had planned, but I have another followup lecture next week to improve on things.

At the same time, I'm currently talking with a publisher of technical books about writing a book about building wireless sensor networks with Arduino and XBee. I'll have to do some extensive research into the details of XBee for this, but I'm looking forward to see how I like writing such a technical book. If this blog is any indication, that should work out just fine.

For now, these are just small things next to my other work, but who knows what I'll come across next?