These are the ramblings of Matthijs Kooijman, concerning the software he hacks on, hobbies he has and occasionally his personal life.
Most content on this site is licensed under the WTFPL, version 2 (details).
Questions? Praise? Blame? Feel free to contact me.
My old blog (pre-2006) is also still available.
See also my Mastodon page.
Sun | Mon | Tue | Wed | Thu | Fri | Sat |
---|---|---|---|---|---|---|
1 | 2 | |||||
3 | 4 | 5 | 6 | 7 | 8 | 9 |
10 | 11 | 12 | 13 | 14 | 15 | 16 |
17 | 18 | 19 | 20 | 21 | 22 | 23 |
24 | 25 | 26 | 27 | 28 | 29 | 30 |
31 |
(...), Arduino, AVR, BaRef, Blosxom, Book, Busy, C++, Charity, Debian, Electronics, Examination, Firefox, Flash, Framework, FreeBSD, Gnome, Hardware, Inter-Actief, IRC, JTAG, LARP, Layout, Linux, Madness, Mail, Math, MS-1013, Mutt, Nerd, Notebook, Optimization, Personal, Plugins, Protocol, QEMU, Random, Rant, Repair, S270, Sailing, Samba, Sanquin, Script, Sleep, Software, SSH, Study, Supermicro, Symbols, Tika, Travel, Trivia, USB, Windows, Work, X201, Xanthe, XBee
Recently, I've been setting up awstats, a webserver log analyzer, to
parse my Lighttpd logs. When I'm done, I might post some details on my
setup and the glue scripts used, but for now, I just want to comment on the
right LogFormat
configuration value to use for lighttpd.
When googling around, a lot of people either not mention
LogFormat
at all, or suggest to use LogFormat=1
, which means the
Combined Log Format (CLF). However, lighttpd use a different log format!
In fact, the CLF is very similar to Lighttpd's log format, but
it differs in the second field. In CLF, the second field is the identd
username, which is ignored by awstats. In Lighttpd's format, this is the
virtual host of the current request, which is very relevant if you're logging
multiple virtual hosts to the same logfile. This similarity is th reason that
LogFormat=1
does work for most people, but it's better to use the proper
configuration:
LogFormat="%host %virtualname %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"
I've taken this format string from the only correct posting I found online,
but the forum of that posting seems to interpret the %ua
in the last field
as a newline (probably u for unicode and a for 0x0a, which is the ASCII code
for a newline...), so it took me a while to realize that it was correct.
Recently I have been doing some Flash debugging for my work at Brevidius.
In a video player we have been developing (based on
work done by Jeroen Wijering) we needed to escape some url parameter,
since our flash code could not be certain what would be in the value (and
characters like & and = could cause problems). The obvious way to do this is
of course the escape
function in ActionScript. This function promises to
escape all "non-alphanumeric characters", which would solve all our problems.
However, afters implementing this, we find that there are spaces magically
appearing in our GET parameters. Upon investigation, it turns out that there
are plus signs in our actual values (it's Base64 encoded data, which uses
the plus sign). However, the escape
function apparently thinks a
plus sign is alphanumeric, since it does not escape it (note that the flash
10 documentation documents this fact). Which shouldn't be a
problem, since a plus sign isn't special in an url according to RFC1738:
Thus, only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL.
(Note that RFC3986 does recommend escaping plus signs, since they might be used to separate variables, but that's not the case here).
However, the urls we generate in flash point to PHP scripts and thus pass
their variables to PHP. Unfortunately, PHP does not adhere to the RFC's
strictly: It interprets plus signs in an url as spaces. Historically, spaces
in an url were replaced by plus signs, while spaces should really be encoded
as %20 nowadays. There is of course a simple way get Flash (or any other
url-generating piece of code) work properly with PHP: Simply encode plus signs
in your data as %2B (which is the "official" way). This makes sure you get a
real plus in your $_GET
array in PHP, and the problem is resolved.
After some searching, and asking around in ##swf
on Freenode, I found
the encodeURIComponent
function, which is similar to escape
, but does
encode the plus sign. If we use this function, we can again send data with
spaces to PHP! And since encoding more than needed is still fine according to
the specs, there are no downsides (except that you need Flash >= 9.0).
So, if you're developing in Flash, please stop using escape
, and use
encodeURIComponent
instead.
I've recently been hacking a bit with Flash (or rather, with Adobe's Flex compiler. This is a freely available commandline compiler, which actually works on Linux as well.
However, out of the box the commandline compiler, mxmlc
, is obnoxiously
slow. It takes nearly 10 seconds to compile a simple Hello, world! example,
and compiling the video player I'm working on takes over 30 seconds. Not
quite productive.
This is a documented "flaw" in the mxmlc
compiler, caused by the fact
that it has to start up a big java program everytime, loading thousands of
classes and because it always recompiles the entire source.
The official solution to get faster compile times is to use the
Flex Compiler Shell (fcsh
), which is included with the Flex SDK.
Basically, it's a caching version of the mxmlc
compiler, that keeps running
in the background and caches compiled files.
fcsh
is intended to be used by IDEs. The IDE starts fcsh
and then
communicates with it through its stdin/stdout. This means fcsh
is really a
simple thing, without any support for listing on sockets or properly
daemonizing.
Using fcsh speeds up the build time from nearly 40 seconds to just a few seconds (depending on how many changes were made). The first compilation run is still slow, of course.
I've been trying to use fcsh
with the ant build system, which makes
things a bit tricky. Since there is no long-running process (like an IDE)
which can keep fcsh
running, this needs some way for fcsh
to be run in the
background, connected to some fifo or network socket so we can start it on the
first compilation run and then connect to it in subsequent compilation runs.
A quick google shows that there are already a few fcsh
wrappers to do this,
some of which intended to work with ant as well. At a quick glance, it seems
that the flex-shell-script-daemonizer seems the most useful one. It is
created in Java and runs as a daemon (unlike the alternatives I saw, which
were Windows-only due to some useless GUI). It has two modes: In server mode
it starts fcsh
and connects it to a TCP network socket and in client mode it
takes a command to run and passes it over the netowrk to the running server.
There is also some stuff to integrate make ant use fcsh
for compiling
actionscript files. However, this requires changing the build methods and
stuff in build.xml, which I don't want to do (I'm trying to minimize my
changes to the sources, since I'm modifiying someone else's project).
So, I created a small shell script that can be used as a drop in replacement
for mxmlc
. It simply takes joins together all its arguments to a single
command and passes that to the flex-shell-script-daemonize command. If the
fcsh
daemon is not yet running, it automatically starts it (and does some
half-baked daemonization, since neither fcsh
nor
flex-shell-script-daemonizer does that...)
While writing the script, I also found out that the fcsh
"shell", doesn't
have any way to quote spaces in arguments in the command (at least, I couldn't
find any and there was no documentation about it). This means that, AFAICS,
there is no way to support spaces in paths. How braindead is that... I guess
FlexBuilder, the official IDE from Adobe doesn't use fcsh
after all and
instead just includes the compiler classes directly...
Of course, just as I finished the script, I encountered flexcompile,
which basically does the same thing (and includes the network features of
flex-shell-script-daemonizer as well). It's written in python. However, it
does require the path to flex and the mxmlc
part of the path to be passed in
as arguments, so it's not a 100% drop-in replacement (which I needed due to
the way the build system I was using was defined). Perhaps it might be useful
to you.
Anyway, here's my script. Point the JAR
variable at the jar generated by
flex-shell-script-daemonizer and FCSH
to the fcsh
binary included with the
Flex SDK.
#!/bin/sh
# (Another) wrapper around FlexShellScript.jar to provide a mxmlc-like
# interface (so we can just replace the call to mxmlc with this script).
# For this, we'll have to call FlexShellScript.jar with the "client"
# argument, and then the entire mxmlc command as a single argument
# (whereas we will be called with a bunch of separate arguments)
JAR=/home/matthijs/docs/src/upstream/fcsh-daemonizer/FlexShellScript.jar
FCSH=/usr/local/flex/bin/fcsh
PORT=53000
# Check if fcsh is running
if ! nc -z localhost "$PORT"; then
echo "fcsh not running, trying to start..."
# fcsh not running yet? Start it (and do some poor-man's
# daemonizing, since the jar doesn't do that..)
java -jar "$JAR" server "$FCSH" "$PORT" > /dev/null 2> /dev/null <
/dev/null &
# Wait for the server to have started
while ! nc -z localhost "$PORT"; do
echo "Waiting for fcsh to start..."
sleep 1
done
fi
# Run the client. Note that this does no quoting of parameters, since I
# can't find any documentation that fcsh actually supports that. Seems
# like spaces in path just won't work...
# We use $* instead of $@ for building the command, since that will
# expand to a single word, instead of multiple words.
java -jar "$JAR" client "mxmlc $*" "$PORT"
Since a while, I've been using git-buildpackage to manage the Debian packaging for OpenTTD. This offers the option to sign the tags created by git-buildpackage, which is of course cool. However, I've found that I always need to load my GPG key into my gpg-agent first to make that work. If I didn't, I'd see the following:
You need a passphrase to unlock the secret key for
user: "Matthijs Kooijman <matthijs@stdin.nl>"
1024-bit DSA key, ID 8A2FAFBC, created 2005-05-27
gpg: cancelled by user
gpg: skipped "Matthijs Kooijman <matthijs@stdin.nl>": bad passphrase
gpg: signing failed: bad passphrase
error: gpg failed to sign the tag
error: unable to sign the tag
This also occurs when calling git tag
directly, but not when calling gpg
--sign
.
I've finally done some research to this, and found that this can be solved by
setting the GPG_TTY
variable in every shell. According to the gpg-agent
manual, you should add the following to your ~/.bashrc
:
GPG_TTY=$(tty)
export GPG_TTY
And whaddayaknow, it works! Sounded pretty silly to me (why not just use
$tty
directly?), but it actually makes some sense. First of all, $(tty)
is
not an evironment variable, but an invocation of the tty
command.
Secondly, it seems of course silly to put this in the environment when the
passphrase-asking code could just find out the current tty by itself. However,
it seems that it is actually gpg-agent itself that initiates the passphrase
dialog. When I unset $GPG_AGENT_INFO
in my terminal, git tag
asks for a
passhprase normally (but just with an "Enter passphrase:
" prompt, instead
of the full-screen curses-based prompt I get normally).
This probably means that whatever code is calling gpg-agent from within
git tag
, is sending along the value of the GPG_TTY
variable (or perhaps
simple all of the environment?). I guess the gpg
command automatically sets
the GPG_TTY
variable internally, which is why using gpg
directly just
worked.
This also explains why, after entering my passphrase, my key was added to the agent as well.
This morning, I was trying to enable X forwarding, to run applications on my
server (where I have GHC available) to my local workstation (where I have
an X server running). The standard way to do this, is to use SSH with
the -X
option. However, this didn't work for me:
mkooijma@ewi1246:~> ssh -X kat
Last login: Wed May 20 13:48:13 2009 from ewi1246.ewi.utwente.nl
matthijs@katherina:~$ xclock
X11 connection rejected because of wrong authentication.
Running ssh with -vvv showed me another hint:
debug2: X11 connection uses different authentication protocol.
It turned out this problem was caused by some weird entries in my
.Xauthority
file, which contains tokens to authenticate to X servers. The
entries in the file can be queried with the xauth
command:
matthijs@katherina:~$ xauth list
#ffff##: MIT-MAGIC-COOKIE-1 00000000000000000000000000000000
#ffff##: XDM-AUTHORIZATION-1 00000000000000000000000000000000
localhost/unix:10 MIT-MAGIC-COOKIE-1 00000000000000000000000000000000
(I replaced the actual authentication keys with zeroes here). The last entry is the useful one. It is the proxy key added by ssh when I logged in. That is the one it should send over the ssh forwarded X connection (where ssh will replace it with the actual key, this is called authentication spoofing). However, I found that for some reason X clients were sending the XDM-AUTHORIZATION-1 key instead (hence the "different authentication protocol" message), causing the connection to fail.
I've solved the issue by removing the #ffff##
entries from the .Xauthority
file (but since I couldn't just run xauth remove #ffff#
, I turned it around
by readding only the one I wanted:
matthijs@katherina:~$ rm ~/.Xauthority
matthijs@katherina:~$ xauth add localhost/unix:10 MIT-MAGIC-COOKIE-1 00000000000000000000000000000000
I'm still not sure what these #ffff##
entries do or mean (I suspect xdm has
added them, since I am running xdm on this machine), but I've made inquiries
on the xorg list.
As a last note: If you want to use X forwarding and enable the GLX protocol
extensions for OpenGL rendering, you need to disable security checks in
the X forwarding, by running ssh -Y
instead of ssh -X
.
Recently, I switched Window manager again. I still haven't found the window manager that really works for me (having run Ratpoison, Enlightenment, ion3 and wmii in the past), perhaps I should write my own. Anyway, a while back a friend of mine recommended XMonad, a Haskell based window manager. Even though having a functionally programmed window manager is cool, I couldn't quite find my way around in it (especially it's DIY statusbar approach didn't really suit me).
While googling around for half-decent statusbar setups for XMonad, I came across the Awesome window manager. It's again a tiling window manager, as are most of the ones I have been using lately, but it just made this cool impression on me (in particular the clock in the statusbar, which is by default just a counter of type time_t, ie number of seconds since the epoch. Totally unusable and I replaced it already, but cool).
Awesome allows you to tag windows, possibly with multiple tags and then show one tag at a time. (which is similar to having workspaces, but slightly more powerful). But, one other feature which looks promising, is that it can show multiple tags at once. So you can quickly peak at your IRC screen, without losing focus of your webbrowser, for example. I'm not yet quite used to this setup, so I'm not sure if it is really handy, but it looks promising. I will need to fix my problems with terminal resizes and screen for this to work first, though....
In the last few weeks I've been playing around with embedded Linux. Bert, a flatmate, brought two old thin clients from his work. They're bot Wyse WT8440XL devices, containing a 400Mhz K6-II processor, some peripherals, and a "DiskOnChip". This last one is flash device that is used instead of a hard disk. It's soldered directly onto the mainboard and as far as I can tell, it interfaces with the system directly into the memory controller, similar to what a BIOS chip does. Anyway, this means I have no hard disk to put a full OS on, but only 96MB of flash storage (And yes, I know I can just plug in some USB storage, which is even what I do for testing, but not using the internal flash chip wouldn't be half as much fun :-p)
It feels a bit weird to write about windows, but perhaps other people have some use for this.
At Inter-Actief, we normally log in with non-privileged accounts. Every now and then, we (the sysadmins) want to do something with privileges. For this, we use the runas utility, which is included with Windows XP by default. It runs a given command with different privileges (after entering a password, of course).
One of these things is browsing the file system and copying files. In other words, you want to get an explorer window with privileges. Simply running
runas.exe /user:Administrator explorer
Doesn't work. This will notice there is already a running explorer process and let that existing process open a new window instead (which will not have privileges).
Previously, we misused Internet Explorer for accomplishing this. Internet Explorer is really just another face of explorer (But with a different process name), so if you would start a new instance of Internet Explorer and navigate to "C:" instead of an url, you'd get a perfectly usable explorer, running as Administrator.
For example,
runas.exe /user:Administrator "C:\Program Files\Internet Explorer\iexplore.exe C:\"
gets you a fancy new explorer window, but only when using IE6. Internet Explorer 7 dropped all those explorer-like features (which is a good thing, btw), but now this trick doesn't work anymore.
Some googling around gave me Aaron Margosis' blog who describes some trick with setting a "run explorer windows in separate processes"-flag for the Administrator user.
This is all nice and fancy, but I won't be logging into every client here to set that bit, only to do it again (or more annoying, forget it) whenever we reinstall a machine.
Fortunately, badri commented another trick: Use the /separate
flag to
explorer. This forces explorer to run as a seperate process, ignoring any
already running instances. This gives a command line such as:
runas.exe /user:Administrator "explorer.exe /separate,C:\"
Be careful to include the ",
" there, between /separate
and the path. I'm
not sure why it is there, but it doesn't work without.
This ended up as bit longer post than I intended, but at least I'm writing again.
For the live roleplaying game Symbols, I've been learning a bit about fonts and how to modify them. In the game, we use different fonts to represent different languages. These fonts are not "normal" fonts, but fonts with symbols instead of letters. This means that a text in such a font is not readable, unless you have the corresponding key that maps each symbols back to a letter.
For this, a number of fonts was found on the Internet. These fonts had a symbol for each letter, but had an important shortcoming: They had no accents. So when a text contained, for example, an é it would be displayed as an empty box, or not at all. This was previously solved by replacing all accented letters with their normal equivalents. This was a manual process, which makes it time consuming and error-prone.
I decided to explore FontForge, an open source font editing program to add accents to these fonts. The program turned out to be well-suited to the task.
I first, manually, made all accented characters references to their unaccented equivalents (instead of empty glyphs). Then I created a grave, acute, dieresis and circumflex (yay for my Wacom tablet). Then, I copied references to these into all the accented characters (all the other accented characters simply stayed unaccented). After some manual positioning, the new font was done.
Now, for the actual topic of this post. I now had one font that had the proper references to have all accented characters. I planned on copying a big part of the FontForge .sfd file from that font to all the others, so they would automatically define all accented characters as well.
But, it turned out that FontForge uses the glyph number (based on the order in which the glyphs are in the file) instead of the proper (Unicode) character name/number to save references. Since these glyph numbers were not consistent among the fonts I had to modify, this got completely messed up.
So, I tried the scripting approach. Fortunately, FontForge is easily
scriptable, in python as well as in some built in language. I opted for the
latter and experimented with scripting. By doing something like the following,
I could create the â automatically (I had to apply the fix in this
post before CopyReference()
would work as expected).
Select('a')
CopyReference()
Select('acircumflex')
Paste();
Select('circumflex')
CopyReference()
Select('acircumflex')
PasteInto();
But, I didn't feel like writing hundreds of these lines manually, so I also wrote a Perl script to extract all references from an existing font file (the one I created manually before). In hindsight, it might have been faster to have done it manually, but now I have this pretty script:
#!/usr/bin/perl
# Usage: pipe in a sfd through stdin.
# ./parse.pl < Guardion/Guardion.sfd > fix.ff
while(<STDIN>) {
if ( /^StartChar: (.*)$/ ) {
$code = $1;
push(@bases, $code);
}
if ( /^Refer: ([0-9]*)/ ) {
push(@{$uses{$1}}, $code);
}
}
while (($key, $value) = each(%uses)) {
foreach $other (@{$value}) {
print("Select('$other')\n");
print("Clear()\n");
}
}
while (($key, $value) = each(%uses)) {
print("Select('$bases[$key]')\n");
print("CopyReference()\n");
print("SelectNone()\n");
foreach $other (@{$value}) {
print("SelectMore('$other')\n");
}
print("PasteInto()\n");
}
You simply give it an sfd file to parse and it outputs a script that will reproduce all the references from a given font in another font (using File -> Execute Script...). It does not do any position of referenced glyphs, so I still needed to do the positioning of accents manually
I'm using subversion to manage code for Brevidius, the company I work for. In the code, I use doxygen to document the code. Because I do not care to regenerate the documentation after every change I make, I use subversion's post-commit hook to regenerate the documentation on every commit.
Because the generation of documentation takes long, I added a & to the call to doxygen-post-commit-hook in my post-commit file. This was supposed to put the doxygen generation in the background, so my svn client wouldn't take so long for every commit. This didn't work at all.
The command was properly backgrounded (any commands I put after the call to
doxygen-post-commit-hook were executed quickly), but ps aux --forest
showed
that although doxygen-post-commit-hook was indeed no longer a child process of
post-commit, post-commit was now a defunct process.
It turns out that just backgrounding the command is not enough, one should also redirect inputs and outputs (perhaps not all three, but that works at least). So, calling doxygen-post-commit-hook is now:
/home/data/svn-hook-scripts/doxygen-post-commit-hook "$REPOS" "$REV"
"classbased cms/Doxyfile" "/var/www/doxygen/Brevidius" < /dev/null >
/dev/null 2> /dev/null &
This issue is also referenced in an subversion mailing list, though not directly.
For those who are interested, here is the doxygen-post-commit-hook too.
#!/bin/sh
# Arguments are REPOS, REV, DOXYFILE, OUTPUT_DIR, in that order. DOXYFILE is
# the path inside the repository (no leading /).
REPOS=$1
REV=$2
DOXYFILE=$3
OUTPUT_DIR=$4
TMPDIR=/tmp/doxygen
EMPTYFILE=$TMPDIR/empty
svn export file://$REPOS -r $REV $TMPDIR
echo > $EMPTYFILE
cd "`dirname "$TMPDIR/$DOXYFILE"`"
echo -e \
"CASE_SENSE_NAMES=YES\nOUTPUT_DIRECTORY=$OUTPUT_DIR\nHTML_HEADER=$EMPTYFILE\nHTML_FOOTER=$EMPTYFILE" \
| cat `basename "$DOXYFILE"` - | doxygen -
rm -r $TMPDIR