Warning: dumb joke modified from a dumb joke that used to contain a character of a certain ethnicity. Also note: this is not representative of lawyers in general so please do not sue me.

A lawyer and an SysAdmin are sitting next to each other on a long flight. The lawyer is thinking that SysAdmins are so dumb that he could get an advantage over on them easy…

So the lawyer asks if the SysAdmin would like to play a fun game. The SysAdmin is tired and just wants to take a nap, so he politely declines and tries to catch a few winks. The lawyer persists, and says that the game is a lot of fun. “I ask you a question, and if you don’t know the answer, you pay me only $5; you ask me one, and if I don’t know the answer, I will pay you $500,” he says.

This catches the SysAdmin’s attention and to keep the lawyer quiet, he agrees to play the game.

The lawyer asks the first question. “What’s the distance from the Earth to the Moon?”

The SysAdmin doesn’t say a word, reaches in his pocket, pulls out a five-dollar bill, and hands it to the lawyer.

Now, it’s the SysAdmin’s turn. He asks the lawyer, “What goes up a hill with three legs, and comes down with four?” The lawyer uses his laptop and searches all references he could find on the Net. He sends e-mails to all the smart friends he knows, all to no avail. After one hour of searching he finally gives up. He wakes up the SysAdmin and hands him $500. The SysAdmin pockets the $500 and goes right back to sleep.

The lawyer is going nuts not knowing the answer. He wakes the SysAdmin up and asks, “Well, so what goes up a hill with three legs and comes down with four?”

The SysAdmin reaches in his pocket, hands the lawyer $5 and goes back to sleep.

Don’t mess with SysAdmins!

I’m all for performing service management using mechanisms built-in to a configuration management system (i.e. you change a config file and the config management system automatically restarts the daemon), but occasionally you get into a situation where you want to interactively do the same exact thing on several machines.

For example, I recently wanted to bounce the ClamAV daemon running on four of my incoming mail servers simultaneously. This process would require the exact same commands on all four machines, ideally run at the same time.

Rather than type the same thing four times, let me tell you about a few programs that can help in this sort of situation where you need interactive input to multiplex to different sessions. The one I used for the last example was OSX specific (since most of my time is spent typing at a Mac laptop):

• iTerm – iTerm is a spiffy rewrite of the native OSX Terminal application that has a bunch of cool features. The one I used is called “Send Input to all Tabs” (under the Shell menu). iTerm lets you run multiple sessions in different tabs of the same window. When you turn “Send Input to all Tabs” anything you type in one tab is automatically echoed to every other tab. I ssh’d to each of the mail servers, turned on this feature, and typed the commands I needed a single time, yet all four machines obeyed these commands. If you run OSX and you haven’t played with iTerm, you really should.

Before I discovered this feature in iTerm, I used to use cssh, from ClusterSSH. It opens up a smaller input window and as many other xterms as you need. Anything typed in the input window is echoed to those windows. This program isn’t OSX specific, it can be used any place you can build the right Perl modules. I stopped using it a while back because I had issues with it building under MacPorts (if I built the modules it needed using the Apple-provided Perl libraries, everything was peachy, but building it into MacPorts in an attempt to avoid polluting the OSX install of Perl lead to a script that would segfault upon running).

I recently noticed a new rewrite of the same idea for OSX called csshx. It uses the native OSX terminal vs. X11. I haven’t tried it, but it looks promising. I suspect there are other utilities like this available. I wouldn’t be surprised if there was a way to do this with GNU Screen or tmux. Are there any you use for this sort if interactive multiplexing that I’m missing?

Here’s a quick tip on how to make an fsck (or specifically an fsck_hsfs) run much faster. I learned this as part of debugging some corruption with the backup image on my Time Capsule.

/bin/fsck_hfs -f -d -c {%mem} /dev/rdisk{N}

where {%mem} is ½ to 1⁄3rd the amount of memory you have in your computer. If you need to determine the amount of memory in a machine from the command line, a great blog post on Mac OS X Command Line Goodies suggests you can use:

/usr/sbin/system_profiler SPHardwareDataType | grep Memory

For example, you can use -c 512m if you have a 1GB machine.

and

{N} is the correct device node. This command will show you the list of disk nodes:

diskutil list

For example, you might use /dev/rdisk0s2.

The -d turns on debugging and -f instructs fsck_hsfs to force a repair if it finds any problems.

In case you were curious, the -c setting is the part of the incantation that gives it its real magic. It lets the command cache a considerable amount of information in memory and that speeds up any scan a tremendous amount. If you ever run a disk repair from Disk Utility, you’ll notice the fsck_hsfs command it spawns lacks this option and hence is much slower.

Just had to deal with a total swap out of all of my SSL certificates at work thanks to the ipsCA screwup that also nailed Bob Plankers and Chris Siebenmann and I’m sure lots of other sites (especially .edu). I’m really peeved too, but rather than hissing and spitting like I really feel like doing, let me pass on a few SSL resources that I found useful while debugging the new chained certs we started using:

• Debian’s SSL keys page is a lovely resource for where and how each common open source package keeps its particular SSL-related configs. Your file system locations may vary but this at least gives you a start for the 30+ packages they list
• UnixCraft’s How To Verify SSL Certificate From a Shell Prompt is a swell tutorial on the various openssl command line options you can use to debug certs.
• Paul Heinlein’s OpenSSL Command-Line HOWTO is a great HOWTO for all sorts of openssl operations.
• The OpenSSL verify manual page lists the error messages you might get when attempting to verify, for example, a chained cert.
• SSLShopper.com, in addition to being a good place to compare certificate authorities, has a surprising amount of really good technical information about all things SSL. For example, The Most Common Java Keytool Keystore Commands and The Most Common OpenSSL Commands are genuinely useful.

Hope this are helpful to you should you get put in the same bind.

Back in the days of yore (when sysadmins still had to bang two rocks together to do their job), we didn’t have command line history. No, we typed every command by hand. If we had to type the same 110-character command-line several times, we typed those 550 characters ourselves, mistakes and all, and by gum, we LIKED it!

And then shells became available with history functionality and, well, we kind of liked that too. And, um, then came along libraries like GNU readline that were written to provide even more pleasant, vizz-eee-ul history recall and editing to anything that presents a command line prompt, and well, ok, we REALLY that too. Because, quite frankly, typing long command line strings over and over again, sometimes because our typing skills aren’t always sharp, is a real drag.

But alas, not every program that provides a command-line prompt or handles interactive input provides this new-fangled user experience (I’m looking at you sqlplus, maybe if scott had a real tiger…). In the past, a sysadmin had to look admonishingly at one of these programs, perhaps working around the lack of interface using something like Expect.

Maybe I’m the last admin on the planet to hear about this idea, but it came as a pleasant surprise to find out that there are a number of programs out there whose whole purpose in life is to add the readline-yumminess to programs that don’t have it already. The first one I encountered was rlwrap by Hans Lub. Now you can type:

$ rlwrap sqlplus scott/tiger@oracle_instance

And you get a reasonable command-line environment even from within sqlplus.

I know I would have offered at least a vestigial body part several times in the past to have the “add readline to command X” super power several times during my career. (apparently this tip has been making people in the Oracle community happy for a while now).

The rlwrap README mentions two other similar packages: rlfe which even ships with the GNU readline distribution and cle. I note that cle hasn’t been touched for over ten years (the last release’s 10th birthday was just over a week ago). I can’t speak to how well rlfe works, but kudos to Lub for continuing to update the rlwrap package.

Hope this tip brings you a similar amount of joy.

I spent the good part of my morning today debugging an issue with a user’s access to a Subversion repository/trac service we provide. That person could not get even a basic svn checkout to work as expected. It would keep throwing errors like this:

Server sent unexpected return value (403 Forbidden) in response to OPTIONS request for {url}

I went up and down the stack, I poked and prodded the two web servers in question (Apache 2 and Lighttpd) and their authentication sources with multiple clients. I watched packet traces. I checked and rechecked every config I could lay my hands on to no avail. It was driving me crazy.

Finally, I realized that the mod_dav_svn file semantics for paths in an svn.access file distinguished between:

[repos:/dir/]
user1 = rw
user2 = rw

and

[repos:/dir]
user1 = rw
user2 = rw

The first only works if you’ve explicitly granted the same permissions to the parent directory as well, i.e.:

[repos:/]
user1 = rw
user2 = rw

while the second “does the right thing” to allow the users to have access to that path component without having to permit the parent directory as well.

Hopes this saves you some time.

’bout time I add some sysadmin content here.
If you ever have a trac instance go boom with an error like “DatabaseError: database disk image is malformed” you can often fix the problem by doing an sqlite dump/reload:

$ cd /path/to/trac-sqlite-db-directory # find the trac.db file
$ su www-data   # switch to the owner of the db file
$ mv trac.db trac.db.damaged
$ sqlite3 trac.db.damaged .dump|sqlite3 trac.db