I feel that I would be terribly remiss if I didn’t point people to Michael Reilly’s Discovery News article Sea Otters, the Cutest Way to Fight Global Warming. It notes a new study mentioned in New Scientist is suggesting Sea Otters have an unexpected and pretty significant role in combatting global warming.

Yet another reason to work towards helping the sea otters.

Warning: dumb joke modified from a dumb joke that used to contain a character of a certain ethnicity. Also note: this is not representative of lawyers in general so please do not sue me.

A lawyer and an SysAdmin are sitting next to each other on a long flight. The lawyer is thinking that SysAdmins are so dumb that he could get an advantage over on them easy…

So the lawyer asks if the SysAdmin would like to play a fun game. The SysAdmin is tired and just wants to take a nap, so he politely declines and tries to catch a few winks. The lawyer persists, and says that the game is a lot of fun. “I ask you a question, and if you don’t know the answer, you pay me only $5; you ask me one, and if I don’t know the answer, I will pay you $500,” he says.

This catches the SysAdmin’s attention and to keep the lawyer quiet, he agrees to play the game.

The lawyer asks the first question. “What’s the distance from the Earth to the Moon?”

The SysAdmin doesn’t say a word, reaches in his pocket, pulls out a five-dollar bill, and hands it to the lawyer.

Now, it’s the SysAdmin’s turn. He asks the lawyer, “What goes up a hill with three legs, and comes down with four?” The lawyer uses his laptop and searches all references he could find on the Net. He sends e-mails to all the smart friends he knows, all to no avail. After one hour of searching he finally gives up. He wakes up the SysAdmin and hands him $500. The SysAdmin pockets the $500 and goes right back to sleep.

The lawyer is going nuts not knowing the answer. He wakes the SysAdmin up and asks, “Well, so what goes up a hill with three legs and comes down with four?”

The SysAdmin reaches in his pocket, hands the lawyer $5 and goes back to sleep.

Don’t mess with SysAdmins!

I’m all for performing service management using mechanisms built-in to a configuration management system (i.e. you change a config file and the config management system automatically restarts the daemon), but occasionally you get into a situation where you want to interactively do the same exact thing on several machines.

For example, I recently wanted to bounce the ClamAV daemon running on four of my incoming mail servers simultaneously. This process would require the exact same commands on all four machines, ideally run at the same time.

Rather than type the same thing four times, let me tell you about a few programs that can help in this sort of situation where you need interactive input to multiplex to different sessions. The one I used for the last example was OSX specific (since most of my time is spent typing at a Mac laptop):

• iTerm – iTerm is a spiffy rewrite of the native OSX Terminal application that has a bunch of cool features. The one I used is called “Send Input to all Tabs” (under the Shell menu). iTerm lets you run multiple sessions in different tabs of the same window. When you turn “Send Input to all Tabs” anything you type in one tab is automatically echoed to every other tab. I ssh’d to each of the mail servers, turned on this feature, and typed the commands I needed a single time, yet all four machines obeyed these commands. If you run OSX and you haven’t played with iTerm, you really should.

Before I discovered this feature in iTerm, I used to use cssh, from ClusterSSH. It opens up a smaller input window and as many other xterms as you need. Anything typed in the input window is echoed to those windows. This program isn’t OSX specific, it can be used any place you can build the right Perl modules. I stopped using it a while back because I had issues with it building under MacPorts (if I built the modules it needed using the Apple-provided Perl libraries, everything was peachy, but building it into MacPorts in an attempt to avoid polluting the OSX install of Perl lead to a script that would segfault upon running).

I recently noticed a new rewrite of the same idea for OSX called csshx. It uses the native OSX terminal vs. X11. I haven’t tried it, but it looks promising. I suspect there are other utilities like this available. I wouldn’t be surprised if there was a way to do this with GNU Screen or tmux. Are there any you use for this sort if interactive multiplexing that I’m missing?

Sorry I left this blog languish for a bit. Let me catch you up on a few things that have happened in my professional life in rough chronological order:

• I was quite surprised (stunned really, ask me about it some time) and tremendously honored to receive the 2009 SAGE Outstanding Achievement Award at the last LISA. For those who were not at the 2009 LISA, you missed my 1985 Sally Fields acceptance speech imitation.
• It was great to have the opportunity to present to the other sysadmins in the ‘hood at the January 2010 BBLISA meeting.
• At a friend’s urging, I ran for and was elected to a seat on the USENIX’s Board of Directors. If you haven’t heard of USENIX before, it is the organization that sponsors the LISA conference (and some other great ones). Come June, I look forward to serving USENIX and the wider sysadmin community in that capacity to the best of my ability. Thanks to those who voted to give me that opportunity.
• I’m also very honored to be asked to give the opening keynote session at the upcoming PICC Conference sponsored by LOPSA-NJ. I’ve also been asked to teach a class there based on this website’s favorite book. PICC is the first regional conference I’ve attended in a while and I think it is going to be great.

That’s what is up with me. How are things with you?

Here’s a quick tip on how to make an fsck (or specifically an fsck_hsfs) run much faster. I learned this as part of debugging some corruption with the backup image on my Time Capsule.

/bin/fsck_hfs -f -d -c {%mem} /dev/rdisk{N}

where {%mem} is ½ to 1⁄3rd the amount of memory you have in your computer. If you need to determine the amount of memory in a machine from the command line, a great blog post on Mac OS X Command Line Goodies suggests you can use:

/usr/sbin/system_profiler SPHardwareDataType | grep Memory

For example, you can use -c 512m if you have a 1GB machine.

and

{N} is the correct device node. This command will show you the list of disk nodes:

diskutil list

For example, you might use /dev/rdisk0s2.

The -d turns on debugging and -f instructs fsck_hsfs to force a repair if it finds any problems.

In case you were curious, the -c setting is the part of the incantation that gives it its real magic. It lets the command cache a considerable amount of information in memory and that speeds up any scan a tremendous amount. If you ever run a disk repair from Disk Utility, you’ll notice the fsck_hsfs command it spawns lacks this option and hence is much slower.

Just had to deal with a total swap out of all of my SSL certificates at work thanks to the ipsCA screwup that also nailed Bob Plankers and Chris Siebenmann and I’m sure lots of other sites (especially .edu). I’m really peeved too, but rather than hissing and spitting like I really feel like doing, let me pass on a few SSL resources that I found useful while debugging the new chained certs we started using:

• Debian’s SSL keys page is a lovely resource for where and how each common open source package keeps its particular SSL-related configs. Your file system locations may vary but this at least gives you a start for the 30+ packages they list
• UnixCraft’s How To Verify SSL Certificate From a Shell Prompt is a swell tutorial on the various openssl command line options you can use to debug certs.
• Paul Heinlein’s OpenSSL Command-Line HOWTO is a great HOWTO for all sorts of openssl operations.
• The OpenSSL verify manual page lists the error messages you might get when attempting to verify, for example, a chained cert.
• SSLShopper.com, in addition to being a good place to compare certificate authorities, has a surprising amount of really good technical information about all things SSL. For example, The Most Common Java Keytool Keystore Commands and The Most Common OpenSSL Commands are genuinely useful.

Hope this are helpful to you should you get put in the same bind.

Back in the days of yore (when sysadmins still had to bang two rocks together to do their job), we didn’t have command line history. No, we typed every command by hand. If we had to type the same 110-character command-line several times, we typed those 550 characters ourselves, mistakes and all, and by gum, we LIKED it!

And then shells became available with history functionality and, well, we kind of liked that too. And, um, then came along libraries like GNU readline that were written to provide even more pleasant, vizz-eee-ul history recall and editing to anything that presents a command line prompt, and well, ok, we REALLY that too. Because, quite frankly, typing long command line strings over and over again, sometimes because our typing skills aren’t always sharp, is a real drag.

But alas, not every program that provides a command-line prompt or handles interactive input provides this new-fangled user experience (I’m looking at you sqlplus, maybe if scott had a real tiger…). In the past, a sysadmin had to look admonishingly at one of these programs, perhaps working around the lack of interface using something like Expect.

Maybe I’m the last admin on the planet to hear about this idea, but it came as a pleasant surprise to find out that there are a number of programs out there whose whole purpose in life is to add the readline-yumminess to programs that don’t have it already. The first one I encountered was rlwrap by Hans Lub. Now you can type:

$ rlwrap sqlplus scott/tiger@oracle_instance

And you get a reasonable command-line environment even from within sqlplus.

I know I would have offered at least a vestigial body part several times in the past to have the “add readline to command X” super power several times during my career. (apparently this tip has been making people in the Oracle community happy for a while now).

The rlwrap README mentions two other similar packages: rlfe which even ships with the GNU readline distribution and cle. I note that cle hasn’t been touched for over ten years (the last release’s 10th birthday was just over a week ago). I can’t speak to how well rlfe works, but kudos to Lub for continuing to update the rlwrap package.

Hope this tip brings you a similar amount of joy.

I spent the good part of my morning today debugging an issue with a user’s access to a Subversion repository/trac service we provide. That person could not get even a basic svn checkout to work as expected. It would keep throwing errors like this:

Server sent unexpected return value (403 Forbidden) in response to OPTIONS request for {url}

I went up and down the stack, I poked and prodded the two web servers in question (Apache 2 and Lighttpd) and their authentication sources with multiple clients. I watched packet traces. I checked and rechecked every config I could lay my hands on to no avail. It was driving me crazy.

Finally, I realized that the mod_dav_svn file semantics for paths in an svn.access file distinguished between:

[repos:/dir/]
user1 = rw
user2 = rw

and

[repos:/dir]
user1 = rw
user2 = rw

The first only works if you’ve explicitly granted the same permissions to the parent directory as well, i.e.:

[repos:/]
user1 = rw
user2 = rw

while the second “does the right thing” to allow the users to have access to that path component without having to permit the parent directory as well.

Hopes this saves you some time.

We just purchased a few new phones to replace our aging tin cans on a string at home. On the sage advice of some other techies I know, we purchased a few phones based on the DECT standard. IMHO, one of the coolest things about the DECT phones is they freakin’ talk to each other.

I know this sounds like one of those “Primitive man fascinated by shiny objects” little exclamations (it sounds so ridiculously naive to even me as I say it, especially given my background), but hey, it is sooo COOL that the headsets sync stuff like phonebook information amongst themselves (I guess via the base station) and I didn’t have to do jack to make that happen besides provide one POTS connection and plug in a few power bricks. I guess I’m so used treating every single handset in my life as a completely separate entity that this idea really impressed me. I wanted to walk into the kitchen and stare reproachfully at all of the appliances there and say “So, what’s wrong with you guys? How come you aren’t sharing information? The phones can do it.”

The one other cool, though less unexpected thing these phones do is treat the caller id information that comes in every time the phone rings as something I might care about. I think it is a no-brainer that I might want to use this information to add information to the phones’(!) phonebooks or block subsequent calls from a certain caller and now that’s available. They also keep their internal clock in sync with the telco’s signals, another nice touch.

I suppose you get all of this cool stuff if you start to play with VoIP (e.g. get into the Asterisk game), but for a consumer experience it was a surprise.

No, really. They talk to each other.

P.S. Warning: DECT phones have some relatively recently discovered security vulnerabilities (I know, what doesn’t? but they were new to me, I hadn’t seen them in my usual infostream). It looks like the best source of info about these vulnerabilities is this site here. See the thesis published (right hand column) for the best description of the issue and a list of phones they tested.

Well, I think the book has hit the big time because I randomly stumbled on not one, not two, but four separate places where people had uploaded my book to some of the grey matter (both kinds) sharing sites. In particular, there’s a Thai-speaking user whose been very busy being naughty. At least one site I found had a very pretty web gui for downloading the book. And while I admire the pluck of people who work that hard to make it easy for other people to engage in less than savory activities, it still leaves me feeling a little uneasy and perhaps a bit sad.

I’ve ready my fair share of articles about how “illegal” distribution can have all sorts of positive effects on book sales and other promotional activity which is why I’m not advocating the villagers grab their pitchforks and smokey torches and join me in storming these sites to demand justice. But you won’t find me giving my blessing any time soon.

If you happened to have stumbled upon this site through a chain of events related to one of these ehemm, distribution points, I’d ask two things:

• Enjoy the book. There’s a Yiddish phrase that roughly translates to “If you are going to do something wrong, at least enjoy it.” I didn’t write the book for the money, I wrote it so people could learn from it and take a little pleasure reading it. (though the small royalty check is kind of nice too).
• Keep karma neutral. You don’t even have to right things by me by buying a copy of my book or attending one of my classes. Just go do something giving to offset the taking. It will make you, me and the person you give to happier.

Thanks.