Like Diogenes¹ I’ve been wandering around from Puppet user to Puppet user asking about the questions I brought up in the first post in this series. It seemed like such a sub-optimal hole in the workflow for the standard use case, I couldn’t believe others weren’t seeing and addressing it with some best practice I hadn’t heard about yet.

One of the people who was kind enough to suffer my investigation was was Eric Shamow, Manager of the System Operations Group at Advance Internet. He even tolerated this during the Q&A for his excellent talk at the PICC 2011 conference.  Eric introduced me to Nigel Kersten at Puppet Labs who graciously agreed to participate in the discussion about workflow Eric and I were having.

One of the things that came out of that discussion² was an idea that came the closest to any I’ve heard for addressing my workflow concerns. I’d like to tell you about it now by paraphrasing what Nigel suggested in my own words. Any errors in the following are mine, any cool ideas found there should be attributed to Nigel.

For this explanation, I’m going to bring back some of the stellar diagram artwork from the first post in the series. My artistic skills haven’t really improved since that post, so apologies in advance.

The Big Idea

Let’s review the cast of characters from our first post:

We have a web server whose configuration is managed by Puppet. The files seen and served by the Puppet server are all kept in a version control system of some sort.

Now let me introduce you to the new members that makes this all work:

Nigel calls these “development Puppet servers” but I’m more partial to the name “personal Puppet servers.” because I think it makes the concept even clearer. Here’s the idea:

Everyone on the team gets their own personal Puppet server. When a team member wants to work on the web server’s configuration, she:

1. makes a clone of the main Puppet server’s node configuration (e.g. its /etc/puppet/modules directory).
2. re-homes the Puppet client on the web server to point to her personal Puppet server.
3. works on the web server’s configuration as it is kept on her personal Puppet server, changing it, pushing it to the web server and testing it until it is just the way she wants it.
4. pushes the changes she made on her personal Puppet server to the main Puppet server³.
5. re-homes the web server’s Puppet client back to the main Puppet server.

Let’s see this process in pictures before we look at this solution in more detail:

First we clone the configuration information from the main Puppet server to our personal Puppet server. Then we tell the Puppet client on the web server to look to our Puppet server instead of the “real” one for its configuration.

[Now pretend I've inserted a picture here of a sysadmin/devop furiously editing the web server config on a personal Puppet server, pulling it from that server to the web server, reloading the config on the web server, and repeating until happy with the changes.]

Finally, we merge our changes back to the main VCS repository, re-home the web server back to the “real” Puppet server, and everything is copacetic:

Pros and Cons

So what do I like so much about this solution?

1. Once you’ve re-homed to your personal Puppet server, you can change, push, and test your new config in a very tight, fast, and lightweight loop. Given this addresses my major concern, this solution is already a win with the rest being gravy.
2. That loop uses the same process to generate and distribute configs as the “normal” Puppet process. If your main Puppet server’s config does a bunch of fancy templatey, databasey, Ruby mangley⁴ things to produce the final web server configuration, they will all be present and tested during the “editing/development” process. This means you are much less likely to have to debug the Puppet part of the config after it gets to the real Puppet server⁵.
3. The editing loop can be customized to the person who is doing the changing. If they want to be highly fastidious in their use of a VCS  locally on their personal Puppet server,  topic-branching, merging and rebasing like a madman,  great. If they want to skip that and just change files willy-nilly, fine. Both types of people will have to “do the right thing” when they merge their changes to the main VCS repository.
4. There is no longer a need to have a “personal” environment⁶. Instead of switching in and out of /etc/puppet/mypersonalenvironment/modules (and keeping that in sync with the target directory), all of the team members can make their changes in their target environment (/etc/puppet/modules, or /etc/puppet/development/modules, etc.) on their local machine. This is one less context shift during the process and one less thing each person has to keep alive in their head⁷.

What’s not to love about this idea?

1. More moving parts. You have to be able to stand up more Puppet servers easily (though Nigel points out that standing up a Puppet server for just a few nodes, providing all of the Puppet dependencies are installed, is just a matter of typing:
   puppet master –verbose –no-daemonize –<other settings> ).
2. Your main Puppet infrastructure and configs have to be engineered with portability in mind. I was hand-waving when I said the personal Puppet server would behave in the exact same way as the production master. That’s only true if you haven’t created a configuration that isn’t tightly tied to a specific machine (e.g. the database backend only accepts connections from localhost). If a clone of the configs would break on any every machine besides the master, that’s a problem.
3. The other place I put on a pair of rose colored glasses was when I asserted that you would not have to debug the Puppet config when it got the real server. There are lots of issues that could crop up when a config is deployed at scale that you wouldn’t necessarily see with one Puppet client or working on a single web server. You may make a change that ripples out to more machines than you anticipate⁸ I think this proposed setup helps eliminate one tier of potential failures, but there’s always another one above it.
4. This idea doesn’t necessarily remove the VCS-related complexities I whined about in my last post. Merging, branches, etc, all may still be a fun ball of wax. Similarly, there’s no real protection between two people making mutually incompatible changes to different parts of the web server. Not really news, but I thought I should cop to it anyway.
5. There’s a definite lack of granularity here. Once I re-home a machine to my personal Puppet server, the entire machine stops getting configuration updates from anyone but me. If I’m cool and refresh my Puppet server’s clone periodically, that helps some, but we’re still fundamentally dealing with things on a per-machine level. It would be nice to be able to re-home just a portion of a machine’s configuration. I’ve never heard of a Puppet client receiving information from more than one Puppet server, but I bet you could fake it. One possible but potentially icky method would be to go back to something similar to the ideas in the Pro Puppet book. You could construct a directory tree from the amalgamation of several different VCS repositories. My assertion is it would be better to build this functionality into the Puppet layer vs. trying to cobble together something using a VCS tool.
6. There are potentially some security questions around SSL certificates. See the details section below.
7. Someone might forget and (or intentionally) leave the web server re-homed to the wrong place. Ideally you would have monitoring in place to scream should this happen.

Details, Details

There are a number of implementation details, some obvious, some a little more hidden that need to be solved to put a scheme like the one I’ve described together. For example, Puppet’s SSL/Certificate authority stuff is going to have to come into play. In the initial setup with only a single Puppet production master, all of the clients have presumably already been authorized to use that server. That’s the whole puppetca -s stuff we go through to have the client’s certificates signed for the server. You have to handle a similar situation for each  personal Puppet server⁹.

There are other questions, like where do you host the personal Puppet servers (on an individual workstation?, a new VM you spin up for a person in a central farm?, perhaps on your laptop using Vagrant, etc.?), how do make sure you have the necessary bits on disk for Puppet to run?¹⁰, are the firewall rules ok? (which machines are allowed to talk to which Puppet servers) and so on. I’m sure you can think of more.

And Now Ladies and Gentlemen, The Future

There are three sets of futures to discuss:

1. My future — we’re going to try using this setup on our initial deploy at $WORK unless we hit some showstopper on the way. I can try and report back on our experiences if anyone would find that valuable.
2. Puppet’s future — discussing this in greater depth with Nigel and Eric, it seems like there is a healthy eagerness on the part of Puppet Labs and the rest of the community to wrestle with problems like these. I’ve used the word “workflow” throughout these posts, but I’m painfully aware that I have just touched the very surface of the topic. I believe the tool has the potential to evolve to make some of the sorts of stuff I’ve discussed here easier to pull off¹¹. And further workflow-related additions are on the way.
3. Our future together — I realize that these three blog posts might have an “asked, then answered” story arc, but I think the questions around how you use a configuration management tool like Puppet within a larger context  are far from being exhausted. I hope you’ll comment here or keep bringing questions like this to the mailing lists for those tools. Please tell me how you are addressing these sort of issues.

1. though I’d like to think a little nicer. He seemed like quite a character.
2. and it is possible I’ll post more here from it in future episodes
3. or more precisely, to the VCS system that feeds the main Puppet server
4. ok, I will stop now
5. I realize there is a bit of handwaving here, more on that in the cons section
6. in the Puppet sense of the word. This is essentially a person’s own directory tree within the /etc/puppet module directory
7. and hey, if you like using a ton of personal environments, there is nothing about this solution that precludes you from doing so
8. Luke’s talk here suggests they are working on or are going to work on better tools to let you understand the scope of Puppet config changes.
9. maybe you can make it easier on yourself by using autosign, caveat implementor, or pre-generating certs. I haven’t thought more about this but I wonder if you could perform some slight-of-hand using interface aliases/CNAMES
10. the Pro Puppet book suggests you can manage Puppet with Puppet which sounds peachy to me
11. I like it when a tool makes it easy to “do the right thing,” i.e. follow best practices.

Even though my exploration into the questions I broached in my last post didn’t actually continue with what you will find in this post, I’m going to pretend it did because it makes for a better narrative. Please bear with me.

I’ve been working my way through the very new and excellent book Pro Puppet by James Turnbull and Jeffrey McCune. Given my last set of questions, I was excited to hit the third chapter in the book which is all about workflow, how Puppet gets used with a VCS (git) and all that good stuff. And then I started to read…

Chapter three tells the tale of a standard infrastructure (mail, web, DB servers) managed using separate development, test and production environments–all of this handled by a single Puppet install. And in this happy little world we have three team members: the system administrator¹, the developer and an operator who are all attempting to play nice². Sounds pretty much like your workplace, right?

Ok, so let me see if I can summarize just how the authors propose this all should work. First the prep work:

1. Within the /etc/puppet directory, we have a modules directory for the production environment configs. This directory is made into a git repository³.
2. We clone that repository into (newly made) /etc/puppet/environments/development and /etc/puppet/environments/testing directories⁴. They will be used for the dev and testing environments respectively. Git “remote” references are then added between the repositories to make it easier to move things between them as necessary.
3. Next we create a new “bare” central repository⁵ that will be used as a rendezvous point for the three team members to exchange changes between themselves and with the Puppet server config directory (which will now be checked out from this central repository).
4. Each of the team members is expected to check out a working copy of the central repository⁶ into their home directory, then…

Now the actual work to make an edit, each person will:

1. create a branch in their working copy within which they will make their edits.
2. make the change to a file in that working copy
3. commit that change⁷
4. push that staged commit with the new branch in it up to the central repository
5. on the Puppet server itself, logged in as the puppet user, in the right config directory, use git to check out the the right branch from the central repository into that directory⁸. This check out operation will switch that directory to the branch.
6. run the puppet agent command (maybe in –noop mode to make sure the change really makes sense)

Doesn’t that sound like fun? Does the following quote from the book make it sound any more fun?

(speaking of a second team member repeating the process we just described with his own change…)

This process will switch the current development environment away from whatever branch it was previously on. This could potentially interfere with the work of {the first team member}. If this becomes a common problem, it is possible to set up more environments to ensure each contributor has their own location to test their changes without interfering with others.

So we are at 5 separate git controlled spaces, each with its own state (branch, remote references, etc.) and we’re still bound to bump into our colleagues. On top of that we’ve got a lovely multi-step process after a change is made that the book more succinctly elsewhere describes as:

The overall workflow {the second team member} follows is to push their topic branch to the central repository, fetch the changes in the development environment’s repository, check out the topic branch, then run the Puppet agent against the development environment.

I can’t tell whether to be dismayed more by the number of steps, the possibility for human error , the sheer quantity of git commands, the need to have everyone run something manually on the server as a separate shared user or what. At the very least it appears each person has to keep lots of different sets of context (what branch, what environment, what remote repos, what change, what is it going to effect, and so on) in their head for each change to the environment.

Now, I’m sure that some of this can be ameliorated by writing a number of shell scripts⁹, but boy does it give me the heebie jeebies. I know it certainly doesn’t make me feel any better about the questions I raised in the my last post.

Summary: love the book, dislike this particular solution in it.

Luckily, I did find a better answer…

1. who is talked about using a female pronoun, kudos to the authors!
2. as opposed to some the potential Lord of the Flies scenarios
3. Count along with me boys and girls as we create a number of git-controlled directories/repositories in our journey. This will be number one.
4. That would be git repos 2 and 3.
5. Yup, #4.
6. One copy each, but we will just follow one ball at time so call this git area #5.
7. since we are in git-land, the git commit command is perhaps better described as “staging” the change.
8. yes, you heard that all correctly, hope you got it all right
9. I don’t begrudge the authors for not demonstrating that; as an author myself I understand how it sometimes doesn’t make sense to add another layer to an already complex explanation.

I’ve watched the birth and toddlerhood of all of the major configuration management tools (Puppet, Cfengine, Chef, Bcfg2, and so on) and have had the pleasure of knowing and interacting with almost all of their parents over the years. Recently I decided it was high time I get my hands dirty by leading a substantial deployment of one of the tools at $WORK. The first tool I thought I would tackle would be Puppet.

In the process of planning for this effort, I’ve identified what appears to be a fairly large usability/workflow gap in Puppet¹. I’ve tried to talk to a whole bunch of people about how they do things, but as far as I can tell everyone is still making due with fairly rickety rope bridges to get over the gap. It is entirely possible I’ve overlooked something obvious or the problem isn’t as big as the amount of scrutiny I’ve given it. But something in my long-time sysadmin heart tells me we could be doing much better. I’d like to see if I can pose a clear and cogent problem statement here and see if others can help me figure out what I’m missing.

What’s the Problem?

One of the most common scenarios for how Puppet is used seems (to me) to have a “best practices” workflow that is unclear at best and unwieldy at worst. Given how often a sysadmin performs this workflow, I’d really like to know if there is a better way. If there isn’t, I’d like to work with people to invent one. Please read on for the gory details…

The Setup

Let’s see the players in a super simple setup that I think provides a good demonstration of my quandary. Let’s say we have a web server host of some sort² called www configured through a httpd.conf file:

We’ll want this configuration file to be managed by Puppet, so let’s bring in a central Puppet server called puppetmaster.  This server will store a copy of httpd.conf in its local datastore to be served out to Puppet clients:

We’ll assume that there is a Puppet client running on www. It is configured in the default manner to wake up every N minutes³ and ask the server for changes to its config, pull down them down, and implement them. If a change is made locally on www, but the change is not propagated to the Puppet-managed config, it will be overwritten.

It is a best practice for a number of reasons for users to never directly edit the file that the Puppet server sees on disk. Instead, we’ll want the Puppet server to work from a copy of the file that is stored in some version control system. Indeed, even Introduction to Puppet on the official website shows this picture:

with nary a comment about the SVN box, probably because the value of using a VCS is well accepted. I’ll add a similar version control host to my example. The version control system could actually reside on the same host as any of the other components we’ve described but I’m going to break it out for simplicity (and best practices) sake. Let’s call it vcs and add it to the diagram we’ve been building:

As far as I can tell, there’s nothing new or remotely special about the setup I’ve described. Now let’s start to build the puzzle.

Question 1: Where Should You Edit the Web Server’s Config File?

Ok, it’s showtime! Let’s change the web server’s running configuration. For yuks, let’s assume this will be a substantial multi-line change and it is one I’m doing by hand⁴. Changes like this are often iterative, meaning I make a change, test it, fix any problems, test, fix more, and so on.

So where do I do this “iterative development” of the config? Here are two choices:

1. Right there on the web server, in situ where the config file lives.
   First we stop Puppet (don’t want our changes overwritten by the regular update cycle!⁵ ), then we edit. Once we are done editing, we then have to get the changed file into our version control repository and then from there into the Puppet repository⁶. At that point we can restart Puppet⁷.
2. In some working directory, someplace (doesn’t matter where), that is “checked out/cloned” for editing from the version control system.
   Each time I want to test a change, I have to commit the changes back to the VCS and either wait for it to propagate back to the web server or somehow “push” the process to make it happen sooner than the usual interval. Once it gets to the server, either Puppet  or I have to tell the server to reload its config. I can then test if this change was correct. If not, edit the local copy, commit it again. Rinse and repeat.

Let’s examine the pros and cons of each of these options.

The In Situ Option

For the in situ option, the development process has a little less friction. Make a change, tell the web server to load the config. Test. Make another change, reload. It works. Check changes back into VCS so the correct version will be managed via Puppet when you’ve got a version of the config you like. Here’s a diagram of this option:

The places you get into trouble with this option are:
a) remembering to put the config back into Puppet
b) making sure puppet doesn’t stomp on the work you are doing while you are doing it
c) you’ve introduced a change in the “control flow” path that the web server config file took to get to the machine. Because we edited it in place, that change didn’t get to the machine via Puppet. As a result it bypassed any of the processing Puppet might have done to select/produce that file (e.g. templating).

Each of these trouble areas has its own peril:

• Humans are fallible, so for a) you run the risk of version skew if you forgot to check your changes back in⁸.
• Having to stop Puppet on a machine has the obvious side effects if it is not restarted. But what about the case where your colleague has changed another file in Puppet which isn’t getting to that machine because you disabled Puppet there?
• Having a config file get to a machine in two different ways is bound to force you to eventually debug an issue with “the road less traveled,” I’ll lay money on the table that at some point you’ll find yourself  puzzling over why a Puppet-sourced change to a file didn’t do something you expected.

The Version Control System Option

Now let’s look at the other option where we make all of our changes in some working copy and then push them to the VCS and then push them into Puppet so Puppet can get them on to the client. I intentionally used a run-on sentence to demonstrate the multi-step complexity of this option. I call this the “spin the wheel” option because you essentially have to spin the entire giant wheel to get each (even trivial) change to boomerang back to you and be put in place on the web server:

To go this route, you need to have a method to quickly:

1. get the VCS version of the file into the Puppet datastore (more on this in a moment) and
2. get the Puppet client’s config pull to take place out of its normal sequence (not hard for a single machine, but it does need to only happen after #1 has successfully completed)⁹

One hidden drawback of this method which was pointed out to me by a Puppet Labs person is that your VCS changelog tends to get cluttered with trivial changes. If one of the goals of using a VCS in the first place is to have a change log where it is easy to determine what changes happen to your infrastructure over time, it gets annoying when the majority of log messages say “Typo…Typo…Damn…Fixed a spelling error…Almost got it…Nope, not yet…¹⁰

Another interesting consequence of this method comes when we’re working to develop a configuration which is not as localized as the one used in our example so far. Imagine we’re developing a configuration for a service that runs on all of your machines. You have to make sure that the “in development” changes aren’t prematurely propagated out to all of the machines. There are a number of ways you can do this, for example:

• change the Puppet config so the modified file is targeted to be specific to the machine, then change it so it becomes applicable to a larger scope. This means you are changing not only the service’s config, but also the configuration management system’s config. Best keep your wits about you.
• a less “one-off” variation of the previous idea is to use Puppet environments to label the machine you are working on as “special.” You then develop your configs first in a “special” place before moving them over to the standard location. You can also create a similar scheme using either Facter or an external node configuration. All of these methods force you to engineer your Puppet infrastructure to take this workflow into account¹¹. But you have to have thought of this before hand.

Question 2: How Do Configs Get from the VCS Respository to the Puppet Datastore?

We almost touched on this in the last section, but I want to shine a more direct light on this dark little corner as well.  This is the question of how you control the flow of data that resides in your version control system to the place where the Puppet server sees the data and can serve it to its clients. I’ve heard of three different ways people are doing this:

1. A job gets run periodically on the Puppet server to “check out” any changes to the VCS repository. The Puppet server either looks into this working copy for its files or a subsequent part of the job copies¹² the whole or just changed data into the Puppet server’s datastore.
2. The VCS has a post-commit hook added to the repository so that when a magical commit takes place (e.g. the change is tagged in a particular way or the commit happens to a specific branch) the VCS copies the appropriate files to the Puppet datastore.
3. The Puppet server itself gets taught how to be a client of the VCS system and to look right into the VCS tree (e.g. through a web interface via HTTP) for files with specific branches or tags each time it runs. An alternative version of this is to have the Puppet server consult a config file (e.g. that has version numbers in it) which describes the latest version of a branch of the VCS repository. If the config file holds a version number later than the one in the Puppet repository, a check out is performed.

As you can imagine, things get trickier when we try to run the options from Question #1 into the options for Question #2 and we don’t quite get the candy bar we’d hoped. Clearly some of the options for how the data gets from the VCS into Puppet are more amenable to “I just made a change, do it now!” than others.

So, What’s the Right Thing To Do?

Here’s the scenario we’re describing: “You have a config file you need to iteratively change. This config file is kept in a version control system and managed by a configuration management system like Puppet. How do you do it?”

Some of the ideas we discussed above seem more or less “hackish” to me, but none of them really describe a workflow I’m particularly enamored by. I assert that the scenario I described above is one of the most common things you would want to do with Puppet. Given that, I really want to be enamored by the workflow because it is something I’m going to have to do all the time. How can we make this better?

I’ve been pondering this question for a number of months. Here are a few larval ideas:

1. closer integration between Puppet and (some abstract layer on top of the more popular VCS systems). There’s a lot that could be done if a Puppet server could have easy read/write access to the VCS.
2. create a way for a Puppet client to have a different relationship with the server than the current dom/sub paradigm where the client meekly says “Got anything for me? and the server says “Yes, and you will take it, and use it to overwrite what you have.”  This would be best if this different relationship could be temporary and just for a single file or directory. That way you may be able to say “take this web server config file and treat it specially. The client version will eventually be considered the canonical one. Scoop it up when I say I’m ready for it to return to the server.” This is one of the benefits of #1 above.
3. another, perhaps simpler twist on the former item, provide a way to just a “lock this file from Puppet propagated changes” would help. It would certainly be nicer than “I’m taking Puppet down on this machine to avoid having my changes overwritten.”

I don’t think I’ve gotten close to licking this problem with these ideas, so I’d really like to discuss this with people who want to grapple with the same problem. Please feel free to comment on this post or get in touch with me via the comment form here on the site. Let’s talk about it.

The plot thickens… see the next post in this series.

1. and most other config mgmt tools, so even if you use another config mgmt system, I’d still love your input
2. for the purpose of this discussion, it does not matter whether or not this web server is considered production or dev, though if it is the former this question becomes a bit more interesting.
3. or the Puppet client is run every N minutes from cron
4. vs. some automated generation of that configuration
5. we could play beat the clock, but that seems treacherous. It causes people to have to pay attention to an unnecessary time limit. If they get up to pee, they risk  having to revert their changes from the backup copy Puppet leaves behind.
6. as an aside, bcfg2 handles this situation really nicely. It knows how to specify that the changed copy on local disk should now be considered the true version and to suck that change back into the bcfg2 configuration with a minimal amount of effort.
7. you did remember to restart it, right?
8. yes, you can and probably should run some sort of Puppet dry-run report that would tell you if you blew it, but why set up a situation that is bound to fail due to human error?
9. however, it is a non-trivial undertaking if you need that push to happen on multiple machines. Yes, something like mcollective makes the multiple machine part easy. The hard part is matching the business logic Puppet will use to determine which machines should get that new config in something else outside of Puppet so the push happens only on the required machines.
10. yes, various VCS systems have ways to bundle/edit commit log messages, but then the burden shifts to the humans to keep their their commit logs tidy or to go back and edit them. And we all know how that story ends.
11. perhaps not such a bad thing
12. cp, rsync, what have you

I feel that I would be terribly remiss if I didn’t point people to Michael Reilly’s Discovery News article Sea Otters, the Cutest Way to Fight Global Warming. It notes a new study mentioned in New Scientist is suggesting Sea Otters have an unexpected and pretty significant role in combatting global warming.

Yet another reason to work towards helping the sea otters.

Warning: dumb joke modified from a dumb joke that used to contain a character of a certain ethnicity. Also note: this is not representative of lawyers in general so please do not sue me.

A lawyer and an SysAdmin are sitting next to each other on a long flight. The lawyer is thinking that SysAdmins are so dumb that he could get an advantage over on them easy…

So the lawyer asks if the SysAdmin would like to play a fun game. The SysAdmin is tired and just wants to take a nap, so he politely declines and tries to catch a few winks. The lawyer persists, and says that the game is a lot of fun. “I ask you a question, and if you don’t know the answer, you pay me only $5; you ask me one, and if I don’t know the answer, I will pay you $500,” he says.

This catches the SysAdmin’s attention and to keep the lawyer quiet, he agrees to play the game.

The lawyer asks the first question. “What’s the distance from the Earth to the Moon?”

The SysAdmin doesn’t say a word, reaches in his pocket, pulls out a five-dollar bill, and hands it to the lawyer.

Now, it’s the SysAdmin’s turn. He asks the lawyer, “What goes up a hill with three legs, and comes down with four?” The lawyer uses his laptop and searches all references he could find on the Net. He sends e-mails to all the smart friends he knows, all to no avail. After one hour of searching he finally gives up. He wakes up the SysAdmin and hands him $500. The SysAdmin pockets the $500 and goes right back to sleep.

The lawyer is going nuts not knowing the answer. He wakes the SysAdmin up and asks, “Well, so what goes up a hill with three legs and comes down with four?”

The SysAdmin reaches in his pocket, hands the lawyer $5 and goes back to sleep.

Don’t mess with SysAdmins!

I’m all for performing service management using mechanisms built-in to a configuration management system (i.e. you change a config file and the config management system automatically restarts the daemon), but occasionally you get into a situation where you want to interactively do the same exact thing on several machines.

For example, I recently wanted to bounce the ClamAV daemon running on four of my incoming mail servers simultaneously. This process would require the exact same commands on all four machines, ideally run at the same time.

Rather than type the same thing four times, let me tell you about a few programs that can help in this sort of situation where you need interactive input to multiplex to different sessions. The one I used for the last example was OSX specific (since most of my time is spent typing at a Mac laptop):

• iTerm – iTerm is a spiffy rewrite of the native OSX Terminal application that has a bunch of cool features. The one I used is called “Send Input to all Tabs” (under the Shell menu). iTerm lets you run multiple sessions in different tabs of the same window. When you turn “Send Input to all Tabs” anything you type in one tab is automatically echoed to every other tab. I ssh’d to each of the mail servers, turned on this feature, and typed the commands I needed a single time, yet all four machines obeyed these commands. If you run OSX and you haven’t played with iTerm, you really should.

Before I discovered this feature in iTerm, I used to use cssh, from ClusterSSH. It opens up a smaller input window and as many other xterms as you need. Anything typed in the input window is echoed to those windows. This program isn’t OSX specific, it can be used any place you can build the right Perl modules. I stopped using it a while back because I had issues with it building under MacPorts (if I built the modules it needed using the Apple-provided Perl libraries, everything was peachy, but building it into MacPorts in an attempt to avoid polluting the OSX install of Perl lead to a script that would segfault upon running).

I recently noticed a new rewrite of the same idea for OSX called csshx. It uses the native OSX terminal vs. X11. I haven’t tried it, but it looks promising. I suspect there are other utilities like this available. I wouldn’t be surprised if there was a way to do this with GNU Screen or tmux. Are there any you use for this sort if interactive multiplexing that I’m missing?

Sorry I left this blog languish for a bit. Let me catch you up on a few things that have happened in my professional life in rough chronological order:

• I was quite surprised (stunned really, ask me about it some time) and tremendously honored to receive the 2009 SAGE Outstanding Achievement Award at the last LISA. For those who were not at the 2009 LISA, you missed my 1985 Sally Fields acceptance speech imitation.
• It was great to have the opportunity to present to the other sysadmins in the ‘hood at the January 2010 BBLISA meeting.
• At a friend’s urging, I ran for and was elected to a seat on the USENIX’s Board of Directors. If you haven’t heard of USENIX before, it is the organization that sponsors the LISA conference (and some other great ones). Come June, I look forward to serving USENIX and the wider sysadmin community in that capacity to the best of my ability. Thanks to those who voted to give me that opportunity.
• I’m also very honored to be asked to give the opening keynote session at the upcoming PICC Conference sponsored by LOPSA-NJ. I’ve also been asked to teach a class there based on this website’s favorite book. PICC is the first regional conference I’ve attended in a while and I think it is going to be great.

That’s what is up with me. How are things with you?

Here’s a quick tip on how to make an fsck (or specifically an fsck_hsfs) run much faster. I learned this as part of debugging some corruption with the backup image on my Time Capsule.

/bin/fsck_hfs -f -d -c {%mem} /dev/rdisk{N}

where {%mem} is ½ to 1⁄3rd the amount of memory you have in your computer. If you need to determine the amount of memory in a machine from the command line, a great blog post on Mac OS X Command Line Goodies suggests you can use:

/usr/sbin/system_profiler SPHardwareDataType | grep Memory

For example, you can use -c 512m if you have a 1GB machine.

and

{N} is the correct device node. This command will show you the list of disk nodes:

diskutil list

For example, you might use /dev/rdisk0s2.

The -d turns on debugging and -f instructs fsck_hsfs to force a repair if it finds any problems.

In case you were curious, the -c setting is the part of the incantation that gives it its real magic. It lets the command cache a considerable amount of information in memory and that speeds up any scan a tremendous amount. If you ever run a disk repair from Disk Utility, you’ll notice the fsck_hsfs command it spawns lacks this option and hence is much slower.

Just had to deal with a total swap out of all of my SSL certificates at work thanks to the ipsCA screwup that also nailed Bob Plankers and Chris Siebenmann and I’m sure lots of other sites (especially .edu). I’m really peeved too, but rather than hissing and spitting like I really feel like doing, let me pass on a few SSL resources that I found useful while debugging the new chained certs we started using:

• Debian’s SSL keys page is a lovely resource for where and how each common open source package keeps its particular SSL-related configs. Your file system locations may vary but this at least gives you a start for the 30+ packages they list
• UnixCraft’s How To Verify SSL Certificate From a Shell Prompt is a swell tutorial on the various openssl command line options you can use to debug certs.
• Paul Heinlein’s OpenSSL Command-Line HOWTO is a great HOWTO for all sorts of openssl operations.
• The OpenSSL verify manual page lists the error messages you might get when attempting to verify, for example, a chained cert.
• SSLShopper.com, in addition to being a good place to compare certificate authorities, has a surprising amount of really good technical information about all things SSL. For example, The Most Common Java Keytool Keystore Commands and The Most Common OpenSSL Commands are genuinely useful.

Hope this are helpful to you should you get put in the same bind.

Back in the days of yore (when sysadmins still had to bang two rocks together to do their job), we didn’t have command line history. No, we typed every command by hand. If we had to type the same 110-character command-line several times, we typed those 550 characters ourselves, mistakes and all, and by gum, we LIKED it!

And then shells became available with history functionality and, well, we kind of liked that too. And, um, then came along libraries like GNU readline that were written to provide even more pleasant, vizz-eee-ul history recall and editing to anything that presents a command line prompt, and well, ok, we REALLY that too. Because, quite frankly, typing long command line strings over and over again, sometimes because our typing skills aren’t always sharp, is a real drag.

But alas, not every program that provides a command-line prompt or handles interactive input provides this new-fangled user experience (I’m looking at you sqlplus, maybe if scott had a real tiger…). In the past, a sysadmin had to look admonishingly at one of these programs, perhaps working around the lack of interface using something like Expect.

Maybe I’m the last admin on the planet to hear about this idea, but it came as a pleasant surprise to find out that there are a number of programs out there whose whole purpose in life is to add the readline-yumminess to programs that don’t have it already. The first one I encountered was rlwrap by Hans Lub. Now you can type:

$ rlwrap sqlplus scott/tiger@oracle_instance

And you get a reasonable command-line environment even from within sqlplus.

I know I would have offered at least a vestigial body part several times in the past to have the “add readline to command X” super power several times during my career. (apparently this tip has been making people in the Oracle community happy for a while now).

The rlwrap README mentions two other similar packages: rlfe which even ships with the GNU readline distribution and cle. I note that cle hasn’t been touched for over ten years (the last release’s 10th birthday was just over a week ago). I can’t speak to how well rlfe works, but kudos to Lub for continuing to update the rlwrap package.

Hope this tip brings you a similar amount of joy.