Quickly Sync JPG mtime with EXIF CreateDate

I borrowed a few things, made a few changes, and now all my photos have the correct timestamp on the filesystem. Hooray!

The script expects to be in the root directory of the album hierarchy, but that is merely because I am lazy. And it was an excuse to learn about how to find where a bash script lives. Also, a refresher handling spaces in paths in bash using read.

Finally, exiftool is awesome! (perl-Image-ExifTool on Fedora 16.)

#!/bin/bash
 
#
# Script to sync the filesystem date with that from EXIF CreateDate field.
#
 
set -e
 
# http://hintsforums.macworld.com/showpost.php?p=523850&postcount=20
self="$(cd "${0%/*}" 2>/dev/null; echo "$PWD"/"${0##*/}")"
 
# http://www.perlmonks.org/?node_id=767176
find $(dirname $self) -name '*.jpg' -print0 | while read -d $'\0' f
do
  exiftool -S -d "%Y%m%d%H%M.%S" -CreateDate "${f}" \
  | awk '{ print $2 }' \
  | xargs -I % touch -m -t % "${f}"
done

Opscode Chef Xtra: Obtaining network interface data

It is somewhat of a challenge to obtain interface information out of the data Ohai makes available for network interfaces. For information only about inet addresses, the following works:

node_addresses = {}
node[:network][:interfaces].each do |iface, vals|
  vals[:addresses].each do |addr, h|
    next unless h['family'] == 'inet' && !addr.match('127.0.0.1')
    iface_data = Hash.new
    iface_data = h.dup
    iface_data['address'] = addr
    node_addresses[iface] = iface_data
  end
end

Getting a little crazy with FileEdit

In case there is any doubt, you can go nuts with Chef::Util::FileEdit. If one is using search_file_replace, internally it is simply:

new_contents << ((method == 1) ? replace : line.gsub!(exp, replace))

Meaning if need be, I can do something silly:

ruby_block 'fix remi.repo' do
  action :nothing
  block do
    f = Chef::Util::FileEdit.new('/etc/yum.repos.d/remi.repo')
    f.search_file_replace(/\$releasever/, '%i' % major)
    f.search_file_replace(/(\[remi\].*?)enabled=0(.*?\[)/m, '\1enabled=1\2')
    f.write_file
  end
end

Lovely! The above is needed as I only want to enable [remi], but not [remi-test] which resides in the same file. (Of course I could just ship my own .repo file, too. Choices, choices.)

Opscode Chef Xtra: A Deletable Template via a Definition

While there is no delete action recognized by the Chef template resource, it is possible to fake it using a definition. For example, a definition for managing a configuration file for the multifaceted DNS server dnsmasq might look like the following:

define :dnsmasq_conf, :enable => true, :template => 'dnsmasq.conf.erb' do
	include_recipe 'dnsmasq'
 
	conffile = "/etc/dnsmasq.d/#{params[:name]}.conf"
 
	if params[:enable]
		template conffile do
			source params[:template]
			owner 'root'
			group 'root'
			mode 00644
			backup false
			notifies :restart, 'service[dnsmasq]', :delayed
		end
	end
 
	unless params[:enable]
		file conffile do
			action :delete
			notifies :restart, 'service[dnsmasq]', :delayed
			only_if {::File.exists?(conffile)}
		end
	end
end

The above definition follows the usual pattern of either being enabled or disabled. The former uses the expected template resource. The latter leans on the file resource to actually handle deletion of the template, taking care to do so only if the file actually exists first. Definitions allow one to combine resources in all kinds of interesting ways.

Opscode Chef Xtra: Achieving Idempotence in execute Resource

The certainty of outcome offered by other Chef resources is notably lacking from the execute Resource, for Chef has no way of knowing the consequences of the provided shell script fragment. Fortunately, it’s possible to ensure idempotent behavior with the appropriate application of care. As an example, perhaps one needs to load several database dump files for an unenlightened Web based application that has not adopted a migrations based strategy.

How to ensure the dump files are loaded in the correct order, but never more than once, while avoiding duplicate work should a dump file fail to import? One can leverage lexically sorted filenames coupled with lock files in such a scenario as demonstrated below.

db_files = ['func.sql', 'schema.sql', 'data.sql.bz2']
 
db_files_with_idx = db_files.inject({}) do |h, f|
	h[f] = "#{h.keys.length.to_s.rjust(2, '0')}_#{f}"
	h
end
 
db_files_with_idx.each do |name, name_with_idx|
	db_file = "/root/db_files/#{name_with_idx}"
	remote_file db_file do
		action :create_if_missing
		source "http://example.com/#{name}"
	end
end

For simplicity, the database files are defined directly in the recipe, but could be factored out in an attribute. A hash is then created — a candidate for refactoring into a library later — that creates filenames for local storage. Afterward, each file is downloaded using the remote_file Resource.

The output would thus be the following:

irb(main):009:0> pp db_files_with_idx
{"data.sql.bz2"=>"02_data.sql.bz2",
 "schema.sql"=>"01_schema.sql",
 "func.sql"=>"00_func.sql"}

Next, the execute Resource is called upon, but as it is not idempotent on its own, the behavior must be supplied:

execute "load dump" do
	action :run
	cwd '/root'
	command <<-EOT
	# script from below
	EOT
	not_if {::File.exists?('/root/db_files/.finished')}
end

A hint of that exists in the not_if block, which checks for a the existence of a lock file signaling successful completion of the resource. However, more is required. In particular, a mechanism is necessary to handle a failure in the middle of an import. (MySQL is the database in question, in this example.)

for f in $(ls db_files | sort) ; do
	ext=$(echo $f | awk -F. '{print $NF}')
	lck="/root/db_files/.seen_${f}"
	# Skip successfully imported dump
	test -f $lck && continue
		case "${ext}" in
		bz2)
			cmd=bzcat
		;;
		*)
			cmd=cat
	esac
	echo "Loading database dump file: ${f}"
	${cmd} /root/db_files/${f} | /usr/bin/mysql -u root my_db
	ret=$?
	if [ $ret -ne 0 ] ; then
		exit $ret
	else
		touch $lck
	fi
done
touch /root/db_files/.finished

First, the filename names of the database dumps are sorted to match the order defined earlier in the recipe and committed to disk by the remote_file Resource. To add some flexibility, the extension is lopped off using awk, allowing for bzip2 compressed dumps.

Next, a lock file unique to each database dump is tested for existence. If the lock file exists, the dump has been successfully imported and is skipped; as a result, an interruption of the chef-client run by failure or user action will not prevent the recipe from picking up exactly where it left off. Only upon successful importation of the data, as signaled by a return value of 0 from mysql, is a lock file written. Otherwise, the script exits with the non-zero error code, causing the execute Resource to raise an exception.

When success is total, the final lock file referenced in the earlier not_if block is created. Thereafter, the resource shall never run again, unless the lock file is disturbed.

The usage of not_if and only_if in Chef resource definitions along with careful sorting and locking inside the execute Resource brings the loving embrace of idempotent behavior to shell script fragments. Of course, the above could be rewritten entirely in Ruby and run from within a ruby_block Resource, but the same concepts apply and as such is left as an exercise for the reader.

Made the switch to Fedora 15 from Kubuntu 10.10

Finally happened. I made the switch to Fedora. I’d been a Kubuntu user since 2006, but since the switch to Pulse Audio I have had serious problems with sound under Kubuntu in 10.10, 11.04, and 11.10 beta. I have no such trouble with Fedora 15.

I’ll likely be moving all systems I manage over from Debian GNU/Linux to Fedora or CentOS in the coming months. (I realize Ubuntu isn’t exactly Debian. I only ran the former on my laptop, not servers, but I prefer to standardize on a single distribution and it looks like Red Hat derived distributions is where it’s at for me.)

Optical Media Backup Tools in Debian GNU/Linux

Today I happened across a couple of packages to test drive in the near future for photo backup, though neither are specifically tailored for said purpose:

  • backupninja – lightweight, extensible meta-backup system Backupninja lets you drop simple config files in /etc/backup.d to coordinate system backups. Backupninja is a master of many arts, including incremental remote filesystem backup, MySQL backup, and ldap backup. By creating simple drop-in handler scripts, backupninja can learn new skills. Backupninja is a silent flower blossom death strike to lost data.
  • cedar-backup2 – Cedar Backup is a software package designed to manage system backups for a pool of local and remote machines. Cedar Backup understands how to back up filesystem data as well as MySQL and PostgreSQL databases and Subversion repositories. It can also be easily extended to support other kinds of data sources.

SSH key distribution with Ruby and Net::SSH::Multi

When faced with deploying a ssh key to a ton of servers using password authentication, there is but one solution. Ruby, naturally. Below is a script that will iterate through a list of hosts either via STDIN or IO redirection, query for a password once on the command line, then proceed to distribute a specified key to each host.

#!/usr/bin/ruby1.8
 
#
# Jason Boxman <jasonb@edseek.com>
# 20110624
#
# Sanely deploy ssh public key to multiple hosts.
# Will prompt for ssh password using highline.
#
 
require 'optparse'
require 'fcntl'
require 'rubygems'
require 'net/ssh'
require 'net/ssh/multi'
require 'net/ssh/askpass'
require 'highline/import'
 
OptionParser.new do |o|
	o.on('-f', '--keyfile FILENAME',
		'You must specify a public key to distribute') do |filename|
		$keyfile = filename
		$keydata = IO.read($keyfile).gsub(/\n/, '') if File.exists?($keyfile)
		raise 'No keydata' if $keydata.nil?
	end
	o.on('-h') {puts o; exit}
	o.parse!
end
 
# Based upon this thread or $stdin gets messed up:
# http://stackoverflow.com/questions/1992323/reading-stdin-multiple-times-in-bash
old = $stdin.dup
new = File::open('/dev/tty')
$stdin.reopen(new)
passwd = ask("Password?") {|q| q.echo = false}
$stdin.reopen(old)
new.close
 
options = {
	:concurrent_connections => 5,
	:on_error => :ignore,
	:default_user => 'root'
}
sess_options = {
	:password => passwd,
	:auth_methods => ['password'],
	:verbose => :warn
}
 
def get_hosts
	(STDIN.fcntl(Fcntl::F_GETFL, 0) == 0) ?	ARGF.collect {|f| f} : nil
end
 
# Iterate over a group of servers and deploy an SSH key
Net::SSH::Multi.start(options) do |session|
	session.use(sess_options) { get_hosts }
	session.exec <<-EOT
	test -e ~/.ssh || mkdir ~/.ssh
	test -e ~/.ssh/authorized_keys || touch ~/.ssh/authorized_keys
	if ! grep -q "#{$keydata}" ~/.ssh/authorized_keys ; then
		chmod go-w ~ ~/.ssh ~/.ssh/authorized_keys ; \
		echo "#{$keydata}" >> ~/.ssh/authorized_keys
	fi
	EOT
end

Chef and dbconfig-common: A world of hurt

In constructing a Chef cookbook for managing dspam, I happened upon dbconfig-common, a framework that allows a package to manage database backends somewhat transparently. However, it has no affinity for preseeding. Another approach is necessary for the libdspam7-drv-mysql package.

In the relevant section of the recipe below, the file created to cache relevant database values is seeding with the correct values, plus the MySQL administration password. (Eventually, support for PostgreSQL will be added to the cookbook, but it took a while to work out how to seed dbconfig-common.)

# The usage of dbconfig-common complicates things because it won't overwrite
# the shipped mysql.conf without prompting and that apparently cannot easily be
# forced.  It also cannot be preseeded, so libdspam7-drv-mysql.conf must be
# seeded and cleaned up after.
 
directory '/etc/dbconfig-common' do
	owner 'root'
	group 'root'
	mode 0755
end
 
template '/etc/dbconfig-common/libdspam7-drv-mysql.conf' do
	source 'libdspam7-drv-mysql.conf.erb'
	owner 'root'
	group 'root'
	mode 0600
	backup false
	not_if 'test -f /etc/dbconfig-common/libdspam7-drv-mysql.conf'
end
 
script 'rename config' do
	action :nothing
	interpreter 'bash'
	code <<-EOT
		mv -f /etc/dspam/dspam.d/#{drv_name}.conf.ucf-dist \
			/etc/dspam/dspam.d/#{drv_name}.conf
		chown dspam:dspam /etc/dspam/dspam.d/#{drv_name}.conf
		chmod g+r /etc/dspam/dspam.d/#{drv_name}.conf
		mv -f /etc/dbconfig-common/libdspam7-drv-mysql.conf.ucf-dist \
			/etc/dbconfig-common/libdspam7-drv-mysql.conf
		rm -f /etc/dbconfig-common/libdspam7-drv-mysql.conf.ucf-old
	EOT
	only_if "test -f /etc/dspam/dspam.d/#{drv_name}.conf.ucf-dist"
	notifies :restart, resources(:service => 'dspam')
end
 
package drv_package do
	action :install
	notifies :run, resources(:script => 'rename config'), :immediately
	# dbconfig-common uses ucf, which does not have anything to do with dpkg force-confnew sadly.
	#options '-o Dpkg::Options::="--force-confnew"'
end

Naturally, there’s a template file, referenced above, with the values to feed to dbconfig-common.

dbc_install='true'
dbc_upgrade='true'
dbc_remove=''
dbc_dbtype='mysql'
dbc_dbuser='dspam'
dbc_dbpass='<%= node[:dspam][:server][:db_password] =%>'
dbc_dbserver=''
dbc_dbport=''
dbc_dbname='dspam'
dbc_dbadmin='root'
dbc_dbadmpass='<%= node[:mysql][:server_root_password] =%>'
dbc_basepath=''

At some point, the full cookbook will be available on github. I need to work out how to cleanly extract it, probably using the awesome git-subtree project.

Safely unplugging Western Digital My Passport USB drive

Having recently acquired a 1TB WD My Passport SE USB 3.0, I would hate to destroy it. I noticed it shuts down when removed from a Windows system safely, but simply unmounting under Kubuntu via the Device Notifier applet does not have the same effect. Naturally, I become worried about destroying my device.

Fortunately, there is a solution to this. Yan Li devised a script that will correctly power down a USB drive so it can be safely removed. It works great with my WD Passport. Should work for any USB drive though.

It’s necessary to install the sdparm package for it to work, but otherwise works out of the box under Kubuntu 10.10.

Thanks Yan!