This is an old revision of the document!
Table of Contents
Unix Scripts
I use these scripts every day, they might be useful to others. They might even be useful templates for how to write a decent bash script!! You can download them individually by clicking on the links or you can download this tarball to get them all. Install them somewhere on your path - I use ~/bin
. All my scripts honour the -h, –help options.
libraries
These are used in my other scripts
argp.sh
argp.sh - a wrapper around getopt(1) to make life much easier for script developers and more predictable for script users. See also argp.sh
Usage: argp [OPTION...] [--] [args ...] A wrapper for getopt(1) which simulates argp(3) for bash. Normally other scripts pipe their option descriptions in here for automatic help and man-page production and for generating the code for getopt(1). Requires bash-3+. See also argp(1) which is a binary (and much faster) version of this. See http://sourceforge.net/projects/argpsh See http://bhepple.freeshell.org/oddmuse/wiki.cgi/argp.sh It buys you: o all the goodness of getopt o define options in one central place together with descriptions using a simple flat file format or XML. o fewer 'magic' and duplicated values in your code o automatic consistency between the getopt calling parameters, the case statement processing the user's options and the help/man pages o less spaghetti in your own code o easier to maintain o help-page and man-page printing - all from the same data sources o checking of option consistency at runtime o range checking of option values at runtime o pretty easy to use o portable to OS's without long option support - the help page adapts too ############################################################################### Usage: argp.sh / argp reads its configuration from stdin (fd0) (see below for details) and outputs the following: * --help output on stdout fd1 * errors on stderr fd2 * a list of commands on fd3 The calling program can then 'eval' that list of commands in order to set environment variables as set by the user in command line options. Also, any remaining arguments after the options are removed are set to $1 $2 ... eg. if the user typed 'myprog --foobar --barfoo=three a b c' then after 'eval'ing the fd3 output of argp.sh / argp you would have FOOBAR="set" BARFOO="three" $1 $2 ... = a b c Usually, the calling program would invoke argp.sh / argp like this: exec 4>&1 eval "$(echo "$ARGS" | argp.sh "$@" 3>&1 1>&4 || echo exit $? )" exec 4>&- Here's a bit of an explanation in case you're interested: exec 4>&1 # make fd4 a copy of fd1 stdout eg /dev/pts/0, the terminal eval $( # inside this sub-shell fd1 is now a pipe echo "$ARGS" | # 3>&1: connect fd3 to fd1 (the pipe) so that anything printed # on fd3 is fed to the pipe (and eventually eval'd). # # 1>&4: fd1 is connected to fd4 ie /dev/pts/0 so that anything # that argp.sh / argp prints on stdout (eg the --help output) ends up # on the terminal: argp.sh "$@" 3>&1 1>&4 || echo exit $? ) exec 4>&- # close fd4 ############################################################################### It's easier to look at a running script but here's the manual: ############################################################################### this is a comment. # The following line defines our program name. We might also write # this as ARGP_PROG=$(basename $0): ARGP_PROG=my-script # '--quiet' is a built-in option, but we normally delete it if 'verbose' is # supported. ARGP_DELETE=quiet # this defines the program's version string to be displayed by the -V option: ARGP_VERSION=1.6 # this string is added as a prefix to all option names on output, eg if you need to use an option that clashes with a system environment parameter eg --home: ARGP_PREFIX=FOO_ # if this is set then multiple invocations of a string or array option will # be concatenated using this separator eg '-s s -s b' will give 'a:b' ARGP_OPTION_SEP=: # Here are the options that this program will accept: ######## ##### ### #### ##### ########### # name sname arg type range description ######## ##### ### #### ##### ########### name=default : 'name' must be present and unique. 'default' must be given but may be '' sname : the single-letter short name of the option (use '' if a short name is not needed) arg : the name of the argument to be used in --help (not for booleans) type : b (boolean), i (integer), d (double), s[:] (string), a[:] (array) or u (url). Default is boolean if arg is not given, otherwise string. If a boolean is initialised to '0', then it is incremented every time it is given. If a boolean is initialised to anything else then it flips between 'default' and 'range' every time it is given. If 's:' or 'a:' then ':' overrides ARGP_OPTION_SEP for this option. Any syntactically clean string can be used instead of ':' ie not ' or ". range : for numeric options: min:max eg 1.2:4.5 or min-max eg 1-3 for string options: an extended regexp for array options: a space separated list of alternates in quotes "..." or '...' for boolean options: the value to assign the option when set desc : leave empty for hidden options. Long descriptions can span lines by putting '\' at the end of the line and by terminated the description with a '.' in the first column. ############################################################################### xamples: # This is the simplest possible option definition. It specifies a # hidden option (--hidden) which does not appear in the help or man # pages. It defaults to being a simple flag (boolean) with a default # value of '' and 'set' if --hidden is on the command line. HIDDEN= # a boolean option with a short name and a numeric default which is # incremented every time the --bool, -b option is given: BOOL='0' b '' b '' description of this option # this is a (boolean) flag which gets set to the string 'foobar' when # --flag, -f is on the command line otherwise it is set to 'barfoo': FLAG='barfoo' f '' b 'foobar' a flag # here is an integer value option which must sit in a given range: INT=1 i units i '1:3' an integer. # here is an array value ie just a string which must take one of # the values in the 'range': ARRAY=a a widgets a 'a b' an array # this option is a simple string which will be checked against a regex(7): STRING='' s string s '^foo.*bar$|^$' a string. # a double value which is checked against a range: DOUBLE=1 d miles d 0.1:1.3 a double. # this is a URL which will be checked against the URL regex and has a # default value URL='http://www.foobar.com' u url u '' a url. # delete this one as we want to use the 'q' option for something else: ARGP_DELETE=quiet # this uses the same short letter 'q' as the --quiet option which was # deleted above QUAINT='' q '' s '' a quaint description # here we define the non-option arguments that the command takes: ARGP_ARGS= [--] [args] ARGP_SHORT=This is a short description for the first line of the man page. ARGP_USAGE= This is a longer description. Spring the flangen dump. Spring the flingen dump. Spring the flangen dump. Spring the flingen dump. ############################################################################### XML can also be instead of the above (provided xmlstarlet is available): <?xml version="1.0" encoding="UTF-8"?> <argp> <prog>fs</prog> <args>[--] [pattern]</args> <short>Search for filenames in sub-directories.</short> <delete>quiet</delete> <version>1.2</version> <prefix>FOO_</prefix> <usage>Presently this is just a shorthand for: find . -follow \$EXCLUDE -type $TYPE -name '*pattern*' -print 2>/dev/null |sort </usage> <option name="EXCLUDE" sname="x" type="s" arg="directory">exclude directory</option> <option name="DIR" sname="d" type="b" default="f" range="d">search for a directory rather than a file</option> <option name="SECRET" type="b"/> </argp> Note that the attribute 'name' is required - the others are all optional. Also the option value should be on a single line (the program usage can be on multiple lines). ############################################################################### Note that --verbose, --help, --quiet and --version options will be added automatically. Also, a hidden option '--print-man-page' is provided to print a skeleton man(1) page which can be used as the starting point for a full man page. Another hidden option '--print-xml' prints out the XML equivalent of the argp input. If POSIXLY_CORRECT is set, then option parsing will end on the first non-option argument (eg like ssh(1)). ############################################################################### Here is a sample of the output when the command is called with --help: Usage: my-script [OPTION...] [--] [args] This is a longer description. Spring the flangen dump. Spring the flingen dump. Spring the flangen dump. Spring the flingen dump." Options: -a, --array=<widgets> an array Must be of type 'a'. Must be in the range 'a b'. -b, --bool description of this option -d, --double=<miles> a double. Must be of type 'd'. Must be in the range '0.1-1.3'. -f, --flag a flag -i, --int=<units> an integer. Must be of type 'i'. Must be in the range '1-3'. -q, --quaint a quaint description Must fit the regex ''. -s, --string=<string> a string. Must fit the regex '^foo.*bar$|^$'. -u, --url=<url> a url. Must fit the regex '^(nfs|http|https|ftp|file)://[[:alnum:]_.-]*[^[:space:]]*$'. -v, --verbose be verbose -h, --help print this help message -V, --version print version and exit Options: -q, --quiet be quiet -v, --verbose be verbose -h, --help print this help message -V, --version print version and exit
bh-stdlib
general scripts
backup-copy
backup-copy - my backup script - using this, I can keep a backup at a cost of only about 2Mb/day. Uses the hard-linking power of rsync(1)
Usage: backup-copy [OPTION...] [--] [repository] Backup interesting files as set in the config file. Simple usage: backup-copy # does a backup - put this in cron!! backup-copy -l # shows what backups are available backup-copy --delta :now -v # shows what's changed since last backup backup-copy -f /etc/passwd # show distinct versions of that file Details. Backups will go to: <repository>/backup-copy/$HOST/YYYY/MM/DD-hh_mm with <repository> in the form of either 'hostname:directory' or 'directory' (default is /var). 'directory' must already have a subdirectory 'backup-copy/$HOST' otherwise the --init option can be given to create it. If using a remote system as <repository>, you need to have the ssh key pushed to the remote for login without password. Environment parameters can be set in the config file as follows: MUST_BE_ROOT="yes" # to restrict to root (or "" or unset) DEFAULT_DESTINATION= # repository to use (overridden by a repository on the command line) TEST_FILE= # invariant inoded file to test consistency EXCLUDES= # list of rsync patterns to exclude INCLUDES= # list of rsync patterns to include EXPIRE= # remove backups older than this number of days KEEP_MONTHLY= # only keep one backup per month of anything older than this Options: -b, --brief On all diff operations, report only whether the files differ, not the details of the differences. -c, --config=config-file override the config file (the default is the first in the list: /home/bhepple/.config/backup-copy.achar /home/bhepple/.backup-copy.achar /home/bhepple/.config/backup-copy /home/bhepple/.backup-copy /etc/backup-copy) Default is '/etc/backup-copy'. -C, --check-links=dates don't backup - just check the backup tree ensuring that hard links are correct (this can be slow, particularly on a remote repository). The argument 'dates' is in the format '[first|last|prev|YYYY/MM/DD-HH_MM[,last|prev|YYYY/MM/DD-HH_MM]'. The dates may be wildcarded (with * and ?) but if two dates are specified, the first may not be wildcarded. The dates 'first', 'last' and 'prev' are expanded to the first, last and previous backup dates respectively. If only one date is given, then all files in the backup repository are compared to the files under that date directory. If two args are given then the comparison is between the first date directory and the second one(s) - but intermediate dates are not interpolated. eg to check the latest backup against the previous one, use 'prev,last'. Note that when duplicates are found then the new link is created using the first date directory as the base. --debug even more verbosity --delta=dates don't backup - just diff two backups. The dates are given as '[date1][,date2]' with a date in the format 'first|last|prev|now|YYYY/MM/DD-HH_MM' eg '2011/01/18-21_13,2011/01/26-21_13'. A blank date means the last (latest) backup. The dates 'first', 'last' and 'prev' are expanded to the first, last and previous backup dates respectively. A date of 'now' compares against the live system (this can be quite slow on a remote backup as the files must be downloaded before the diff is done. Fast on a local backup.) Thus to compare the live system with the latest backup use ',now'. You can get the list of available backups with the --list option. The number of files that differ is returned in the exit code. -D, --diff do a diff after the backup -f, --file-vers=file don't backup - just list the distinct versions of 'file' available in the repository with their sizes. Note that this isn't terribly useful if run on a directory or symbolic link. -H, --host=host override the host name. Default is 'achar'. -i, --init initialise repository if necessary -l, --list don't backup - just list the backups in the repository -n, --dryrun dry run - just print what would be done --progress show progress -q, --quiet be quiet -w, --warn warn about suspicious filenames -h, --help print this help message -V, --version print version and exit Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.
ff
ff - recursively find files (wrapper for find(1))
Usage: ff [OPTION...] [--] [pattern] Presently this is just a shorthand for: find . -follow -type f -name '*pattern*' -print 2>/dev/null |sort Options: -C, --cd=directory search from this directory rather than cwd. -d, --dir search for a directory rather than a file. Default is 'f'. -v, --verbose be verbose -x, --exclude=directory exclude directory. -h, --help print this help message -V, --version print version and exit Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.
fs
fs - recursively search for patterns in files (wrapper for find(1) and grep(1))
Usage: fs [OPTION...] [--] pattern [directory [...]] Uses find(1) and grep(1) to recursively search files for regular expressions. Ignores .svn .git nbproject .libs .deps CVS* directories and *~ TAGS #*# and cscope* files. Always applies -I (ignore binary files) and --colour. 'fs -n' is suitable as emacs' grep function. Options: -c, --c-files only looks at *.[ch] *.cpp *.cxx files -C, --context=lines print 'context' lines of context. Must be of type 'i'. Must be in the range '1:100'. --exclude-dir=directory exclude directory --extension=ext extensions to search eg .el -E, --extended-regexp use extended regular expression (as grep) -F, --fixed-strings use fixed strings (as grep) -H, --h-files only looks at *.h files -i, --ignore-case ignore case (as grep) -j, --java-files only looks at *.java files -l, --files-with-matches filename only (as grep) -L, --files-without-matches filename only: without match (as grep) -m, --m4-files only looks at *.m4 files -M, --make-files only looks at *akefile* *.mk files -n, --line-number prefix each hit with the line number (as grep) -p, --python-files only looks at *.py files -r, --ruby-files only looks at *.rb files -s, --spec-files only looks at *.spec files --shell-files only looks at *.sh files -v, --verbose be verbose -x, --xml-files only looks at *.xml files -h, --help print this help message -V, --version print version and exit Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.
fit
fit - fits a bunch of files/directories into CD or DVD sized subdirectories ready for burning
Usage: fit [OPTION...] [--] Options: -c, --cd fit to 700Mb CD (default). -d, --dvd fit to 4.3Gb DVD. -m, --mb=limit fit to 'limit' Mb. Must be of type 'i'. -n, --confirm no dry run - just do it! -p, --prefix=prefix use prefix for dir names. Default is 'fit_'. -s, --smallest-first fir smallest first -v, --verbose be verbose -h, --help print this help message -V, --version print version and exit Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.
m
m - pretty display of man pages (needs 'v' and a postscript or pdf viewer) eg m less
to get a pretty man page for less(1)
Usage: m [OPTION...] [--] [ section ] name A wrapper for 'man'. Depends on the script 'v' available from the same place. Displays a pretty man page if X is available (otherwise just does 'man'). 'section' narrows the search for a man page to that section. 'name' is the name of the manual page. On Solaris, adds the '-s' option for section name. eg. Do 'm man' for more information about 'man' Options: -v, --verbose be verbose -h, --help print this help message -V, --version print version and exit
mass-mv
mass-mv - renames and or moves files according to a regex; can also clean up filenames removing whitespace and/or weird characters
Usage: mass-mv [OPTION...] [--] [ filename(s) ] Renames files en masse. Options: -a, --convert-all replace all runs of non-'[:alnum:].-' with '_' -c, --chop=string chop out a string (not a regex) -l, --lower lower case filenames -n, --dry-run just print what it would do -R, --recursive descend subdirectories -s, --sed=string pass old filename through a sed command (eg 's/asdasd/dfgdfg/') -S, --spaces remove spaces -u, --upper upper case filenames -v, --verbose be verbose -h, --help print this help message -V, --version print version and exit Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.
newest
newest - recursively list files in directories in date order. If called as oldest, reverse the sort direction. If called as biggest or smallest list by file size.
Usage: newest [OPTION...] [--] [directory ...] Find the newest file in the directory(s) (or PWD). Options: -l, --long print more info about each file -r, --reverse find the oldest rather than the newest -v, --verbose be verbose -h, --help print this help message -V, --version print version and exit
r
r - run a command with sudo or popup a root xterm
Usage: r [OPTION...] [--] [command] Run 'command' as root via the 'root' program if available else sudo. If command is not given then start up a root shell. You might want to customise the XTERM setting in this script if you want something different from "myterm --" Note that you need to take care of the /etc/sudoers file by adding the NOPASSWD:: flag to your line. Ideally be a member of wheel group and: %wheel ALL=(ALL) NOPASSWD: ALL Options: -v, --verbose be verbose -X, --use-x prefer X instead of console -h, --help print this help message -V, --version print version and exit
refactor
refactor - uses find(1) and sed(1) to search and replace
Usage: refactor [OPTION...] [--] pattern [ subst ] Do pattern and substitution on all files in sub-directories. If 'subst' is not given then a 'camel case to underscore' replacement will be done eg CamelCase -> camel_case. Does not touch .svn, 'cscope*', 'TAGS' or '*~' files. It's often a good idea to use -n,--dryrun and copy, paste and modify the command before running it manually. Options: -c, --allc-files only looks at *.[ch], *.cxx, *.hpp & *.cpp files -C, --c-files only looks at *.c, *.cxx & *.cpp files --exclude=directory exclude 'directory' --extension=ext extensions to search ie *.$ext -H, --h-files only looks at *.h files -j, --java-files only looks at *.java files -m, --m4-files only looks at *.m4 files -M, --make-files only looks at *akefile* Minfo*.cfg files -n, --dryrun dryrun -p, --python-files only looks at *.py files -P, --partial match on any occurance of 'pattern' ie don't add \\<..\\> -s, --spec-files only looks at *.spec files --shell-files only looks at *.sh files -v, --verbose be verbose -h, --help print this help message -V, --version print version and exit
v
v - view or execute any type of file going rather beyond the scope of lessopen - in the simple case of a text file this is just shorthand for less(1)
Usage: v [OPTION...] [--] file(s) View a file using an application appropriate to its type depending on whether you're running from a console or under X. Prefer console apps over X. Does not rely on lesspipe.sh Passes all files to the viewer if it makes sense to do so. Also, only the first file is inspected to decide how to view/edit/exec. Options: -a, --all view all media files under PWD (recursively) -e, --edit edit or exec instead of view (default if run as 'x') -n, --dryrun just print what would be done -v, --verbose be verbose -x, --use-x prefer X instead of console -h, --help print this help message -V, --version print version and exit
xcheck
xcheck - runs a command (usually X) and shows stdout and stderr in a popup window if there's an error eg xcheck – remote-x11vnc -x xdm
Usage: xcheck [OPTION...] [--] <x-command> Run a X command capturing stdout and stderr. If it gives an error, popup a viewer with the messages. If it survives for 5 seconds, exit this script and delete any temporary files. Also removes any crufty xwin-utils backup file /home/bhepple/.local/share/xwin/$WINDOW_ID. Options: -f, --force always display output -v, --verbose be verbose -w, --watch=time time to watch the app. 0==forever. Default is '5'. Must be of type 'i'. Must be in the range '0:1000'. -h, --help print this help message -V, --version print version and exit Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.