Moving multiple files

One common requirement on *nix systems is to rename (or move-and-rename) more than one file, matching a wildcard. However, if you try to do this:

mv ACK*.xml /code/xml/ACK*.xml.done

it wont work for more than one file. By the way, this is one of those odd places where Windows command line works better!

How to work around this? As below:

ls ACK*.xml | sed -e 's/.*/mv & \/code\/xml\/&.done/' | sh

I have already talked about sed, sh and this style of scripting: it can work also on Windows. In the target path all slashes need to be escaped like this: \/ since its being used inside a sed script. Of course, this was easy because we had to suffix something to the filename – had we needed to add something in the middle, it would have been more difficult (and a bigger sed script would be needed).

Here is a more ‘capable’ script: it takes two parameters, first one the path to the source directory, and next one the path to the target directory. It moves ACK*.xml files from source to target/.processed (.processed folder under target) renaming them with the current datetime stamp.

Customise it to suit your needs.

Share

Deleting zero byte files

In the past I have shown ways to run Linux scripts on windows based system. I have also talked about one use of the find command, in conjunction with grep command to search for files having a given text content (say a word) in multiple folders.

Today I will show you another use of the find command: to automate tasks such as deleting zero byte files. This is a pretty common cleanup task that’s carried out on machines that are involved in EDI file transfers.

Here is the script:

find . -size 0 | sed -e 's/^/rm /' | sh

It deletes all zero byte files in the current folder, and in the folders below it.

In order to understand this, you may need to read about pipes. The task is carried out in three steps:

  1. find . -size 0 searches for all zero byte files in the given folder and the folders below it and returns the filepaths.
  2. sed -e 's/^/rm /' turns the list of names into a script – for example if the name is ‘/data/x’ it changes it to ‘rm /data/x’. More on sed here.
  3. the last steps simply forwards the script to the Linux shell for execution.

This is also a very flexible script and can be customised to carry out a wide variety of tasks. Please post your variations as comments.

Share

Automate encryption with GPG

Privacy
Privacy

This blogpost requires some familiarity with GPG.

Today I want to share the scripts for running GPG in the batch (unattended) mode. You can have a password on the keys if you want, but since this is the automated mode, you may want to use keys without a password. Irrespective of whether or not you use a password – use a separate set of keys that you will not use for anything that not batch processed. The scripts are primarily for Linux. If you need them for Windows, please read this and post any problems you face in comments section, and I will help.

The script for encryption is here:

#! /bin/sh
GNUPGHOME='/apps/gpg/'
export GNUPGHOME
gpg --batch -r <reciepient> --output $2 --passphrase-fd 3 --sign --encrypt $1 3</apps/gpg/passph

The first line shows the location of the shell – please change it according to where the shell is on your system. The second line has the location of the folder where the keys are located – the pubring.gpg and secring.gpg. In the last line, replace <reciepient> with the name on the reciepient key. The /apps/gpg/passph points to the location of the file containing the passphrase. The script will both sign and encrypt the file – change this to suit your needs. The script expects the name of the input file and the name of the output files as parameters in that order.

The script for decryption is here:

#! /bin/sh
GNUPGHOME='/apps/gpg/'
export GNUPGHOME
gpg --batch --passphrase-fd 3 --status-fd 1 --decrypt $1 3</apps/gpg/passph | grep '\[GNUPG:\] GOODSIG'
if [ $? -eq 0 ]; then
  gpg --batch --passphrase-fd 3 --output $2 --decrypt $1 3</apps/gpg/passph
fi

The first three lines are similar. Line 4 just checks if the file has a valid signature. If you want to skip this step remove lines 4, 5 and 7. No effort is made to see who signed the file. In my scenario, I could control the people who could sign by having only those public keys in the repository, and the repository could only be written to by ‘root’.

Please post comments in case of questions, concerns.

Share

Searching files within multiple folders

How to search file contents for a specific phrase using grep within multiple folders. Shows ways for both Linux and Windows.

On Linux, the normal way to search for some text within a file is to use grep (Global Regular Expression Print). However, grep has a limitation: it cannot automatically search folders within the current folder. It can only search within files in the current folder. Today I will show you how to use grep to search within all files and folders inside a current folder (recursively).

Windows users – despair not. If you find the standard windows search brain-dead or want to automate the task through scripts, you can also use this script. I have already explained various ways to run Linux scripts on windows – use the one that suits you.

We will couple grep with the find command, to unleash the power.

Here is what you need to do on Linux (or Cygwin):

find . -type f -exec grep -iH 'dedicated' {} \;

and use this for UnxUtils on Windows:

find . -type f -exec grep -iH "dedicated" \"{}\" ;

This does a case insensitive search for the word ‘dedicated’ in the current folder and all subfolders under it. Change -iH to -H for case sensitive search.

You can read the manuals for find and grep and change the commands to suit your needs – this method provides a lot of flexibility. Post your precise usage in comments, especially for Windows.

Share

Running Linux scripts on Windows

How to run Linux scripts on Windows.

Running Linux shell scripts on Windows is very useful to me. We may need to tweak some scripts depending on the environment in use.

One of the best ways to run them, is within Cygwin. This is something I have already written about.

If Cygwin is too bulky for you, you can use UnxUtils. This is a small set of widely used Linux Utilities, and includes the zsh shell. If you use this you will need to limit the scripts to those that use the commands available within this set. However, it still packs in a lot of paunch.

You might need to ensure the file path syntax expected by the given script matches the environment. Cygwin supports both forward-slash and backward-slash. However, please check the documentation/test to see what works.

Last option, is to run Linux on Windows using Virtual PC. That’s the best bet if you can invest the time needed to do the setup. Instructions are available here. However, this will not be able to access your Windows file system. If that is important for you (like it is to me in most cases), please stick to Cygwin. On the other side, if you are testing code that will eventually run on a Linux machine, the VPC method is better.

Share

Portable software

Portable software refers to programs that can be stored onto a media (such as USB drives, CDs, external harddisks etc.) and run directly from there on multiple computers. In other words, software that you can move from one computer to another without the need to re-install.

There are a lot of sites that provide such software for download. Google for “portable software” in general, or, say for “portable firefox” in particular. My favourite portable apps are the GIMP, and portable FileZilla (a GUI FTP tool).

Cygwin is a Linux like environment for Windows. It can run within Windows and access the filesystem. Its very useful to people like me who need Windows as the main OS, but need to test Linux shell scripts and other utilities. I even have a portable version of Cygwin, created based on these instructions. In fact, I was able to improve the procedure slightly. If someone needs help, please contact me through comments. I want to upload the ISO of the DVD created – if someone can provide the bandwidth and hosting space, I can mail a copy of the DVD to him.

Share