unix | Digital Wealth

Is the Windows registry a good idea?

Compared to Unix config files, and even to Windows 3.1 ‘ini’ files is the windows registry a good idea? This was the question that presented itself in my mind this weekend.

First, what is the windows registry? Windows registry is a Hierarchical database that stores configuration settings for Windows, and for the software installed on your machine.

Now, coming back to the original question I feel while at the core, the registry is a good idea – keeping every setting in a centralised place – it encourages non-portable software. If you install, say, CorelDraw at a certain location on your harddisk, and want to move it to a different location, you need to uninstall it and then reinstall again. Ditto for moving from one computer to another. A time-consuming process – especially if you use multiple computers – one at work, one at home, and another at a cafe.

Why is it that I say the registry, per se, encourages non-portable software? The reason is, when you had config files, and you moved software, the files moved with it too. With registry this doesn’t happen. Note however, either way its possible to write portable or non-portable software. Just that, when a programmer writes code without portability as a key focus, and uses config files, the end product is more likely to be portable.

What can be done about it? When a software program runs, it needs to check if the keys it needs are present or not. If not, it should try to default the parameters, and add them to the registry.

One current approach to writing portable software, is to use config files and provide an update program: if you move the software, you run the update program. This program will detect the file-paths and update those in the config files. The same can be used for registry.

Moving multiple files

One common requirement on *nix systems is to rename (or move-and-rename) more than one file, matching a wildcard. However, if you try to do this:
mv ACK*.xml /code/xml/ACK*.xml.done

it wont work for more than one file. By the way, this is one of those odd places where Windows command line works better!

How to work around this? As below:
ls ACK*.xml | sed -e 's/.*/mv & \/code\/xml\/&.done/' | sh

I have already talked about sed, sh and this style of scripting: it can work also on Windows. In the target path all slashes need to be escaped like this: \/ since its being used inside a sed script. Of course, this was easy because we had to suffix something to the filename – had we needed to add something in the middle, it would have been more difficult (and a bigger sed script would be needed).

Here is a more ‘capable’ script: it takes two parameters, first one the path to the source directory, and next one the path to the target directory. It moves ACK*.xml files from source to target/.processed (.processed folder under target) renaming them with the current datetime stamp.

Customise it to suit your needs.

Deleting zero byte files

In the past I have shown ways to run Linux scripts on windows based system. I have also talked about one use of the find command, in conjunction with grep command to search for files having a given text content (say a word) in multiple folders.

Today I will show you another use of the find command: to automate tasks such as deleting zero byte files. This is a pretty common cleanup task that’s carried out on machines that are involved in EDI file transfers.

Here is the script:
find . -size 0 | sed -e 's/^/rm /' | sh

It deletes all zero byte files in the current folder, and in the folders below it.

In order to understand this, you may need to read about pipes. The task is carried out in three steps:

find . -size 0 searches for all zero byte files in the given folder and the folders below it and returns the filepaths.
sed -e 's/^/rm /' turns the list of names into a script – for example if the name is ‘/data/x’ it changes it to ‘rm /data/x’. More on sed here.
the last steps simply forwards the script to the Linux shell for execution.

This is also a very flexible script and can be customised to carry out a wide variety of tasks. Please post your variations as comments.

Automate encryption with GPG

This blogpost requires some familiarity with GPG.

Today I want to share the scripts for running GPG in the batch (unattended) mode. You can have a password on the keys if you want, but since this is the automated mode, you may want to use keys without a password. Irrespective of whether or not you use a password – use a separate set of keys that you will not use for anything that not batch processed. The scripts are primarily for Linux. If you need them for Windows, please read this and post any problems you face in comments section, and I will help.

The script for encryption is here:

#! /bin/sh GNUPGHOME='/apps/gpg/' export GNUPGHOME gpg --batch -r <reciepient> --output $2 --passphrase-fd 3 --sign --encrypt $1 3</apps/gpg/passph

The first line shows the location of the shell – please change it according to where the shell is on your system. The second line has the location of the folder where the keys are located – the pubring.gpg and secring.gpg. In the last line, replace <reciepient> with the name on the reciepient key. The /apps/gpg/passph points to the location of the file containing the passphrase. The script will both sign and encrypt the file – change this to suit your needs. The script expects the name of the input file and the name of the output files as parameters in that order.

The script for decryption is here:
#! /bin/sh GNUPGHOME='/apps/gpg/' export GNUPGHOME gpg --batch --passphrase-fd 3 --status-fd 1 --decrypt $1 3</apps/gpg/passph | grep '\[GNUPG:\] GOODSIG' if [ $? -eq 0 ]; then gpg --batch --passphrase-fd 3 --output $2 --decrypt $1 3</apps/gpg/passph fi

The first three lines are similar. Line 4 just checks if the file has a valid signature. If you want to skip this step remove lines 4, 5 and 7. No effort is made to see who signed the file. In my scenario, I could control the people who could sign by having only those public keys in the repository, and the repository could only be written to by ‘root’.

Please post comments in case of questions, concerns.

Searching files within multiple folders

How to search file contents for a specific phrase using grep within multiple folders. Shows ways for both Linux and Windows.

On Linux, the normal way to search for some text within a file is to use grep (Global Regular Expression Print). However, grep has a limitation: it cannot automatically search folders within the current folder. It can only search within files in the current folder. Today I will show you how to use grep to search within all files and folders inside a current folder (recursively).

Windows users – despair not. If you find the standard windows search brain-dead or want to automate the task through scripts, you can also use this script. I have already explained various ways to run Linux scripts on windows – use the one that suits you.

We will couple grep with the find command, to unleash the power.

Here is what you need to do on Linux (or Cygwin):

find . -type f -exec grep -iH 'dedicated' {} \;

and use this for UnxUtils on Windows:

find . -type f -exec grep -iH "dedicated" \"{}\" ;

This does a case insensitive search for the word ‘dedicated’ in the current folder and all subfolders under it. Change -iH to -H for case sensitive search.

You can read the manuals for find and grep and change the commands to suit your needs – this method provides a lot of flexibility. Post your precise usage in comments, especially for Windows.

Running Linux scripts on Windows

How to run Linux scripts on Windows.

Running Linux shell scripts on Windows is very useful to me. We may need to tweak some scripts depending on the environment in use.

One of the best ways to run them, is within Cygwin. This is something I have already written about.

If Cygwin is too bulky for you, you can use UnxUtils. This is a small set of widely used Linux Utilities, and includes the zsh shell. If you use this you will need to limit the scripts to those that use the commands available within this set. However, it still packs in a lot of paunch.

You might need to ensure the file path syntax expected by the given script matches the environment. Cygwin supports both forward-slash and backward-slash. However, please check the documentation/test to see what works.

Last option, is to run Linux on Windows using Virtual PC. That’s the best bet if you can invest the time needed to do the setup. Instructions are available here . However, this will not be able to access your Windows file system. If that is important for you (like it is to me in most cases), please stick to Cygwin. On the other side, if you are testing code that will eventually run on a Linux machine, the VPC method is better.

Portable software

Portable software refers to programs that can be stored onto a media (such as USB drives, CDs, external harddisks etc.) and run directly from there on multiple computers. In other words, software that you can move from one computer to another without the need to re-install.

There are a lot of sites that provide such software for download. Google for “portable software” in general, or, say for “portable firefox” in particular. My favourite portable apps are the GIMP, and portable FileZilla (a GUI FTP tool).

Cygwin is a Linux like environment for Windows. It can run within Windows and access the filesystem. Its very useful to people like me who need Windows as the main OS, but need to test Linux shell scripts and other utilities. I even have a portable version of Cygwin, created based on these instructions. In fact, I was able to improve the procedure slightly. If someone needs help, please contact me through comments. I want to upload the ISO of the DVD created – if someone can provide the bandwidth and hosting space, I can mail a copy of the DVD to him.

Online judge

Sometime back I came across Sphere Online Judge (SPOJ). A ‘judge’ is a mechanism through which you can verify your programming solutions.
The SPOJ contains a huge range of programming problems. You need to code a solution in one of the supported languages (from Unix shell scripting to C to Java to esoteric languages like B**fcuk). When you submit the solution, the judge will execute your code with predetermined input and match against predetermined output. If all goes well, your solution will be accepted. Otherwise, it will say ‘compilation error’, or ‘incorrect output’ or ‘time limit exceeded’. It will not tell you the input values or expected output, nor will it tell you the input for which your program is going wrong.

Its very interesting, the problems are very challenging and I gave some problems a try. At the end of it, you can see the report generated by the judge specific to you:

http://www.spoj.com/users/hardeeps/

Also, you can see the submission history:

http://www.spoj.com/status/hardeeps/

or, showcase a signed certificate that the judge provides:

http://www.spoj.com/status/hardeeps/signedlist/

You can see that I have tried various languages, including Java, Bash (Unix shell) and Perl. I am yet to prove my programming skills in C. 🙂

The result AC means that the solution is accepted. The certificate also shows that I was able to calculate PI correctly to 2500 places of decimal in less than 24 seconds – I used the identity described in this blogpost.

I have tried to understand how to verify the digital signature on the certificate but have not been successful. The author informs me that a PHP function is being used to generate the certificate. It would be good if someone can get this working (please post in comments).

Value of Pi

Wikipedia defines Pi or p as “a mathematical constant which represents the ratio of any circle’s circumference to its diameter in Euclidean geometry”. As a kid I used to be interested in the calculation of Pi. The value of Pi, to 100 places of decimal is:

3.1415 9265 3589 7932 3846 2643 3832 7950 2884 1971 6939 9375 1058 2097 4944 5923 0781 6406 2862 0899 8628 0348 2534 2117 0664

I have calculated this using the Unix command bc. The command for this is based on an identity (that I think is credited to Ramanujan) and is:
24*a(1/8)+8*a(1/57)+4*a(1/239)
where a stands for the arctangent function.

The formula is:
pi = 24*arctan(1/8)+8*arctan(1/57)+4*arctan(1/239)

First, load the bc language with associated library using “bc -l”. Then, set the scale to the number of digits using “scale=100”. Afterwards run the identity I gave above.

To experimentally calculate Pi experimentally, there are two ways: One using a random number generator and the other by physically measuring the circumference of a given circle.

Consider a square of length unity. Within that, a circle is drawn, having unity diameter. If a point is taking within the square at random, the probability of that point also lying within the circle is Pi*(1/2)*(1/2) which is Pi/4. Now, start taking points at random (x,y) and see if x*x+y*y<=1/4 or not (if it is, then that means the point lies within the circle). Maintain the count of total number of points taken (t), and the number that fell within the circle(c). Now Pi can be calculated as:

Pi=4*c/t

The other method is to take a circular bottle (measure the radius r) or tin and tie a thread around its circumference. Measure the circumference(c). Now Pi is c/2r.

The following two sentences contain the value of Pi: the number of letters in each word indicates the corresponding number in the value of Pi.

1. “May I have a large cup of coffee.”

2. “How I want a drink, alcoholic of course, after the heavy chapters involving quantum mechanics.”

This makes the value of Pi easy to remember.

Lastly, how useful are the digits of Pi as a source of random numbers? Not bad, according to a study: “while sequences of digits from pi are indeed an acceptable source of randomness – often an important factor in data encryption and in solving certain physics problems – pi’s digit string does not always produce randomness as effectively as manufactured generators do”.

Always wanting to do my own thing, I downloaded the value of Pi to one million places from a website, split it into (x,y,z) coordinates, each having 5 digit precision. Here is the graph that got generated:

A bell curve, but could have been better – seems to mimic the results from the study.