Is the Windows registry a good idea?

Compared to Unix config files, and even to Windows 3.1 ‘ini’ files is the windows registry a good idea? This was the question that presented itself in my mind this weekend.

First, what is the windows registry? Windows registry is a Hierarchical database that stores configuration settings for Windows, and for the software installed on your machine.

Now, coming back to the original question I feel while at the core, the registry is a good idea – keeping every setting in a centralised place – it encourages non-portable software. If you install, say, CorelDraw at a certain location on your harddisk, and want to move it to a different location, you need to uninstall it and then reinstall again. Ditto for moving from one computer to another. A time-consuming process – especially if you use multiple computers – one at work, one at home, and another at a cafe.

Why is it that I say the registry, per se, encourages non-portable software? The reason is, when you had config files, and you moved software, the files moved with it too. With registry this doesn’t happen. Note however, either way its possible to write portable or non-portable software. Just that, when a programmer writes code without portability as a key focus, and uses config files, the end product is more likely to be portable.

What can be done about it? When a software program runs, it needs to check if the keys it needs are present or not. If not, it should try to default the parameters, and add them to the registry.

One current approach to writing portable software, is to use config files and provide an update program: if you move the software, you run the update program. This program will detect the file-paths and update those in the config files. The same can be used for registry.


NTFS: Alternate Data Streams

Not many people working on Windows using the NTFS file system are aware of a feature called Alternate Data Streams (ADS).

First things first: A file system is that part of an operating system which deals with storage and management of files on the disk. Its responsible for the ability to retrieve data from a file when needed. An operating system can support multiple file systems – for example, Windows XP supports at least FAT and NTFS.

Now we come to ADS. Its a feature of NTFS, and allows data to be stored in “hidden” blocks linked to a file. Normal operations such as reading or writing to the file do not reveal the existence of this data. So it can happen that your hard disk is full, yet the file sizes do not reveal the reason. Here is a small ‘test’ I did in this regards (for people who understand command line – a more ‘Windows’ example is available here):

C:\>copy con hello.txt
This is visible to all.
        1 file(s) copied.

C:\>dir hello.txt
 Volume in drive C has no label.
 Volume Serial Number is A8EB-38D4

 Directory of C:\

10/15/2008  12:56 PM                25 hello.txt
               1 File(s)             25 bytes
               0 Dir(s)   8,501,428,224 bytes free

C:\>echo This is visible only to some > hello.txt:hiddendata

C:\>type hello.txt
This is visible to all.

C:\>dir hello.txt
 Volume in drive C has no label.
 Volume Serial Number is A8EB-38D4

 Directory of C:\

10/15/2008  12:56 PM                25 hello.txt
               1 File(s)             25 bytes
               0 Dir(s)   8,501,428,224 bytes free

C:\>more < hello.txt:hiddendata
This is visible only to some, and is part of ADS

This is what I did: I created a file called hello.txt with some text. Thereafter, I checked the file-size and it was 25 bytes. I then added some text to the file, but under an ADS named ‘hiddendata’. I again checked the file-size: it was still shown as 25 bytes, and the contents did not show the content of the ADS. However, when I directly asked to be shown the content of the ADS, it was shown to me.

Windows may use ADS to store access related information, or anything it wants to. In particular this is used to store “zone” information – when a file is downloaded from the internet and you see a warning when executing the file – its because of this zone information stored in the ADS. To remove ADS from a file, a quick way is to copy it to a FAT device and then copy it back. Since FAT cannot hold ADS, it gets deleted.

Virus writers, however, may use this feature for anything: they can store their own files without anyone noticing. Can you run an executable file directly off an ADS? You bet:

C:\>start /b c:/hello.txt:hello.exe


Hello, World.

Hence it’s important to keep an eye on your NTFS drives. One way to do this is to use a tool called LADS. It comes with a very nice FAQ on ADS. Another tool, one that I haven’t tried myself is LNS.

The official word from Microsoft is here, and another interesting PDF is here.

LADS can be run on a partition, say C: as below:

lads c: /s

If you notice anything strange beyond “Zone.Identifier” or “Thumbs.db:encryptable” – especially a file with ‘exe’ extension – it can be a cause for concern.


Deleting zero byte files

In the past I have shown ways to run Linux scripts on windows based system. I have also talked about one use of the find command, in conjunction with grep command to search for files having a given text content (say a word) in multiple folders.

Today I will show you another use of the find command: to automate tasks such as deleting zero byte files. This is a pretty common cleanup task that’s carried out on machines that are involved in EDI file transfers.

Here is the script:

find . -size 0 | sed -e 's/^/rm /' | sh

It deletes all zero byte files in the current folder, and in the folders below it.

In order to understand this, you may need to read about pipes. The task is carried out in three steps:

  1. find . -size 0 searches for all zero byte files in the given folder and the folders below it and returns the filepaths.
  2. sed -e 's/^/rm /' turns the list of names into a script – for example if the name is ‘/data/x’ it changes it to ‘rm /data/x’. More on sed here.
  3. the last steps simply forwards the script to the Linux shell for execution.

This is also a very flexible script and can be customised to carry out a wide variety of tasks. Please post your variations as comments.


Automate encryption with GPG


This blogpost requires some familiarity with GPG.

Today I want to share the scripts for running GPG in the batch (unattended) mode. You can have a password on the keys if you want, but since this is the automated mode, you may want to use keys without a password. Irrespective of whether or not you use a password – use a separate set of keys that you will not use for anything that not batch processed. The scripts are primarily for Linux. If you need them for Windows, please read this and post any problems you face in comments section, and I will help.

The script for encryption is here:

#! /bin/sh
gpg --batch -r <reciepient> --output $2 --passphrase-fd 3 --sign --encrypt $1 3</apps/gpg/passph

The first line shows the location of the shell – please change it according to where the shell is on your system. The second line has the location of the folder where the keys are located – the pubring.gpg and secring.gpg. In the last line, replace <reciepient> with the name on the reciepient key. The /apps/gpg/passph points to the location of the file containing the passphrase. The script will both sign and encrypt the file – change this to suit your needs. The script expects the name of the input file and the name of the output files as parameters in that order.

The script for decryption is here:

#! /bin/sh
gpg --batch --passphrase-fd 3 --status-fd 1 --decrypt $1 3</apps/gpg/passph | grep '\[GNUPG:\] GOODSIG'
if [ $? -eq 0 ]; then
  gpg --batch --passphrase-fd 3 --output $2 --decrypt $1 3</apps/gpg/passph

The first three lines are similar. Line 4 just checks if the file has a valid signature. If you want to skip this step remove lines 4, 5 and 7. No effort is made to see who signed the file. In my scenario, I could control the people who could sign by having only those public keys in the repository, and the repository could only be written to by ‘root’.

Please post comments in case of questions, concerns.


Searching files within multiple folders

How to search file contents for a specific phrase using grep within multiple folders. Shows ways for both Linux and Windows.

On Linux, the normal way to search for some text within a file is to use grep (Global Regular Expression Print). However, grep has a limitation: it cannot automatically search folders within the current folder. It can only search within files in the current folder. Today I will show you how to use grep to search within all files and folders inside a current folder (recursively).

Windows users – despair not. If you find the standard windows search brain-dead or want to automate the task through scripts, you can also use this script. I have already explained various ways to run Linux scripts on windows – use the one that suits you.

We will couple grep with the find command, to unleash the power.

Here is what you need to do on Linux (or Cygwin):

find . -type f -exec grep -iH 'dedicated' {} \;

and use this for UnxUtils on Windows:

find . -type f -exec grep -iH "dedicated" \"{}\" ;

This does a case insensitive search for the word ‘dedicated’ in the current folder and all subfolders under it. Change -iH to -H for case sensitive search.

You can read the manuals for find and grep and change the commands to suit your needs – this method provides a lot of flexibility. Post your precise usage in comments, especially for Windows.


Running Linux scripts on Windows

How to run Linux scripts on Windows.

Running Linux shell scripts on Windows is very useful to me. We may need to tweak some scripts depending on the environment in use.

One of the best ways to run them, is within Cygwin. This is something I have already written about.

If Cygwin is too bulky for you, you can use UnxUtils. This is a small set of widely used Linux Utilities, and includes the zsh shell. If you use this you will need to limit the scripts to those that use the commands available within this set. However, it still packs in a lot of paunch.

You might need to ensure the file path syntax expected by the given script matches the environment. Cygwin supports both forward-slash and backward-slash. However, please check the documentation/test to see what works.

Last option, is to run Linux on Windows using Virtual PC. That’s the best bet if you can invest the time needed to do the setup. Instructions are available here. However, this will not be able to access your Windows file system. If that is important for you (like it is to me in most cases), please stick to Cygwin. On the other side, if you are testing code that will eventually run on a Linux machine, the VPC method is better.


Portable software

Portable software refers to programs that can be stored onto a media (such as USB drives, CDs, external harddisks etc.) and run directly from there on multiple computers. In other words, software that you can move from one computer to another without the need to re-install.

There are a lot of sites that provide such software for download. Google for “portable software” in general, or, say for “portable firefox” in particular. My favourite portable apps are the GIMP, and portable FileZilla (a GUI FTP tool).

Cygwin is a Linux like environment for Windows. It can run within Windows and access the filesystem. Its very useful to people like me who need Windows as the main OS, but need to test Linux shell scripts and other utilities. I even have a portable version of Cygwin, created based on these instructions. In fact, I was able to improve the procedure slightly. If someone needs help, please contact me through comments. I want to upload the ISO of the DVD created – if someone can provide the bandwidth and hosting space, I can mail a copy of the DVD to him.


Installing Knoppix within Windows

There are some problems if you try to install Knoppix within MS Virtual PC 2004. This post shows you how to get it right.

Knoppix is a Linux flavour that can run directly off a CD or DVD. All you have to do is boot off the CD and it starts running – it has all the basic applications like Internet Browser, Wordprocessing application etc. If you have FAT partitions, you will also be able to write to the disk.

However, I prefer booting into Windows and then running Knoppix as an application. (Although, I have a separate Debian installation as well for my main Linux needs.)

This can be done through Microsoft Virtual PC. I use the 2004 version dowloadable here. Also you need to have an ISO of the Knoppix CD. If you have a CD, you can make an ISO yourself, or download here.

Start MS VPC 2004, create a virtual harddisk and load the ISO, then reboot the VPC machine. Now follow the instructions below:

  1. Create a new VM with virtual HDD.
  2. Attach the Knoppix ISO to the VM and reboot.
  3. On the Linux prompt type: linux install IGNORE_CHECK=1 sudo knoppix_installer
  4. On the menu that comes, select 3. Partition
  5. When it asks for it, select Template 1
  6. Then back to main menu, select configure installation and accept all defaults
  7. Then back to main menu, select start installation [now it will take time]
  8. When the system boots, mouse wont be working so shut it down (but at least let it open the desktop first). Release the ISO and reboot VPC.
  9. All file edits (required in steps below) will need to be done as root, by doing su first
  10. Now, Grub will come up. On the first line, press e. Then on the next screen select the kernel line and press e. After this, add i8042.noloop at the end and press Enter. Press Esc and Enter.
  11. Select the first option and boot.
  12. After login, edit file /boot/grub/menu.lst and do the following for the first configuration only (one that says Default):
    1. Add i8042.noloop at the end of kernel line
    2. Remove the savedefault line since it causes problems later
  13. Now edit file /etc/sysconfig/desktop and change kdm to xdm
  14. Enter the command ‘reboot’ on the console
  15. Let the system boot, login – mouse will work.
  16. If the user config box comes up fine, else go to Settings->Desktop setting wizard. Now, select India in the first box and English-US in the second and press the first button.
  17. Keep following the wizard. At the end select launch KDE control center.
  18. Go to Regional and accessibility->Keyboard layout and bring up English layout
  19. Done, take backup of the VHD file

Most of the steps should also apply if you want to install Knoppix on a normal HDD. Why we have to go this route – for example why mouse isnt working – beats me. Could be bugs – I have collected all this information from different places on the web, according to the problems I faced. If you know why we have to do this way, please place comments. Also, suggest improvements and let me know if this helped.


Licensing and information about the blog available here.