eXtropia: the open web technology company
Technology | Support | Tutorials | Development | About Us | Users | Contact Us
Free support
 ::   Support forums
 ::   Frequently asked questions
 ::   Functional Specifications
 ::   eXtropia Tutorials
 ::   Books by eXtropia
 ::   Other books
 ::   Offsite resources
eXtropia ADT Documentation


[ TOC ]
[ TOC ]


To install and customize CGI applications requires that you have some basic knowledge of how CGI works. Although this is a huge topic that is covered well in many books, we will cover the fundamentals here as they apply to using this application.

As a supplement, we also provide a host of free online tutorials that are meant to be accessible to beginners. These tutorials cover the basics of web application development and can be found at the following URL:


The fact is that even if you hire someone else to install your applications or fix your problems, you will still need a basic understanding of what is going on just to explain to someone what your problem is or what you want your application to do.

Please take some time to get the basics down. In the long run it will save you a great deal of time.

[ TOC ]

The 12-Step CheckList

The process of installing and customizing a CGI/Perl application can be broken down into an easy-to-follow checklist of 12 items.

Here is what you'll need to do.

1. Prepare your web account for running CGI applications.

2. Obtain (download) application installation file.

3. Unpack the application installation file on your web server.

4. Set the permissions correctly for application files and directories.

5. Modify the Perl path line.

6. Configure the application.

7. Modify the look-and-feel.

8. Run the application from the web.

9. Debug the application if any debugging is required.

10. Review the security of the script.

11. Submit the application for user testing.

12. Register yourself as a user with the application author so that you will receive important email updates such as bug reports, security alerts, and new version announcements

Let's go over each of these points in greater detail.

[ TOC ]

Step One: Prepare Your Site

Before you even download a CGI application for your website, there are several things you must do in order to make sure that you will be able to use the downloaded CGI application.

Specifically, you must:

1. Make sure that the web server environment is configured to permit the use of CGI applications

2. Make sure that you understand the site-specific features of that configuration. How have your system administrators configured CGI functionality differently from other web sites?

The quickest way to make sure that your web server is configured to permit CGI applications is to send an email to your administrator or check the documentation that your Internet Service Provider (ISP) should provide.

Generally, your web server system administrator will respond by telling you that CGI applications are allowed, but that they may only be installed in a special directory such as cgi-bin.

    NOTE: If the system administrator says that CGI applications are not allowed, then you cannot run CGI applications. In this case, we recommend that you change to an ISP that allows you to run CGI applications.

    NOTE: If you run the CGI application using a web browser and you get the code of the application printed out to the browser window, or if the browser application prompts you to download the "unknown file", then it is likely that the web server is not recognizing your CGI application or that it is not allowing it to execute. This is a problem that your system administrator must address.

cgi-bin (or a similarly-named directory) is a directory used to store ``executable'' CGI applications. Keeping CGI applications limited to specific directories is advisable since CGI applications may open security holes. Therefore, most system administrators allow only files inside of a special directory such as cgi-bin to be executed. Keeping all the CGI applications in one place does not solve the problem of security, but it does keep all the suspect applications in one manageable area.

The system administrator may also tell you that all CGI applications are run through a CGI wrapper. CGI wrappers add another degree of security, but can make installation a little more difficult.

There are several CGI wrapper products available on the web (such as CGIWrap available at http://www.unixtools.org/cgiwrap/) and many large ISPs use them. All our applications work in conjunction with CGI wrappers but sometimes, installation may require some futzing around. Typically, this will involve changing the lines that define file and directory locations because one of the things that CGI wrappers do is to modify what the web server sees as the root directory on a file system. In order to find out what the real path a CGI application runs on when passed through a wrapper, you should contact your system administrator.

Finally, the system administrator may tell you that you need to register the CGI executable directory and/or CGI application by either running a special CGI registration program, or by adding a special security access file (such as .htaccess) to the directory where the application is installed.

All of these factors will be specific to the web server that you are using. Each system administrator will have her own way of setting up CGI on the web server. Thus, there is no way to create an application configuration that will run on your site by default.

Unfortunately, you are going to need to do some site-specific investigation and modify the relevant configuration parameters in the distribution files for an application to work on your site.

To make the process of investigating your local web server configuration easier and because we understand that you may not know exactly which questions you need to ask, we have provided a sample letter that you can send to your system administrator. This letter should get you the information you need.

    Dear system administrator,
    I would like to run a pre-written, open source CGI application on 
    my website. My website URL is __________ (eg. http://www.mydomain.com/). 
    My home directory is______________ (eg. /usr/home/....)
    I downloaded the application from http://www.extropia.com/. The 
    application name is _______________. I have attached the application 
    to this email in case you want to review it. 
    [Donít forget to attach the downloaded distribution TAR/GZ file]
    Please answer the following questions so that I may continue with this 
    1.  What web server software are you running? (eg. Apache, NCSA, 
        Netscape, IIS)
    2.  What platform is the web server running on? (eg. LINUX, Solaris, 
        Windows NT)
    3.  What is the path to the Perl 5 interpreter? (eg. /usr/local/bin/perl)
    4.  What is the path to an Internet mail program that I may use for CGI 
        applications? (eg. /usr/sbin/sendmail or C:/Mail/Blat/blat.exe)
    5.  Is your web server using a CGI wrapper? If so, how can I access the 
        documentation for the wrapper? Also, are there any special problems 
        reported by other users with using this CGI wrapper? How does the 
        wrapper affect how CGI applications see the file system?
    6.  Can I get access to the server logs in order to check for CGI errors 
        and/or hack attempts? If so, what do I need to do?
    7.  Do CGI applications need to be stored in specific directories? If 
        so, what must I do to create or use such a directory? (eg. directory 
        name, permissions, etc...)
    8.  Do I need to register my CGI application with the web server? (eg. 
        .htaccess file)
    9.  Is there anything else I should know about the environment, especially 
        where it affects security and scalability? (eg. use of Mod Perl, FastCGI, 
        Server Side Includes or backend database availability etc...)
    10. Is there a relational database that I may use to store my data? If 
        so, what drivers/modules should I use to connect to it and how can 
        I create new tables?
    11. I would like to make sure that client data remains confidential. 
        Hence, I might require the use of encryption such as PGP. Can you 
        point me towards documentation on the local encryption resources 
        that I may use?
    12. What is the site backup policy? If any of my files are corrupted, 
        how can I rollback to the right versions?
    13. In which directories are the standard Perl modules located so that 
        I can find out which I may use. If I require additional modules, 
        how can I install them?
    Thank you very much,
                [Your Name Here]

Building a good relationship with your system administrator will make both your lives easier. A fluid communication flow will help build trust and assure that you get support when you need it.

[ TOC ]

Step Two: Obtain an Installation File

Once you have worked out all the specifics of your web server configuration, you can go ahead and download the application. To do so, you can go to any number of excellent CGI storehouses, many of which contain free pre-written applications that you can use. If you need help finding these sites, check out the Offsite Resources Page at the following URL:


You can, of course, download any eXtropia applications from the eXtropia website:


[ TOC ]

Step Three: Unpack The Installation File

All eXtropia applications are distributed as TAR/GZ files called app_name.tar.gz (such as addressbook.tar.gz). TAR is a UNIX command that allows you to create a single archive file containing many files. Such archiving allows you to maintain directory relationships and facilitates transferring complex programs with many separate but integrated parts that must have their relationships preserved. TAR has a motley of options that allow you to do archiving and unpacking in many ways. However, for the purpose of unpacking eXtropia applications, the commands will be fairly simple.

[ TOC ]

Unpacking on UNIX

Preferably, you can transfer the TAR file directly to your web server and expand it there. If your web server is UNIX-based, you can expand the file with the following command

    tar xvfpz file_name.tar.gz 


    tar xvfz file_name.tar.gz (if "p" won't work) 

If the file is not .tar.gz, use:

    tar xvfp file_name.tar


    tar xvf file_name.tar (if "p" won't work)

TAR will go through the archive file and extract each individual directory and file, expanding them into their appropriate places beneath the current directory.

If tar xvfz did not work for you, try:

    gunzip file_name.tar.gz (removes the .gz and decompresses) 


    tar xvfp file_name.tar 

    tar xvf file_name.tar (if "p" won't work) 

    NOTE: The "xvfp" letters in the TAR command above are parameters that instruct the program to extract the files and directories out of the ".tar" file.

    Tar Extraction Parameters
    Parameter   Description
    x           Tells tar to extract the files.
    v           Tells tar to output information about the status of 
                its extraction while it is performing the work.
    f           Informs tar to use the ".tar" filename as the source 
                of the files to be extracted. The reason the "f" 
                parameter has to be used is that tar, by default, 
                archives files and directories to a tape 
                drive. TAR is actually short for "[T]ape [AR]chive".
    p           Notes that the original permissions should be maintained.
    z           UnZips the original file and removes .gz 

When you perform unarchiving, the TAR file will expand into the intended directory structure. You can then edit the files necessary to complete the installation directly on your web server machine.

As we said earlier, it is preferable to expand and edit the application on the web server itself. However, you can also expand and edit the application files on your local workstation and then FTP the individual files to the web server. This is explained in the next section. If you do so however, you must make sure to transfer the unarchived files in ASCII mode.

[ TOC ]

Unpacking on Windows and Macintosh

If you are not using a UNIX-based web server, or don't have command line access (such as TELNET) to your UNIX-based web server, you will probably be using Winzip (Windows) or Stuffit Expander (Mac) to expand the TAR file. This will also work with any .tar.gz file without problems.

You'll also use some text editor to edit the application files. If you are looking for a good text editor, we recommend Programmer's File Editor (PFE) or Super Note Tab that are both available at http://www.shareware.com/ for Windows. SimpleText and ClarisWorks are good editors for Mac. And, vi, emacs or pico are good editors for UNIX.

If you use a Windows-based text editor however, you need to be very careful about accidentally inserting platform-specific, invisible control characters (like carriage return characters) into the files. If you are editing the files on a Windows box, this is often a problem because Windows programs are well-known for their desire to insert Windows-only characters into files.

You will know that invisible characters have infected the files if you get a 500 Server Error when trying to run the application from the web, and error messages like the following if you run the application from the command line:

                Illegal character \015 (carriage return) at 
                app_name.cgi line 2. 


                Can't find string terminator " [some text here]"
                anywhere before EOF

Generally, this problem can be solved either by choosing a text editor that does not insert the characters or by setting your FTP program to upload edited files to the web server machine using ``ASCII mode'' instead of ``BINARY mode''. You should be able to set the FTP program to transfer in ASCII mode using the program's preferences. We recommend using WS_FTP that has this functionality and is available at http://www.shareware.com/.

    Note: The editors that come with windows such as notepad and wordpad typically create files that contain hard carriage returns (\r) characters. As mentioned before, there are other editors you can download to edit program files. However, even with these you may need to do some reconfiguration.

    For example, with PFE (Programmer's File Editor), when you open an application file, you should see the word "UNIX" in the 8th box to the right on the status bar of PFE. If it says DOS, then you are in DOS mode and must be changed. This can be done by going into the Options/Current Modes menu item. Then select the Saving configuration parameter and click the "Use UNIX Conventions" on.

However, if the files have already been sent over to a UNIX-based web server, you can strip bad characters using:

    $ perl -pi.old -e 's/\r//g' filename

To make the strip process recursive, combine it with the find command like so:

    $ find . -name *.pm -exec perl -pi.old -e 's/\r//g' {} ";"

This command renames the original files to the original name plus .old, (eg. mlm.cgi.old). To get rid of these files (once you've checked that the new ones work), you can use another find command:

    $ find . -name *.old -exec rm {} ";"

    WARNING: As you might imagine, doing recursive, mass deletes is quite a dangerous thing to do. It is much better to solve the bad character problem using one of the other techniques.

[ TOC ]

What You Get When You Have Unpacked The Application

Unpacking will yield a fairly deep directory tree of application and application support files. All eXtropia applications are distributed using a standard directory structure as shown below:

    app_root/ (eg. mlm)
        app_name.cgi (eg. mlm.cgi)
            AppName/ (e.g. MLM)
                SomeView1.ttml (e.g. MLMTOCView.ttml)
                SomeView2.ttml (e.g. MLMFindRecordFormView.ttml)
            AppName/ (e.g. MLM)
            Plugin/ (optional)
                AppName/ (e.g. MLM)
            AppName/ (e.g. MLM)
                CPAN modules that we distribute within
                There are a whole heck of eXtropia modules.

Let's take a look at each of the major areas of this directory structure.

The Application Executable: app_name.cgi

app_name.cgi (eg. webdb.cgi) is the application executable that defines the configuration variables used to modify how the application will run. You must modify this file if you want to change the default behavior of the application. We will discuss how to modify this file in greater detail in Step Six: Configuring the Application.

You should be clear that there is no application logic in this file and very little 'programming-like' things to worry about. The file is really just for configuring the app and launching it from the web.

The HTMLTemplates Directory

All application views are defined as templates such as HTMLTemplates/HTMLTemplates/Appname/TemplateName.ttml (e.g. HTMLTemplates/MLM/MLMTOCView.ttml). Every template may include a number of other templates.

If the application doesn't provide its specific template, the default one from the HTMLTemplates/Default/ directory is served instead.

For example here are some of the templates:

The Datafiles Directory

The Datafiles directory contains files that the application needs to write to and read from. The following files automatically created on the first use of the application.


  • app_name.count.dat

    This file is used to assure unique row identification in file-based data sources. This file will typically contain a single number that corresponds to the last unique row id in the app_name.dat file. Note that if you modify the application executable to use a Datasource other than DataSource::File, you may not need this file. Note also that if the file is not included in the installation by default, it will be created automatically the first time the application requires it.

  • app_name.log

    This file contains log entries covering application usage. Note that if the file is not included in the installation by default, it will be created automatically the first time the application requires it.

  • app_name.users.dat

    This file records all authorized users if the application uses authentication.

    The fields in this data file will be specified by the authentication data source configuration in the application executable. Weíll show you how this is done later in this chapter. Note that if the file is not included in the installation by default, it will be created automatically the first time the application requires it. Note that if you use an authentication driver other than File (such as LDAP), you may not require this file.

  • app_name.dat

    This file stores data for applications that use file-based data sources.

    As we will discuss later in this chapter, all eXtropia applications use Extropia::File-based data sources because they are the most easy to use and the most cross-platform. Thus, every application that requires a data source will have an app_name.dat (eg. webguestbook.dat). However, if you use a DataSource other than File (such as DBI that allows integration with standard RDBMS systems such as Oracle, Sybase, or SQL Server), you may not require this file.

    The fields that a data source contains will be specified in the data source setup section of the application executable. We will discuss this in much greater detail later. Note that if the file is not included in the installation by default, it will be created automatically the first time the application requires it.

The Datafiles/TemplatesCache/ directory is used for cached templates. The Datafiles/Sessions/ directory is used for session files.

You won't really ever need to modify the files in this directory as the application will do it automatically. Unfortunately, however, you will have to make the files and directory writable so that the web server may write data on demand.

The Modules Directory

The Modules directory contains the eXtropia and supporting modules used by the application executable and the application object. You usually will not modify the files in this directory. The directory, and all the files and subdirectories it contains, are provided as tools for the application objects.

It is unlikely that you will ever have to open these files. However, if you do, you should consult the documentation for each module individually.

The ActionHandler Directory

The ActionHandler directory includes the default (Default) and application specific action handlers, which are invoked during request processing and some of them prepare the data that is then served by the templates. Usually you shouldn't mess with these, unless you want to adjust the core functionality of the application.

For some applications, there is ActionHandler/Plugin directory which includes plugins, which are similar to actionhandlers in functionality, but reused by many action handlers. Think of these as a collection of re-usable functions. If you change the data structures used by the application, they are the second place to adjust after the application itself.

Step Four: Assign File Permissions

Expanding the archive file is only one part of the equation of installing a CGI application and getting it to actually run. Frequently, the web server needs to be given special permission to run your applications and have the applications perform their job with the appropriate rights. (Only the case for UNIX-based web servers)

The cardinal rule for setting up web server software is that the server should be given only minimal capabilities. More often than not, it means the web server is run as a user that has no rights to do anything significant -- the user ``nobody''. By default, ``nobody'' usually does not have permission to read any files in directories that you create. However, when you download applications, you must make sure that the applications can be read and executed by the web server software. In other words, ``nobody'' or more correctly ``anybody'', has to be able to get to the files.

In UNIX, the magic command for performing this task is ``chmod'', that is explained in detail in the Customization and Installation FAQ at the following URL:


However, the following chart is provided here as a quick reference.

    PERMISSION      COMMAND               NOTES
    U   G   W
    rwx rwx rwx  chmod 777 filename    Used only for directories that must 
                                       contain writable files.  Use only with  
                                       extreme care.
    rwx rwx r-x  chmod 775 filename    Appropriate for executable files and 
                                       directories being edited by you or 
                                       members of your work group.
    rwx r-x r-x  chmod 755 filename    Appropriate for executable files and 
                                       directories being edited only by you.
    r-x r-x r-x  chmod 555 filename    Appropriate for production executables and 
    rw- rw- rw-  chmod 666 filename    Used only for files that must be 
                                       writable. Use only with extreme care.
    rw- rw- r--  chmod 664 filename    Used for files being edited by you or by 
                                       members of your working group.
    rw- r-- r--  chmod 644 filename    Used for files being edited only by you.
    r-- r-- r--  chmod 444 filename    Used for non-executable production files
    U = User 
    G = Group 
    W = World
    r = Readable
    w = writable
    x = executable 
    - = no permission:

The actual permissions required for the subdirectories and files used by this application are shown below:

    PERMISSION      COMMAND               NOTES
    U   G   W
    rwx rwx rwx  chmod 777 filename    Use only to test. Rarely required for Datafiles.
    rwx rwx r-x  chmod 775 filename    Used for all directories (other than the 
                                       Datafiles directory) and for the 
                                       application executable.
    rwx r-x r-x  chmod 755 filename    Suitable for most applications.                               
    rw- rw- rw-  chmod 666 filename    Never use this.
    rwx r-- r--  chmod 744 filename    Used for files that you can change, 
                                       but anyone else can read. 

An example of how to use the chmod command appears in the following figure:

    NOTE: If you are looking to modify the entire directory structure in one go, you might try the following commands from the UNIX shell prompt:

     $ chmod -R 444 *
     $ find . -type d | xargs chmod a+x
     $ find . -name *.cgi | xargs chmod a+x
     $ find . -name "Datafiles*" | xargs chmod a+w

    NOTE: If it is at all possible to restrict permissions you are advised to do so. For example, if the web server runs as a particular user, only that user should be given permission to use the files.

Step Five: Modify The Perl Path Line

After you have placed all the files on the web server machine, you are almost ready to try out the application. However, before you can run the application you must make a special type of modification: Changing system path values.

CGI applications written in Perl usually contain a magic first line that expects to find the Perl interpreter in a particular directory. In addition, some CGI applications may expect external programs such as Sendmail to be located in a certain location on the server.

Most of the time, the default references to these locations will be correct, because most servers are set up in a standard way. However, you may run across a situation in which the external supporting programs used by the applications are not where they are expected to be. One of the first steps in setting up applications is to figure out the location of these files so that the applications can be modified to reflect the new file locations.

The classic example of a reference to an absolute path is the reference to the location of the Perl interpreter on the first line of the application executable (files that end in the extension .cgi):


    NOTE: When we say 'first line', we really mean first line. You must not have any other lines (including blank ones) before this command or you will get an error.

This line instructs the server to run the ensuing application through the Perl interpreter and indicates where to find the Perl interpreter. The Perl interpreter is a program that reads your application and translates it into a form that your server can run. In the preceding example, the server will expect to find the Perl interpreter in the directory /usr/bin.

Although many servers contain Perl in /usr/bin, others may have installed it in other areas, such as /usr/local/bin, /opt/bin, or /usr/sbin.

If Perl is not in /usr/bin, your first bit of customizing is to find out where your local copy of Perl is, and using a text editor, change this line to reference the correct location.

Besides asking your system administrator, there are several ways of finding files on your system.

If you are using a UNIX-based web server, the best command to try is ``which''. At the command prompt of your UNIX server, you type which perl and receive the following reply:

    $ which perl

In other words, Perl is located in /bin on this system. Thus, you will need to change the first line of your application to the following:


For Windows or Mac web servers you can just email your system administrator to get the right path for Perl or, if you have access, use the operating system find feature.

    NOTE: If you find that Perl is not installed or if a version of Perl older than 5.003 is installed, you will need to ask your system administrator to download Perl from http://www.perl.com/ and install it.

Step Six: Customize The Application

Based on the experiences we have had with other readers, we imagine that you have gotten this far without hitch and that you are excited to move on. However, we recommend that you donít start customizing yet!

It may very well be that if your ISP is configured in a similar enough way to the one used by eXtropia, that you could actually run the application now!

So instead of trudging on, why not skip ahead to Step Eight: Running the Application and try it out? If you get errors, then you'll have to return to this step and work through them. But hopefully, the applications will fit nicely into your environment.

However, even if it does work, at some point, you will want to return to configure the application to work in a way unique to your needs.

To do that, you will need to configure/customize the application. Because the process of customization can be quite overwhelming at first, we have separated out the discussion into a separate chapter on Customizing eXtropia applications.

Step Seven: Modify The Application Look-And-Feel

Modifying the look-and-feel of the applicaiton can be as simple as modifying the view parameters discussed in the section on customization or as complex as writing new view files.

Whichever the case, you should read the look-and-feel section of this documentation for more information.

Step Eight: Run The Application

If all goes well, you should be ready to run the application. Try it out by pointing your web browser at the application on your web server with a URL such as the following:


Hopefully the application will load just fine. If it does not, perhaps the following list will help you identify the problem.

    Error Message   Diagnosis
    Not Found       You probably have the URL wrong and the web server 
                    cannot find the application.
    Server Error    90% of all users will have gotten their permissions 
                    wrong. Are you sure you got them right?   Go back to 
                    Step 4.
                    Another 5% will have gotten the Perl path adjustment i
                    wrong. Go back to Step 5 and check that you are 
                    not in that group.
                    Another 4% will have introduced an error into the 
                    application executable (eg. webguestbook.cgi) while 
                    editing it. Go back to Step 6.
                    The final 1% will have a genuine web gremlin at work. If 
                    you are sure that you are part of this group, try posting 
                    the problem to the free discussion forum at 

    Script          The web server is not set up correctly to execute CGI
    as text in the
    browser window.      

    The browser      The web server is not set up correctly to execute CGI 
    pops up a 
    'Save applications
    As' dialog 
    about unknown 
    file type.  

    Note that you might also try running the application from the command line if you have shell access to the web server.

Step Nine: Debugging The Application

Hopefully, your application will work just fine. If not, you are going to have to do some debugging. The debug section of this guide has some useful tips that will help you debug the problem.

Step Ten: Application Security

Security is paramount. Please read security section of this guide before you make any application public!!

Step Eleven: Test The Application

Finally, you should submit the application to thorough testing by a selected group of beta users. Testing is covered in Chapter 2 so we will only add is a checklist of typical debugging problems that you might run across with eXtropia applications in particular:

Does the application send email?

By default, eXtropia applications tend to disable emailing by default because of compatibility issues. Generally, we leave it up to you to configure mail in the application executable (eg. webdb.cgi) after you get it working. You may have to configure a different email driver for your application.

Do all the templates display?

Are the files in the Datafiles directory being updated (such as the data file, counter file, or users file)?

Can you input 'bad' data into the forms?

Try entering invalid emails for example.

Step Twelve: Register The Application

You should also register with eXtropia so that we will be able to send you bug fixes and security announcements. To register, simply send an email to register@extropia.com. [ TOC ]

[ TOC ]

[ TOC ]
Master Copy URL: http://www.extropia.com/support/docs/adt/
Copyright © 2000-2001 Extropia. All rights reserved.
[ TOC ]
Written by eXtropia.
Last Modified at 11/27/2001