At some point along the way webmasters around the net realized that HTML
(1) was too limited to do many of the things that they
wanted to accomplish.
How could a webmaster display the current time and date on every page
accessed by a client? How could she collect information about clients
who were accessing her web site? How could she create a web site that
was more than just an information warehouse, but a meaningful and dynamic
Certainly, HTML was great for distributing "pre-prepared" web pages on
request. A client would use a web browser to contact a web server and use
HTTP to ask the
web server for a specific HTML document. (2) The web server would then send the requested document back to
the web browser which in turn, would display the document as defined by
the HTML to the client.
Pretty nifty really, and far superior to older technologies like gopher
and ftp. However, the interaction between the client and the server was
still extremely trivial. The server could only provide HTML documents
that had been specially encoded by a webmaster, and that had been placed
in certain publicly-accessible directories. The interaction between web
browser and web server was pretty mind numbingly simplistic and the
coolness of surfing through hyperlinks quickly became droll.
HTML fell short for anything truly "dynamic".
For example, to put the current date on every page using only HTML would
require a webmaster to manually edit every file, every day. As you can
imagine, this got tiring very quickly for sites with more than 5 pages.
Webmasters needed a way to have HTML pages created and modified
"on-the-fly," with information that could change weekly, daily, by the
second, or for each and every request. And they needed those pages to be
modified automatically, without their constant oversight.
As it so happens, the hardware that web server software runs on typically
has quite a few resources that can be utilized to help solve these
problems. Not only do servers have processing power to spare, they also
have a battery of applications (such as e-mail, database manipulation, or
calendaring) already installed and ripe for utilization.
And thus was born CGI (Common Gateway Interface). (3)
CGI: The Birth of Server Side Web Programming
As with most computer jargon, the term Common Gateway Interface can be
fairly meaningless at first glance. So, before getting into what CGI can do,
let's take a moment to define what CGI actually is.
- Common - CGI programs can be written with many languages. CGI
can be programmed in C, C++, Java, Perl, Visual Basic, or any other
language that can accept user input, process that input, and respond
Further, CGI works with many different types of systems. CGI works
on Mac, NT, Windows, OS2, Amiga, UNIX, and pretty much every
operating system that runs web server software.
By the way, if you use a "platform independent" language like Perl or
Java to write your CGI script, the exact same CGI script can be moved
directly from one platform to another without it breaking!
- Gateway - CGI's usefulness does not stem from what it can
accomplish by itself, but what it can accomplish by making partnerships
with other resources.
I often see CGI as a middleman or a translator whose job it is to help
more powerful resources like databases, graphics generation programs,
or network applications talk to each other and work towards a common
CGI is the gateway between the lone web surfer with her trusty web
browser and the vast web of computers (each with their own
specific language and protocols) and computer programs (each with their
own interfaces and methods of output).
CGI translates between the language the client speaks (perhaps HTMLized
English) and the multitude of languages spoken by the resources the
client wants to utilize (such as SQL for relational databases). This
is a crucial job, cause let me tell ya, my grandmother does not want to
know how to speak SQL when she is browsing the web!
- Interface - CGI is not a language. Neither is it a
program. CGI is a standard of communication, a process, an interface
that provides well-defined rules for creating partnerships. The
benefit is that if everyone follows the rules of the interface, then
everyone can talk to each other. Thus, typically we say that we
write "CGI programs" or "CGI scripts" that perform the functions of the
common gateway interface.
Let's look at the CGI processes in the following chart.
Okay, so that is probably a lot of abstract stuff to take in all at once,
especially if you have not worked with CGI already. So let's back up a
minute and go over what CGI is by looking at it in the wild. Let's look
at some examples of CGI in action.
You can easily see that CGI makes for a much more profound surfing
experience allowing web sites to offer useful and compelling services to
surfers who may be interested in information or products offered. (5) However, there is a dark side!
Well, as you might expect, for all its dynamism, CGI was not a holy grail.
In fact, there are a lot of sysadmins out there who would be ecstatic if
CGI were outlawed. CGI simply causes too many problems.
- CGI introduces security holes. Lincoln Stein writes the
following eloquent warning on the problem:
Unfortunately, there's a lot to worry about [when running a web server
with CGI]. The moment you install a Web server at your site,
you've opened a window into your local network that the entire Internet
can peer through. Most visitors are content to window shop, but a few will
try to peek at things you don't intend for public consumption. Others,
not content with looking without touching, will attempt to force the
window open and crawl in.
It is one thing to allow any freako on the Internet access to your web
server, when the communication is controlled through the boundaries
defined by HTTP and implemented by web browsers. It is another thing
to allow a stranger access to an unlimited amount of applications housed
on the same server through a renegade CGI script.
It's a maxim in system security circles that buggy software opens up
security holes. It's a maxim in software development
circles that large, complex programs contain bugs. Unfortunately, Web
servers are large, complex programs that can (and in
some cases have been proven to) contain security holes.
Furthermore, the open architecture of Web servers allows arbitrary CGI
scripts to be executed on the server's side of the
connection in response to remote requests. Any CGI script installed at
your site may contain bugs, and every such bug is a
potential security hole.
In the WWW Security FAQ, Stein identifies four overlapping types of risk:
- Private or confidential documents stored in the Web site's document
tree may fall into the hands of unauthorized individuals.
- Private or confidential information sent by the remote user to the
server (such as credit card information) might be intercepted.
- Information about the Web server's host machine might leak through,
giving outsiders access to data that can potentially allow them to
break into the host.
- Bugs can allow outsiders to execute commands on the server's host
machine, allowing them to modify and/or damage the system. This
includes "denial of service" attacks, in which the attackers pummel
the machine with so many requests that it is rendered effectively
I recommend checking out the following CGI Security sites if you are
interested in getting more detailed information.
- CGI is at the mercy of HTTP.
It is important to note that HTTP only provides for a one-time,
question/answer type of communication. After all, it was defined
for web browsers and web servers to exchange HTML documents. Thus,
by definition, HTTP is not very dynamic.
One-time, question/answer communication works like this: the web
browser and the web server are only connected as long as it takes
for the web browser to send one document request and the web server
to send one requested document. If the browser wants a second
document, it must recontact the server and ask again. Each request is
new. The server maintains no ongoing connection or record of past
While this is very efficient for network traffic (because the
bandwidth is only used when information needs to be exchanged), it
is a big pain in the butt when it comes to CGI, because CGI is about
conversations, not about one-time question/answers
Imagine that when talking on the phone you had to hang up and redial
every time you said something and received an answer. Imagine
further that every time you called back you had to go over every
previous exchange before you could get to the next piece. That is the
way web browsers work with web servers and this makes communication
This makes communication tough for three reasons.
First, if the client
and server are to maintain information over several exchanges, the CGI
must be responsible for keeping a running dictation of the conversation
so that every time there is a new exchange, the web server can consult
the record of the entire conversation up to that point. This is what
CGI aficionados call "maintaining state". The CGI script must be
able to keep track of certain information like username or the contents
of a virtual shopping cart for every "instance" of a script. (6). That is, there must be a way to tie the current
HTTP request to related ones that have gone on before. Maintaining
state is possible with CGI using hidden variables, by encoding the
URL, or by maintaining a state file on the server, it's just not easy or
Second, every set of question/answers causes the web server to
execute a unique instance of the CGI script. This is pretty
expensive, especially on a high volume web site that may have 100
instances of a CGI script executing at any given moment, each,
perhaps, with its own Perl interpreter. (8) Every
one of those CGI scripts takes a little bit of
umph out of the server engine. If we were not limited to
question/answer format, we would not need to execute so many instances.
Consider the following CGI application executing....
Server: Welcome, what would you like (CGI script executed once)
Client: I would like a list of products you are selling
Server: Here is a list (another one)
Client: I want to purchase this product
Server: Okay. (yep)
Client: I'm done, can I check out?
Server: Yes, what is your credit card number? (another script)
Client: Here it is.
Server: Thanks (another instance of the script that also emails the
results to some store admin) (9)
Yuck, this exchanged caused 5 instances of the store script to be
executed as well as 5 Perl interpreters if the CGI script was written in
Third, CGI is extremely slow. Every time the client does something,
the CGI Script must recreate the entire dialog and execute a new
request. Add a new item to a virtual shopping cart - new request.
Calculate a running total - new request. Submit an order - yet another
request. Each request takes time and since the CGI script must be
executed again and everyone must wait for a busy Internet.
- CGI is ugly. Finally, CGI scripts produce fairly ugly
user-interfaces. Basically, CGI is limited to bland HTML-based forms
and whatever bells and whistles can be provided by surrounding HTML
layout. Thus, no CGI application looks like your swank bootleg
copy of Word.
This may not seem like a big issue at first, but when you start
competing for web hits with multi-million dollar companies, image is
indeed everything. CGI simply cannot compare with web based
applications that are not limited to HTML.
Well, those are some pretty damning flaws. Like I said, many systems
administrators would love to see CGI fall off the face of the Earth.
Unfortunately for those system administrators, the fact is that CGI has
continued to be the workhorse of the web, powering 90% of the dynamic web
pages out there.
The fact is that CGI, especially CGI/Perl is easy to work with and most
non-technically oriented webmasters out there can get their needs filled,
and filled right away. However amazingly, brand-fantasmagorically
wonderful other technologies sound, they are still vaporware as far as
the average web developer is concerned. Either the ISP does not provide
those technologies, or the learning and development curve is too steep or
expensive. And of course for small applications typical of most
websites, the big guns of C or C++ are just overkill.
CGI, for all its flaws, works, and works pretty darn well if done
carefully. "Intranet" developers with massive budgets can yack all they
want to about servlets and SQL gateways and Server Side Includes and
customized server applications written in Java, but for most "Internet"
developers out there, CGI is the only tool available for solving their
problems. And with creativity and care, CGI can also be the right tool.
However, this is not to say that other technologies are not extremely
useful. Several technologies have proven to be just as important as CGI
for the average Internet developer. These technologies focus on putting
the demands of computation in the hands of the client instead of the
server. Thus, things like processing simple requests, maintaining state,
and GUI (Graphical Usr Interface) presentation are handled by the web
surfer's own computer instead of being handled by some web server
hosting a site.
Client-side programming is based on the idea that the computer that the
client is using to browse the web has quite a bit of CPU power sitting
there doing nothing. Meanwhile, web servers are being tasked to death
handling hundreds of CGI requests above and beyond their regular duties.
Thus, it makes sense to share some of that burden between the client and
server by taking some of the processing load off the server and giving it
to the client.
As it so happens, much of what CGI does, can be handled on the client's
side. Typically, the only time the server needs to be involved is when
the web application needs to send email or access datafiles. Things like
maintaining state, filling out forms, error checking, or performing
numeric calculation, on the other hand can be handled by the client's own
computer. The web browser need not check back with a CGI script
every time the user wants to do something. A "script-enabled" HTML page can
carry with it instructions on how to handle certain events.
In the following figure, client-side scripting has reduced server load by
over 80% for every client accessing the CGI script. And of course,
since most of the processing is handled locally, the application as a
whole runs 5 times faster.
Obviously, this solves many of the problems posed by CGI. Client-side
applications maintain security by keeping server processing to a
minimum. They are not restricted by HTTP and the GUI can be as pretty
and sleek as any traditional software package out there.
(Netscape Navigator) and VBScript (Microsoft's Internet Explorer). Both
technologies allow web programmers to encode short program "snippets" into
their HTML documents that can be executed by a web browser.
and Reaz Hoque
other side of the coin,
provides a good list of samples for VBScript.
Actually, script-enabled HTML pages can be fairly dynamic and do indeed
cut down on the work of the sever. Of course, in any real application,
there will need to be a CGI script on the server to email results or
access data, but much of the work, perhaps 75% of it, is done by the
client. This can cut down server load by 80% on complex applications.
Unfortunately, script-enabled HTML pages have their problems too. The
most obvious problem, of course, is that the web browser program must be
able to interpret the language used for scripting. And since Netscape and
Microsoft are too knuckleheaded to build upon common standards, we are
Internet Explorer and VBScripts do the same when viewed with Netscape.
Thus, client-side scripting has remained primarily useful only for limited,
controlled intranets where webmasters can be sure that all users are
using the same browser software to view web pages.
for quick jobs with little complexity. Ticker tape animations and sub
totaling are one thing, but a true web application requires a bit more
Platform Independent Client-Side Applications with Java
That very umph comes with Java. Java was originally developed at Sun
Microsystems in 1991 to provide a platform-independent programming
language and operating system for consumer electronics (TV sets, toasters
In syntax and execution, Java is a lot like a simplified version of C++.
("simplified" should be read in the previous sentence as "an improved").
It is a highly robust, distributed, high performance, object-oriented,
multi-threaded language with all of the usual features. As such, it builds
upon years of C++ development, taking the good and dispensing with the
As it so happened however, Java did not make it into the consumer
electronics market. Instead it wound up in our web browsers.
Java seemed to be a perfect fit for the web. The language itself was
extremely small (as it was built to go inside toasters and alarm clocks
with tiny amounts of memory). Thus it could quickly be transferred over
Further, Java was platform independent. That is, any computer with a Java
virtual machine can run a Java program. Programs can be written anywhere
and be run anywhere. This is crucial because, as we saw in the case of
the client-side scripting languages, if a language can not run on any
machine, it cannot be used on the web that must service every machine,
language, and environment imaginable.
Platform independence works because Java is an interpreted rather than a
compiled language. Unlike C or C++ code, when Java is compiled, it is not
compiled into platform specific machine code, but into platform
independent byte code. This byte code is distributed over the web and
interpreted by a virtual machine (typically built right into a web browser
these days) on whichever platform it is being run. Perhaps a picture
would be useful...
Thus, as a programmer, you need only concern yourself with the generic
Java programming language and compile your applications into bytecode on
whatever system you are using. You can then be assured that your bytecode
will be executed correctly whether your clients are using Macs, Pcs, Unix
boxes or anything else.
Java, of course, demands books worth of explanation and description. So,
of course, we will not delve too deeply into the language here. Instead,
I recommend that you browse through the resources collected at Gamelan which is the be all and end all of
Java resource sites. There you can sample several Java programs yourself
and see how amazing Java really is.
Did I say amazing? Well, Java is certainly a great addition to every web
developers tool box, but as you might have expected, Java has as many
drawbacks as any of the other tools we've discussed already.
Though Java can create interfaces that go way beyond the capability of
and portable, Java still has serious restrictions.
Of particular concern are the security restrictions built into Java such
as the fact that Java programs (Java applet specifically) cannot easily
write files to the local harddrive or get data from servers other than the
ones they came from. While this may make the public more confident about the
language (an important thing and perhaps worth the limitations it causes),
it makes Java programs fairly useless for the average developer who
absolutely needs such capabilities to create full featured applications.
Further, Java programs with a lot of logic take longer to download. If
you went to the Gamelan site linked above and tried to run some example
Java apps, you certainly found that you had to wait quite a bit for them
Similarly, because the programs run on the client's machine, they do not
have access to resources on the server. Thus, a Java program cannot even
query simple flat-file databases located remotely without a proxy (some
other program working as a helper on the server).
Finally, Java is still a new language. As such, it is plagued by all the
bugs, inconsistency and incompatibilities that any new language is faced
with. Though Java boasts platform independence, in reality, programs run
differently from platform to platform...if they work at all. Further,
though programs might be platform independent, they are not browser
independent. Each browser, in fact each operating system, has its own
buggy virtual machine that produces different output for the same
program. Thus, when you distribute a Java program, you can never be sure
exactly how it will run, or if it will be run at all.
Though the restrictions of Java are being addressed slowly, the picture
looks bleak in the short term (next couple of years) for the Internet
developer. Although, code signatures, and other security fixes are
arriving, they will still cause complications for the average web
developer with regards to centrally storing information and trusting it.
Security will be a continuing thorn in our sides.
But even still, if all of the well-publicized inconveniences of Java were
solved tomorrow, there would still be issues preventing the average web
developer from writing all their web apps with Java. For
example, not everyone has a database to program against or can afford
the cost of JDBC (Java database connectivity) proxy servers. In fact, it
is safe to say that "most" web developers do not have those tools to work
with. Typically, Internet Service Providers do not allow customers to
run servers of any kind through their account, let alone complicated
database servers. Thus, in order to perform database management
functions essential to many applications, the average web developer will
still need to work with flat files on the server hardware...and this
There are also issues preventing the spoiled "intranet" developer from
using Java as well. For example, the JDBC standard will not
necessarily help in a corporate environment in which some sort of proxy to a
real database server may still be needed that can communicate across a
firewall with a web server. Not only will Java be blocked by a firewall,
but it cannot use standard encryption standards to provide secure,
In short, though Java is a profound addition to our toolbox, it is not the
answer to all our woes.
Conclusion: Stocking your Toolbox
As any good technician knows, there is no such thing as a "best" tool.
The best tool is dependent on a whole host of factors from the type of
task at hand to the personality of the marketing director. The best tool
is a fantasy.
Instead, every web developer should have at her disposal a wide array of
tools to solve problems. Sometimes a server-side solution will be
appropriate, other times a client-side solution will be best. Your main
goal as a web developer is to develop an intuition about when to use
That said, I would like to suggest one combination of tools that I see as
becoming extremely important for all web developers. The combination is
that of CGI and Java. Consider the following Problems and Solutions...
- The average "Internet" web developer has probably already picked up
Perl/CGI programming. Most have not picked up Java with the
exception of being able to code GUI interfaces using various visual
tools such as Symantec's Visual Cafe or Microsoft J++.
- A Java to CGI interface leverages existing Perl/CGI knowledge so that
the core program logic can be located on a server while merely having
to code a thin (very small and easily downloaded) GUI Java client. In
addition, a developer experienced
in Perl will be able to whip out 80% of their program in a short
period of time using a language like Perl while leaving
a mere 20% (The GUI) left to Java (A hard language for most people).
- Internet developers who do work for sites on a virtual web server
or an ISP typically cannot use Sybase, Oracle, or another commercial
database to store data via JDBC. Frequently, the ONLY option that
these developers or consultants have is to do flat-file processing
using CGI/PERL that generally has precluded the use of Java.
- A Java to CGI interface will allow applets to be created that
can use flat-file databases that an average small-business can afford
- Developers who have already invested a lot of time creating CGI/PERL
for their site do not want to rewrite all their applications in Java.
- A Java to CGI interface will allow existing applications to be
leveraged by allowing a developer to create a Java applet on top of
an existing CGI script with minor modifications to make the CGI output
data conducive to interpretation by the Java applet
As you can see, the benefits and flaws of Java and CGI compliment each
other very well. Using Java frontends and CGI backends presents an
excellent opportunity for web developers on the Internet to create fully
featured applications with the available resources. I would recommend
that every web developer make sure to study up on the interaction of Java
and CGI to be prepared for the contracts that will come forward over the
next few years.
- The Web Developer's Resource Library has an excellent
Primer if you need a refresher on what HTML is! But basically,
HTML (Hyper Text Markup Language) is a way of describing how a web
browser should display text and images. HTML codifies page layout
into a series of instructions called "HTML tags". The job of Web
Browser programs is to translate ugly HTML instructions into the
beautiful web pages we see while surfing.
- Okay so there are quite a few jargon words in that
sentence which might need quick explanations.
In this column we will use the word "client" to refer to a person
who is using a "web browser" program like Netscape Navigator or
Internet Explorer to display HTML documents received from a "web
A web server is a combination of hardware (an actual computer that
stores all of the HTML files) and software (the program that listens
for web browser requests and utilizes the hardware resources to
fulfill those requests).
Web browsers and web servers communicate using HTTP (Hyper Text
Transfer Protocol) which provides a communication standard for efficient
and intelligible dialog. Essentially HTTP allows a web browser to
contact a web server somewhere on the web and ask for a specific
document (or resource). It also allows the server to send the
requested document (or execute the resource) back to the web browser.
Truthfully much of what is done by CGI can also
be done using SSI
(Server Side Includes) which is a service provided
by web server software in which certain HTML comment tags can be used
to execute commands. SSI will not be covered this month since it
demands its own article, however, for the purposes of this
introduction, SSI programs are similar enough in theory to CGI programs
that they can be thought of as the same thing.
When you get some software for your computer and
you have to get the special "Mac Version" or "Windows Version", you
are getting a "platform dependent" program. Unfortunately, when
you move from being a PC user to being a Mac user, you have to buy
all new programs because the programs you bought for Windows will
not work on Mac. The beauty of web programs is that they are
typically "platform independent" which means that you can run them
anywhere. Whether you use a PC, Mac or Unix box, the programs will
work just fine.
CGI is not the only form of server-side scripting
available, of course. For example, Netscape's Live
Wire is an online development environment for Web site
Netscape's scripting language, to create server-based applications
similar to CGI programs. Unlike CGI programs, however, LiveWire
applications are closely integrated with the HTML pages that control
them. However, non-CGI server side strategies are best covered in
their own article.
You can think of an instance of a script
as a unique and independent version of a generic script. It is called an
"instance" because ten web surfers could all execute a CGI script at the
same time. Though each web surfer would be using the same generic
CGI script, each instance of that script would be personalized to that
web surfer. Thus you may have ten instances of the exact same script
running in parallel on the web server hardware.
Hidden variables allow you to maintain
state using the HTML "Hidden" form tag. Essentially, you
include information in your HTML form that will not be visible to
the user when they look at the form in their web browser window,
but which will be transferred to the CGI script with the
user-supplied data. The format of the tag looks something like the
<INPUT TYPE = "HIDDEN" NAME = "first_name" VALUE = "selena">
When the CGI script processes the information that the user enters
into the HTML form, it will also receive the variable "first_name" with
the value of "selena" as well as "last_name" equal to "sol".
<INPUT TYPE = "HIDDEN" NAME = "last_name" VALUE = "sol">
If the user is not using a FORM tag to navigate through a site,
the admin can still encode state information in the URL by
using the HTTP standard for URL encoding. For example, the following
hyperlink would send the same info as above to the CGI script.
<A HREF = "www.extropia.com/test.cgi?first_name=selena&last_name=sol">click
Notice that variables to be passed along are listed after the question
mark, name/value pairs are separated by the ampersand sign, and the
variable name and variable values are separated by an equal sign.
Finally, the CGI script may write out state information to a file on the
server and then simply pass along the location of the file using one or
both of the above methods. This is best when there is a large amount of
By the way, maintaining state can also be achieved using
however, we will not address cookies here because they require their
Perl is a fun language to use because it keeps
the nuts and bolts of machine code as invisible as possible. One of the
does this is by adding an extra step between you and the computer. This
extra step is called a "Perl interpreter". This interpreter (which your
sysadmin must install) reads a Perl program that you write and
translates it "on the fly" into machine code that can be understood
by your computer. Your "executable" can then be moved to any other
system with a Perl interpreter and be run without problems.
Further, the code can be easily modified and understood.
Unfortunately, in order to run your executable, you must also run the
interpreter and this can be expensive in terms of server resources.
In more intense languages like C or C++, there is no interpreter.
You must use a special "compiler" program to translate your code into
machine code. This affords greater power to your programs since you do
not need to run a separate interpreter when you run your executable, but
it does mean that executables are specific to each operating system and
that the source code is stored separately from the executable code.
Notice that CGI scripts must be smart enough to
answer all sorts of questions.