Though Google is responsible for where the search engines stands today, actual search engine was invented much before Google incorporated.

Alan Emtage, a student at McGill University, created the first search engine in 1990 and he named it ‘Archie’. Back then there was no world wide web! FTP was the medium to share data. It was effective in smaller groups but the data became as much fragmented as it was collected. Archie helped solve this data scatter problem by combining a script-based data gatherer with a regular expression matcher for retrieving file names matching a user query. Essentially Archie became a database of web filenames, which it would match with the user’s queries.

Archie had such popularity that in 1993 the University of Nevada System Computing Services group developed ‘Veronica’. Veronica served the same purpose as Archie, but it worked on plain text files. Soon another user interface called ‘Jughead’ appeared with the same purpose as Veronica; both of these were used for files sent via Gopher, which was created as an alternative to Archie by Mark McCahill at the University of Minnesota in 1991.

Now the challenge was to automate the process. And the first internet robot was introduced. Computer robots are simply programs that automate repetitive tasks at a speed impossible for humans to replicate. He initially wanted to measure the growth of the web and created this bot to count active web servers. He soon upgraded the bot to capture actual URL's.
His database became knows as the Wandex. The Wanderer was as much of a problem as it was a solution because it caused system lag by accessing the same page hundred times a day.

By December of 1993, three full-fledged bot fed search engines had surfaced on the web: JumpStation, the World Wide Web Worm, and the Repository-Based Software Engineering (RBSE) spider. The JumpStation gathered info about the title and header from Web pages and retrieved these using a simple linear search. As the web grew, JumpStation slowed to a stop. The WWW Worm indexed titles and URLs. The problem with JumpStation and the World Wide Web Worm was that they listed results in the order that they found them, and provided no discrimination. The RSBE spider did implement a ranking system.

Brian Pinkerton of the University of Washington released the WebCrawler on April 20, 1994. It was the first crawler, which indexed entire pages. Soon it became so popular that during daytime hours it could not be used.

Continue ...