Many years back, search engines were super
basic, but today they are crawled by complex spiders, which are also
referred to as web crawlers, or bots.
That said, there is no doubt that the
development of various search engines has changed how information is
crawled, indexed, retrieved and ranked. Google, Bing, and Yahoo are key
players as far as Search Engine Optimization (SEO) is concerned.
In this article, we are going to delve deeper into how search engines work plus the history and evolution of search engines.
How Have Search Engines Changed Over the Years?
1990: Archie
Archie was the first search engine. It
originated from the term “archives”, only that it does not have the
letter “v”. Archie was created in 1990 by Alan Emtage, who was by then a
student at the McGill University in Montreal.
The role of Archie search engine program
was to download and index directory listings of all the public File
Transfer Protocol (FTP) servers it could locate. Archie created a
database of searchable filenames, which made it easier to find files.
1991: Gopher Protocol
Gopher protocol was created in 1991 by a
team from University of Minnesota namely Mark McCahill, Paul Lindner,
Farhad Anklesaria, Bob Alberti, and Daniel Torrey.
As a communication protocol, its main role
was to distribute, search and retrieve documents in internet protocol
networks. It was a form of anonymous FTP. Gopher is famous for having
been the first search engine to utilize hypertext paradigm.
1992: Veronica
Veronica search engine was named after
Veronica Lodge. It was created in 1992 by Steven Foster and Fred Barrier
who were at the time based at the University of Nevada, Reno.
Its role was to search files stored in
Gopher servers. Also, it was used to perform keyword search,
particularly for Gopher menu titles found in Gopher listings. Veronica’s
backronym was (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives).
1993: Jughead
The Jughead search engine was designed in 1993 by Rhett Jones who was at the time based at the University of Utah.
Jughead, (Jonzy’s Universal Gopher Hierarchy Excavation And Display) was a tool used to retrieve menu information from different Gopher servers.
1993: World Wide Web Wanderer
WWW Wanderer was the first robot designed
in 1993 by Matthew Gray who was at time an MIT student. The role of
Wanderer was to count web servers in order to determine the size of the
web. However, it was at some point used to find URLs and form a database
of websites referred to as Wandex.
1993: ALIWEB
Also known as Archie-Like Indexing of the Web,
ALIWEB was created by Martijn Koster in 1993. It helped users submit
their web pages for indexing. This search engine was based on automated
meta-data collection.
1993: Excite
Excite search engine was introduced in 1993
by a group of six students from Stanford University. Excite helped to
make the search process more efficient by incorporating statistical
analysis of word relationships. Excite officially went online in 1995.
1994: Yahoo
Yahoo was started by Jerry Yang and David
Filo in 1994. Initially, the two had created Yahoo to help them keep
track of their favorite web pages, and it would also entail descriptions
for the different pages. Two years later, the two received funds,
which enabled them to officially launch Yahoo as a corporation.
1994: JumpStation
JumpStation was a crawler-based search
engine, but it is no longer existent. It was created by Jonathan
Fletcher who was at the time based at the University of Stirling in
Scotland.
The main role of JumpStation was to use web crawlers to develop a searchable index of webpage titles and headings.
1994: World Wide Web Worm
This
search engine was a web crawler. It was developed by 1994 by Oliver
McBryan who was at the time a professor at the University of Colorado.
In December 1999, World Wide Web Worm was acquired by GoTo.com, and it
is today part of Overture.
1994: WebCrawler
WebCrawler was developed in 1994 by Brian
Pinkerton who was at the time based at the University of Washington.
Brian was the first person to ever develop a full-text search engine,
which helped to index text on each and every page on the Internet.
1994: Infoseek
Infoseek was developed in 1994 by Steve
Kirsch. It was operated by Infoseek Corporation. Infoseek was not only a
full-text search engine, but it was also used to categorize lists of
web pages. In 1999, Infoseek was acquired by the Walt Disney Company.
1994: Lycos
Lycos was developed in 1994 by Michael
Mauldin who was at the time based at Carnegie Mellon University. Lycos
was famous for its huge catalog of over 57 million indexed documents.
Also, it was famous for not only relevance retrieval, but also for word
proximity, and its ability to match prefixes.
1995: Alta Vista
Alta Vista was a search engine developed in
1995 by DEC. This search engine enabled people to utilize a natural
language user interface, as well as advanced search techniques.
Additionally, Alta Vista allowed people to search for music, videos, as
well as photos.
1996: Inktomi
Inktomi was developed in 1996 by Eric
Brewer and Paul Gauthier during an experiment at the University of
California at Berkeley. The crawler was famous for its ability to search
more than 48 million pages on the internet. In 1999, Inktomi launched a
directory search engine using the concept induction technology. It was
acquired by Yahoo in 2003.
1996: HotBot
This application was developed in 1996 by
Inktomi in collaboration with HotWired. It was famous for its ability to
provide customized website pages and ads based on the user’s type of
browser.
1996: Backrub
Backrub was developed in 1996 by Larry Page
and Sergey Brin who were at the time students of Stanford University.
The key role of Backrub search engine was to rank websites based on the
relevancy of their inbound links, as well as their popularity. Backrub
was given its name because it analyzed backlinks to websites. Later,
BackRub became known as Google.
1996: LookSmart
LookSmart was developed in 1996 by Tracey
Ellery and Evan Thornley. Initially, LookSmart was known as NatGet.
LookSmart was created as a directory service. Today, LookSmart was the
most notable paid insertion directory on the World Wide Web.
1996: Netscape
Netscape was originally known as Mosaic
Communications, which was founded by Jim Clark and Marc Andreessen of
the University of Illinois at Urbana–Champaign. Mosaic was Andreessen’s
idea, and he started working on the search engine in 1993 when he was
still at the University.
Netscape was acquired by AOL in 1998.
Today, Netscape works in partnership with LookSmart, Lycos, Overture,
Ask Jeeves and Google.
1997: Search Engine Watch
Search Engine Watch was developed in 1997
by Danny Sullivan. The key role of this online search engine magazine
was to offer critical information regarding the search industry, how search engine work and how to search the website. How to rank websites was also part of the information offered by Search Engine Watch.
Search Engine Land
was another brainchild of Danny Sullivan which is still alive and thriving today even after Sullivan left the company in 2017.
1997: Ask Jeeves
Askjeeves was developed in 1997 by Garrett
Greuner and David Warthen who were at the time based in Berkeley,
California. When Garrett and David created Askjeeves, they wanted to
help people find answers for their most pressing issues more so because
they were aware that not everyone knew about Boolean search and syntax.
That said, they came up with Ask Jeeves, a
search engine that featured Natural Language Processing, or NLP. Ask
Jeeves, now Ask.com, enabled its users to search for answers using their
day-to-day language including asking questions.
