The internet has been going for just over 20 years now, and during that time it has got smarter and larger.

When the internet started with its first web site at Cern finding information was easy, clicking on the links within the web pages could take you to the whole internet as the whole internet was just one web site. As more web sites were added to the internet the web sites would link to each other so information could be found more easily.

However as the number of web sites and web pages grew linking each new web site and web pages to the existing web sites got more difficult for the web site owners.

First wave of web search

The internet adapted it got smarter, web directory sites emerged where people organised web pages by subject and topic into directories. If you needed to find some information on the internet you would start at a web directory site then click through the subjects or topics until you reached the desired web pages. For example to find out more about the original terminator movie you might visit one of the web directories then click on the link “Arts”, then “Movie”, then “Titles”, then “T”, then “Terminator Series”, then “Terminator, The”, the user would be given a list of web pages from different web sites about the original Terminator movie.

This meant that web sites owners didn't need to list all the other web sites on the internet and could focus on just building content for their own web site. Examples of web directory sites include The World Wide Web Virtual Library and the Open Directory Project.

However as the number of web sites and web pages grew keeping the web page links up to date in the web directory sites became more difficult and for users to navigate through increasing directory layers to find web pages became more time consuming.

Second wave of web search

The internet adapted it got smarter, web search engines emerged where computers automatically crawled the internet to find web pages, the computers would index the page to extract keywords that could be searched against. To find out more about the original terminator movie a user would type “original terminator movie” into the search box in the web search engine, the user would be given a list of web pages on different web sites about the original Terminator movie.

This meant more up to date information could be found as the computers periodically checked web pages more frequently than people and for users finding information through a search box was generally quicker than clicking through the web directory layers. Examples of web search engines include Google and Bing.

However as the number of web sites and web pages grew and web pages contents changed more frequently even the automatic crawling computers have trouble keeping up. Some estimates ( The Deep Web: Surfacing Hidden Value ) suggest that the internet is 500 times larger than what can be indexed by web search engines.

Tron

Tron before the smart update

Third wave of web search

The internet needs to adapt so it can get smarter.

Now imagine web sites with a web agent which is a special web page that could tell you about every current web page on the web site based on a simple query. The way that the special web page acted and responded would be the same across all web sites so that their responses could be collated and sorted through web agent collector web sites and given to the user.

To find out more about the original terminator movie type the user would type “original terminator movie” into the search box in the web agent collator web site, the web site would call many web agents such as Amazon, eBay and Twitter and collate web page responses such as

  • “The Terminator DVD Special Edition $10 only 2 left in stock, used and new from $2” from Amazon
  • “Terminator DVD with Arnold Schwarzenegger, Linda Hamilton, Michael Biehn and Paul Winfi $0.90 - 2 bids - 3 hours left (17:11)” from eBay
  • “@Cinebexleyheath - 50m ago - Ever wanted to see the original terminator movie on the big screen? We are having a one off showing on the 23rd June at 8pm” from Twitter

The information is more up to date such as current stock level, number of bids, and current bid amount as it has come directly from the web site, and not old cached copy of any web page.

The web agents could also make other suggestions that can also be presented to the user such as “buy terminator quadrilogy dvd”, “buy terminator blue ray” and “follow Arnold @Schwarzenegger” to help guide or refine the query.

So whats stopping this......

The technology to put this together can be done today, we have already built a web site that shows the principles working. It's more around the breaking the status quo of how things currently work which is the bigger problem.

Where the first two iterations of the internet didn't require much effort from web sites owners to make the internet smarter as they could be handled by external web sites, the last proposed iteration does. It requires some intelligence to be built into the web agent which will have costs associated with it.

The current way information is found through search engines may be good enough for some web site owners particularly if their web pages don't change frequently. However web sites that provide more dynamic content that changes frequently and is harder to index via search engines are more likely to adopt this new approach.

There is also a bit of chicken and egg problem, web site owners aren't likely to build a web agent for a web agent collator if the web agent collator web site does not get any users. No users are likely to visit the web agent collator web site as there are no web agents supplying responses to view.

To break the status quo several visionary web site owners need to come together to address the chicken and egg problem. These web site owners are likely to be large web sites with dynamic content that changes frequently.

What's next....

To break the status quo we need those visionary web site owners to come together to help build the smarter internet.

Please contact us if you would like to help build the smarter internet.

About Mark Sivill

Mark Sivill has written 14 blogs on fluxgain.

Looking for interesting projects to undertake within IT. Currently focused around cloud technologies such as salesforce and Heroku.