"... knowledge is doubling every ten years."

So said Eric Johnson, president of the US Chamber of Commerce -- in the mid-twentieth century -- as he noted the constancy of change.

That was before the Internet.

With the advent of the World Wide Web and its ability to make information easily available anywhere, knowledge was doubling approximately every 18 months by 2004, according to the American Society of Training and Documentation (ASTD). And IBM predicts that in the next couple of years, information will double every 11 hours [PDF].

This vastly increasing store of knowledge, with its virtually unlimited and unfettered access, has changed the way we use it. We once made a faithful trip to the library or bookstore to obtain books on a particular subject. Now, in areas where change is quick, the wait for such books would be interminable and, when they finally arrived, the information contained in them hopelessly outdated.

Today I get my information from the Internet. Not only from the creators of the products I'm working with, but also from the collective consciousness of all of the other people in the world working with the same products, and willing to share that information for the benefit of all.

Which has essentially turned the knowledge paradigm on its head. No longer do I have the problem of finding a piece of information. I am now faced with the difficulty of sifting out the single bit of knowledge I require from a veritable ocean of possibilities.

The de facto tool to do this is Google, which does an admirable job, especially when given multiple search strings. Google is often augmented by others tools such as blog search engine Technorati. But even Google was challenged when I tried to find information about Eric Johnson, quoted above. Though there was plenty of information about Eric Johnson, apparently one of the most respected guitarists on the planet.

Companies such as Cuil have stormed onto the scene and quickly dropped out of sight, suggesting that the answer to the problem of search is a bigger index, but that doesn’t seem to be it. The difficulty is more one of context.

An ideal search engine or information filter would understand based on the context of my query which Eric Johnson I was looking for. Or when I searched for "Apple" it would know that I was interested in the computer company, and not the delicious fruit. Rather than presenting me with the ocean of possible choices, it would already have narrowed the field down to the slice I was interested in, and then it would allow me to further refine my query using smaller and smaller subsets.

In a move that would make privacy advocates apoplectic, it might also be useful to save your search history and use it as a search parameter, allowing the engine to deduce context from previous searches.

Semantic search engines such as Hakia purport to do something similar by determining links between words on the page. For example, a page containing the words "first" and "union" could refer to First Union Bank. However, there is still no sense of the meaning or context of the page -- just some educated guesses. They may help, but they will not yield the results that come from understanding the actual context, or meaning, of the page.

The knowledge doubling curve is only going to increase, seemingly exponentially, and finding a needle of information in a haystack is literally a billion-dollar problem. Whether the solution comes from an existing company or a new one, the problem must be solved before we are literally strangled by a glut of information.

Larry Borsato has been a software developer, marketer, consultant, public speaker, and entrepreneur, among other things. For more of his unpredictable, yet often entertaining thoughts you can read his blog