The World Wide Web is big. Really big. As of July of 2008, Google found 1 trillion (that's 1,000,000,000,000) unique URLs on the web at once. The search engine has only indexed a fraction of those web pages (the last count I found was 25 billion in 2006).
But that's nothing compared to the "Deep Web" - a part of the Internet that is not easily accessible by search engines (for example, dynamically generated content that exists only momentarily). People have estimated that the Deep Web is several orders of magnitude larger than the "surface Web". There is, however, another part of the Deep Web that is more sinister: the dark side of the Internet used by criminals.
Andy Beckett of The Guardian wrote:
The modern internet is often thought of as a miracle of openness – its global reach, its outflanking of censors, its seemingly all-seeing search engines. "Many many users think that when they search on Google they're getting all the web pages," says Anand Rajaraman, co-founder of Kosmix, one of a new generation of post-Google search engine companies. But Rajaraman knows different. "I think it's a very small fraction of the deep web which search engines are bringing to the surface. I don't know, to be honest, what fraction. No one has a really good estimate of how big the deep web is. Five hundred times as big as the surface web is the only estimate I know."
"The darkweb"; "the deep web"; beneath "the surface web" – the metaphors alone make the internet feel suddenly more unfathomable and mysterious. Other terms circulate among those in the know: "darknet", "invisible web", "dark address space", "murky address space", "dirty address space". Not all these phrases mean the same thing. While a "darknet" is an online network such as Freenet that is concealed from non-users, with all the potential for transgressive behaviour that implies, much of "the deep web", spooky as it sounds, consists of unremarkable consumer and research data that is beyond the reach of search engines. "Dark address space" often refers to internet addresses that, for purely technical reasons, have simply stopped working. [...]
Michael K Bergman, an American academic and entrepreneur, is one of the foremost authorities on this other internet. In the late 90s he undertook research to try to gauge its scale. "I remember saying to my staff, 'It's probably two or three times bigger than the regular web,"' he remembers. "But the vastness of the deep web . . . completely took my breath away. We kept turning over rocks and discovering things."
In 2001 he published a paper on the deep web that is still regularly cited today. "The deep web is currently 400 to 550 times larger than the commonly defined world wide web," he wrote. "The deep web is the fastest growing category of new information on the internet … The value of deep web content is immeasurable … internet searches are searching only 0.03% … of the [total web] pages available."
In every society there are sides that aren't seen, some of them are bad, some are simply ever changing.
The internet is just another society, however it is one that is ideally suited for enormous and rapid growth, and it has done just that.
It's interesting to see the estimated statistics on it though, but when you think about it, most people at one point or another probably use parts of the dark web without even realising it.
We need DeepGoogle!
That sounds even creepier than "Lost in in the Universe"....
That means that even if everyone on the entire planet had a website (which I assume they don't) than each person would need slightly under 200 pages on their site.
With restrictions in China when it comes to internet use this means that a small percentage of the world, i.e. the major players and corporate institutions dominate the web while the rest of us are lucky enough to have a Facebook!
this number approaches 700 billion pages.
Don't forget that the Defense ministries of the major
countries of the world ALONE account for nearly a
TRILLION PAGES OF NON-ACCESSIBLE INFORMATION,in addition to all the private proprietary non-accessible
pages that account for another 500 billion pages world
wide. My perfect metaphor for this phenomena is a
cruise line shipe with a swimming pool stopped over the Mariana Trench, the deepest part of the ocean.
It is not without a sense of prophecy that Stephen Hawking, the famous Oxford physicist and astronomer predicted that by the year 2035, this planet will be
be visible as "glowing" from outer space from the
google-watts of electricity it will be consuming by that time. Because of the heat generated by the BILLIONS of servers necessary to support a vast 'visible' world wide web AND an'invisible' web,
it will be almost impossible for human existence, or any other living organisms to survive this electric
'furnace' Sounds like we have heard about this problem
before - large bubbles popping(subprime mortgages/derivative default swaps/credit card fraud/
"Is there an echo in this room, or what!!!???
Surf while you can, while the water is still cool. . .!!!