?What is the Deep Web and Why Is It Worth Exploring
Words like “deep web” or “dark net” are sometimes used interchangeably although they are altogether
different entities. The simple explanation, and the reason why deep and dark are sometimes confused, is
that the majority of users only use the “surface web”—the most popular and heavily linked websites on
the Internet. That leaves potentially millions of privately owned and operated websites that no one has
officially classified or “indexed” (such as what Google and Bing do by crawling public websites), and
perhaps only a handful of people have actually seen.
Some individuals have likened the deep and dark web to trying to fish in the ocean, or perhaps an
iceberg that rises to the surface but actually goes thousands of leagues under the ocean. As far back as
2001, the deep web was said to be “orders of magnitude” larger than the surface web, and modern 2016
reports suggest that hidden websites and databases are 500+ times greater in number than everything we
see on the surface web.
Now theoretically, there are millions of websites that you would ordinarily never find, unless of course,
you were able to penetrate the deep and dark web.
What we see with our favorite sites (Facebook, YouTube, etc.) could be called the filtered media. You
will have access to only filtered information and news information, as well as entertainment pages,
which have been carefully selected for your viewing. This is a similar practice to how executives plan
network or cable television.
Search sites like Google and Bing may be able to link you to obscure pages that have very low
popularity, thanks to a longtail keyword search such as “Historical Records from the 1800s.” It may not
be a well known page but you can still find it with some diligence.
Sites like LiveLeak and 4Chan are known for sometimes posting controversial content that is later
investigated and taken down, but even these sites are still considered the surface web, because they have
high traffic (and many people who can report illegal or questionable content) and because many of their
pages are searchable.
The deep web refers to sites and pages that have been intentionally blocked from any search engines
indexing the content. Also called the invisible or hidden web, these pages and sites are usually not
illegal or even dangerous; they’re simply not made to be searched. This could include anything fromnon-public entertainment, or perhaps private content that is password protected, such as online banking
records, web mail, pay on demand video or subscription magazines, and various medical or legal
documents that are considered non-public information.
It should be noted that there are two ways to avoid being indexed by a search engine; either manually
install code (or download an automatic plug-in) that blocks search robots from crawling the webpage, or
do not externally link the page (or site) to any other external pages.
Search engine bots find new sites by following a linking trail, page to page, onsite and offsite. Even a
website that doesn’t advertise and that avoids registering with any public search engine or domain name
can still be found via search if it is linked to other public pages.
Deep web pages cannot be searched and thus offer greater privacy than public surface websites.
However, these websites still answer to an authority since they are hosted by well known internet service
providers that offer private or shared server hosting. While they do have privacy, anything you see on
these pages is far from anonymous. There is a long paper trail. Deep web pages can limit access by
different entities. The simple explanation, and the reason why deep and dark are sometimes confused, is
that the majority of users only use the “surface web”—the most popular and heavily linked websites on
the Internet. That leaves potentially millions of privately owned and operated websites that no one has
officially classified or “indexed” (such as what Google and Bing do by crawling public websites), and
perhaps only a handful of people have actually seen.
Some individuals have likened the deep and dark web to trying to fish in the ocean, or perhaps an
iceberg that rises to the surface but actually goes thousands of leagues under the ocean. As far back as
2001, the deep web was said to be “orders of magnitude” larger than the surface web, and modern 2016
reports suggest that hidden websites and databases are 500+ times greater in number than everything we
see on the surface web.
Now theoretically, there are millions of websites that you would ordinarily never find, unless of course,
you were able to penetrate the deep and dark web.
What we see with our favorite sites (Facebook, YouTube, etc.) could be called the filtered media. You
will have access to only filtered information and news information, as well as entertainment pages,
which have been carefully selected for your viewing. This is a similar practice to how executives plan
network or cable television.
Search sites like Google and Bing may be able to link you to obscure pages that have very low
popularity, thanks to a longtail keyword search such as “Historical Records from the 1800s.” It may not
be a well known page but you can still find it with some diligence.
Sites like LiveLeak and 4Chan are known for sometimes posting controversial content that is later
investigated and taken down, but even these sites are still considered the surface web, because they have
high traffic (and many people who can report illegal or questionable content) and because many of their
pages are searchable.
The deep web refers to sites and pages that have been intentionally blocked from any search engines
indexing the content. Also called the invisible or hidden web, these pages and sites are usually not
illegal or even dangerous; they’re simply not made to be searched. This could include anything fromnon-public entertainment, or perhaps private content that is password protected, such as online banking
records, web mail, pay on demand video or subscription magazines, and various medical or legal
documents that are considered non-public information.
It should be noted that there are two ways to avoid being indexed by a search engine; either manually
install code (or download an automatic plug-in) that blocks search robots from crawling the webpage, or
do not externally link the page (or site) to any other external pages.
Search engine bots find new sites by following a linking trail, page to page, onsite and offsite. Even a
website that doesn’t advertise and that avoids registering with any public search engine or domain name
can still be found via search if it is linked to other public pages.
Deep web pages cannot be searched and thus offer greater privacy than public surface websites.
However, these websites still answer to an authority since they are hosted by well known internet service
providers that offer private or shared server hosting. While they do have privacy, anything you see on
these pages is far from anonymous. There is a long paper trail. Deep web pages can limit access by
these technologies:
• Registration and login required
• Unlinked pages
• Non-HTML content, such as multimedia with highly specific file formats
• Scripted content that is only accessible through JavaScript, Flash or other special scripting
• Dynamic content or contextual webpages, which are only returned through a specific form,
query or access context
• Search-prohibited code, such as Robots Exclusion programming, or even using CAPTCHAs to
disqualify searching bots
In simpler words, the deep web is still theoretically possible to surf, if inconvenient.
The Difference Between the Deep Web and Darknet
What differentiates the deep web from the “dark web” is that the dark web is hidden from the public
Internet and is not accessible at all without special software. You cannot accidentally stumble onto the
Dark Web. You must take deliberate steps to download the software.
Darknet refers to an overlay network with restricted access, one requiring the use of different
communications protocols and ports in configuration. There are various darknets you can use to access
the “Dark Web”, the otherwise impenetrable web you’re not supposed to find.You may have heard of file sharing software programs like Napster or LimeWire some years ago, and
these are examples of one “darknet” type; a peer to peer network of users that share files. The “Deep
Web” tends to encompass the darknet but the main difference is in the deliberate steps you must take to
connect to the darknet.
The idea of the darknet actually precedes the “Deep Web”, since the Darknet concept was devised in the
1970s, and referred to networks that were kept apart from ARPANET, which later became the
worldwide connected Internet. The Deep Web resulted because of the need for resistance to an all
searchable Internet. The Darknet concept simply evolved with the preference of privacy and more
exclusive traffic.
The main difference between Deep and Dark is simply that a darknet doesn’t have the classic
“searchability” or even the compatibility to be loaded by unsuspecting users. If you tried to load darknet
URLs provided by underground sources, you would not be able to load them simply by clicking the link
in Internet Explorer, Firefox, Safari, Chrome and so on. The darknet is sometimes considered a portion
of the Deep Web in technical terms, but in semantics, it is something far more guarded.
Later in this book, we’re going to review the best ways to connect to the Dark Web as well as which
ways are safest, which come with a risk and which programs are ideal for surfing anonymously.
Why the Surface Web is Extremely Limited
The question many people have when finding out that a “dark side of the Internet” exists, is why would
you want to search information that is considered “secret”?
The easy answer is that the surface web is extremely limited in terms of providing information, and
much in the same way as a TV network plans its schedule, there isn’t much “exploring” to do on the
surface web.
The surface web is highly guarded and filtered for public use. While some people from English-
speaking countries may think that Darknet software programs are only useful to those living in
communist countries, where the government controls Internet viewing, consider the facts about capitalist
or socialist countries.
Google and Bing, as well as other English search engines, have all been known to manipulate their
keyword search results for esoteric business reasons. Many believe that smaller websites are being
suppressed by larger search engines in favor of well known brand name business. Furthermore, many of
the news stories you see on the surface web are presenting one-sided information, as this stands to help
various commercial or governmental purposes.
Part of the excitement of exploring the Dark Web comes from the idea that you could be reading stories
or web pages that present a completely new perspective of life. Remember that in order to host a
website on the surface web, most of the content published must be considered generally “mainstream”.No hosting provider will allow a user to host content is considered too taboo, controversial or
“alternative” in viewpoint.
Knowing this, it comes as no surprise that many users search darknets for underground political news,
especially since many whistleblowing websites are now being censored from the surface web. Some
hackers dream of being the next “big whistleblower”, and imagine what information they might find by
searching “secret” files that no one is supposed to read.
Some users prefer using a darknet to protect their own privacy and escape what they consider “mass
surveillance” from invasive surface websites as well as the government.
Lastly, yes, the Darknet is sometimes used for illegal purposes, which might include hosting illegal
stores, committing hacking crimes, sharing illegally acquired multimedia (new music and movies), or
even distributing illegal pornography or scenes of real life murder. There is also much in the way of
counterfeit software, identity theft and spam operations.
Last but not least, some enjoy surfing the dark web simply for the joy of finding something new and
unexpected. Many sites on Darknet are harmless and somewhat trivial excursions into niche topics.
Some web hosting users simply don’t want to be searched or indexed and prefer anonymity. Others use
the Darknet market to discover hard-to-find books that would probably not be published on Amazon,
BN.com and other traditional stores.
BBS and Usenet Predecessors
Years ago in the 1980s and 1990s, Bulletin Board Systems were the predecessors of the online world.
These were computer servers running custom software allowing anyone with a PC to connect to the
system using a terminal. Once logged in, users were given access to a small community of users. Here,
they could chat with each other, play games, read news items and share content without any third party
company presiding over the process.
The BBS “Systems Operator” presided over the cyber-community using just a computer server, software
and a phone line that directly connected users to the BBS. Sometimes the “sysops” who owned the
BBS’s would list phone numbers to other BBS’s that you could call and connect to, for more exploring,
file trading and news sharing.
Usenet groups were the next hike in technology and were repositories for discussion groups, connecting
users from multiple locations and allowing file downloads and uploads and text discussions. Unlike the
BBS or the modern day web servers, Usenet technology did not require a central server or dedicated
administration. It existed via a forwarding news system that made use of multiple servers.
The deep and dark web essentially relive those old Usenet / BBS days, where users had access to an
unfiltered, unregulated and largely unknown online world free from filtering and corporate-sponsored
ads.Is Surfing Darknet Legal or Illegal?
In surfing the Dark Web, you may discover something new that no one has ever seen before. You may
find out something you never knew before or may catch a glimpse into an alternative lifestyle or
political/religious point of view that you never knew existed.
You may stumble upon some illegal content, unfortunately, but there are ways to avoid that as we will
discuss later. Lastly, you may and probably will discover many, MANY dead links. That’s all just part
of being an explorer into deep undiscovered territory.
The thrill of exploring the darkest parts of the web comes from venturing boldly into the unknown.
Perhaps it’s the closest we human beings can imagine to exploring space or the deepest part of the
ocean. Half the thrill is searching for something undiscovered or stumbling upon a truly unique or
mysterious page.
There is absolutely nothing illegal or unethical about exploring the darknet with a curious mind. You
have every right to download software for exploring these “locked sites” and must use discretion if and
when you do find something illegal. When in doubt, it’s probably best to leave the page behind. And
for more information on “what can go terribly wrong” be sure to check out our final chapter, which
shares some of the more outrageous but still possible horror stories of deep web exploration.
For now, let’s proceed onto the first chapter which discusses the only way to surf anonymously and
safely if you’re venturing into Darknet.
• Registration and login required
• Unlinked pages
• Non-HTML content, such as multimedia with highly specific file formats
• Scripted content that is only accessible through JavaScript, Flash or other special scripting
• Dynamic content or contextual webpages, which are only returned through a specific form,
query or access context
• Search-prohibited code, such as Robots Exclusion programming, or even using CAPTCHAs to
disqualify searching bots
In simpler words, the deep web is still theoretically possible to surf, if inconvenient.
The Difference Between the Deep Web and Darknet
What differentiates the deep web from the “dark web” is that the dark web is hidden from the public
Internet and is not accessible at all without special software. You cannot accidentally stumble onto the
Dark Web. You must take deliberate steps to download the software.
Darknet refers to an overlay network with restricted access, one requiring the use of different
communications protocols and ports in configuration. There are various darknets you can use to access
the “Dark Web”, the otherwise impenetrable web you’re not supposed to find.You may have heard of file sharing software programs like Napster or LimeWire some years ago, and
these are examples of one “darknet” type; a peer to peer network of users that share files. The “Deep
Web” tends to encompass the darknet but the main difference is in the deliberate steps you must take to
connect to the darknet.
The idea of the darknet actually precedes the “Deep Web”, since the Darknet concept was devised in the
1970s, and referred to networks that were kept apart from ARPANET, which later became the
worldwide connected Internet. The Deep Web resulted because of the need for resistance to an all
searchable Internet. The Darknet concept simply evolved with the preference of privacy and more
exclusive traffic.
The main difference between Deep and Dark is simply that a darknet doesn’t have the classic
“searchability” or even the compatibility to be loaded by unsuspecting users. If you tried to load darknet
URLs provided by underground sources, you would not be able to load them simply by clicking the link
in Internet Explorer, Firefox, Safari, Chrome and so on. The darknet is sometimes considered a portion
of the Deep Web in technical terms, but in semantics, it is something far more guarded.
Later in this book, we’re going to review the best ways to connect to the Dark Web as well as which
ways are safest, which come with a risk and which programs are ideal for surfing anonymously.
Why the Surface Web is Extremely Limited
The question many people have when finding out that a “dark side of the Internet” exists, is why would
you want to search information that is considered “secret”?
The easy answer is that the surface web is extremely limited in terms of providing information, and
much in the same way as a TV network plans its schedule, there isn’t much “exploring” to do on the
surface web.
The surface web is highly guarded and filtered for public use. While some people from English-
speaking countries may think that Darknet software programs are only useful to those living in
communist countries, where the government controls Internet viewing, consider the facts about capitalist
or socialist countries.
Google and Bing, as well as other English search engines, have all been known to manipulate their
keyword search results for esoteric business reasons. Many believe that smaller websites are being
suppressed by larger search engines in favor of well known brand name business. Furthermore, many of
the news stories you see on the surface web are presenting one-sided information, as this stands to help
various commercial or governmental purposes.
Part of the excitement of exploring the Dark Web comes from the idea that you could be reading stories
or web pages that present a completely new perspective of life. Remember that in order to host a
website on the surface web, most of the content published must be considered generally “mainstream”.No hosting provider will allow a user to host content is considered too taboo, controversial or
“alternative” in viewpoint.
Knowing this, it comes as no surprise that many users search darknets for underground political news,
especially since many whistleblowing websites are now being censored from the surface web. Some
hackers dream of being the next “big whistleblower”, and imagine what information they might find by
searching “secret” files that no one is supposed to read.
Some users prefer using a darknet to protect their own privacy and escape what they consider “mass
surveillance” from invasive surface websites as well as the government.
Lastly, yes, the Darknet is sometimes used for illegal purposes, which might include hosting illegal
stores, committing hacking crimes, sharing illegally acquired multimedia (new music and movies), or
even distributing illegal pornography or scenes of real life murder. There is also much in the way of
counterfeit software, identity theft and spam operations.
Last but not least, some enjoy surfing the dark web simply for the joy of finding something new and
unexpected. Many sites on Darknet are harmless and somewhat trivial excursions into niche topics.
Some web hosting users simply don’t want to be searched or indexed and prefer anonymity. Others use
the Darknet market to discover hard-to-find books that would probably not be published on Amazon,
BN.com and other traditional stores.
BBS and Usenet Predecessors
Years ago in the 1980s and 1990s, Bulletin Board Systems were the predecessors of the online world.
These were computer servers running custom software allowing anyone with a PC to connect to the
system using a terminal. Once logged in, users were given access to a small community of users. Here,
they could chat with each other, play games, read news items and share content without any third party
company presiding over the process.
The BBS “Systems Operator” presided over the cyber-community using just a computer server, software
and a phone line that directly connected users to the BBS. Sometimes the “sysops” who owned the
BBS’s would list phone numbers to other BBS’s that you could call and connect to, for more exploring,
file trading and news sharing.
Usenet groups were the next hike in technology and were repositories for discussion groups, connecting
users from multiple locations and allowing file downloads and uploads and text discussions. Unlike the
BBS or the modern day web servers, Usenet technology did not require a central server or dedicated
administration. It existed via a forwarding news system that made use of multiple servers.
The deep and dark web essentially relive those old Usenet / BBS days, where users had access to an
unfiltered, unregulated and largely unknown online world free from filtering and corporate-sponsored
ads.Is Surfing Darknet Legal or Illegal?
In surfing the Dark Web, you may discover something new that no one has ever seen before. You may
find out something you never knew before or may catch a glimpse into an alternative lifestyle or
political/religious point of view that you never knew existed.
You may stumble upon some illegal content, unfortunately, but there are ways to avoid that as we will
discuss later. Lastly, you may and probably will discover many, MANY dead links. That’s all just part
of being an explorer into deep undiscovered territory.
The thrill of exploring the darkest parts of the web comes from venturing boldly into the unknown.
Perhaps it’s the closest we human beings can imagine to exploring space or the deepest part of the
ocean. Half the thrill is searching for something undiscovered or stumbling upon a truly unique or
mysterious page.
There is absolutely nothing illegal or unethical about exploring the darknet with a curious mind. You
have every right to download software for exploring these “locked sites” and must use discretion if and
when you do find something illegal. When in doubt, it’s probably best to leave the page behind. And
for more information on “what can go terribly wrong” be sure to check out our final chapter, which
shares some of the more outrageous but still possible horror stories of deep web exploration.
For now, let’s proceed onto the first chapter which discusses the only way to surf anonymously and
safely if you’re venturing into Darknet.
Comments
Post a Comment