This is an anonymous question sent in by a member who does not want their name disclosed. Please be thoughtful with your responses, as these are usually tough questions to ask and answer.
I know a lot of bad things happen on the dark web, and that the deep web is different in that it might not be suspicious activity (i.e. legit banking activity) -- but how do these sites work?
Do they just throw on this meta tag to 'disappear'?
<META NAME="robots" CONTENT="noindex,nofollow">
I imagine there's more to it..
Anyway, just curious! Not trying to do anything evil š
Top comments (9)
First rule of Dark Web.
I am Weston's inflamed sense of rejection.
š
I'm really curious about this, as well. Occasionally a friend will ask me a question on the subject and I'll mutter something about TOR and then just sheepishly admit I have no idea what I'm talking about š.
You are correct, the deep and dark web are two very different things and the nature of them doesn't help the confusion, and could probably use their own articles but I'll do my best here.
Deep Web
The deep web is any part of the "public" internet that isn't public. The exact definition is a bit murky and most people end up defining it in levels but the simple answer is any part of a website that isn't indexed by search engines. This could be through
robots.txt,<meta name="robots">, or any manner of server side filtering, including but not limited to, requiring a sign in, a pay wall, or dynamic content. As far as raw data, the reason you see numbers like the deep web is 90% of the web is mostly because the amount of unique search results page from google, walmart, etc, or your email inbox, or your netflix queue, will always vastly outnumber the amount of "real" page that google and other search engines actually collect.Dark Web
The dark web is an entirely different concept. The dark web specifically refers to any service only accessible through The Onion Router (aka TOR). TOR is special browser built on top of the TOR protocol which was originally developed by the US Navy for advanced anonymity online. Its achieves this by connecting to 3 or more peers apart from the server you're actually trying to connect to and encrypting your traffic before and after each node hop in a way that each node only knows where it came from and where its going. While TOR has been used illicitly in the past for selling drugs and credit card info and supposedly even hit men, it's also used by whistle blowers, people under tyrannical governments and everyday people who want more anonymity online.
I think for the deep web I'd consider any page that a major search engine wouldn't provide unless I already knew the page I was looking for. That is, some pages/sites are known, but will just never appear in search results, unless the person includes a matching title and page name.
The dark web isn't restricting to just TOR. There are other protocols, such as Freenet or I2P. By the nature of the dark web we can also imagine any number of custom protocols used by a limited audience.
Curiously, the dark web need not be part of the deep web -- a search engine could crawl and index things on the TOR network, for example.
On your deep web part, a page like that is one that is most likely excluded from a
robots.txtOn your dark web part, sorry about that part. You're right about the other networks, totally skipped my mind when I was writing the comment. Their obscurity make them even more dark it seems.
I look on the deep web and the dark web just like how astrophysicists talk about dark matter. Dark matter isn't sinister or evil just because of the name, neither are the deep or dark webs inherently evil. There are evil people everywhere using the public web, the deep web, and the dark web. My entire business is on the deep web and of course it is entirely on the up-and-up, a positive and useful thing, open to the public providing your know the URL. It just makes sense to keep Google and Bing out of it.
This small video will help you understand TOR easily...