Comprehensive Study on Dark Net Finds Majority of Hidden Services Classified as Illicit [Update]

In a new and comprehensive study on the contents of TOR hidden services (.pdf) (so-called Dark Net), intelligence gathering company Intelliagg and »darknet indexing company« Darksum confirm earlier research findings by Daniel Moore and Thomas Rid.

The methodology of both studies is similar. Both used crawlers to retrieve pages, and both used manually-trained automated machine-learning systems for classification. The big difference lies in the size of the corpus of sites processed: While Moore and Rid ‘only’  acquired 5,615 individual *.onion-addresses, Intelliagg/Darksum identified 29,532 addresses of which only 46% were accessible.

In their recent paper on the subject (Moore, Daniel; Rid, Thomas. 2016: Cryptopolitik and the Darknet. Survival, 58:1, 7-38), Moore and Rid find that about 56% of hidden services on the TOR network could be classified as providers of illicit content. According to them, the most common use was related to drugs (423 hidden services) and finance (327 hidden services).

Classification of TOR hidden services (Moore/Rid 2016: 21).

Different forms of violence and arms related services are at the bottom of the table. The latter is not surprising since only recently a study (.pdf) by Geneva-based NGO Small Arms Survey discovered that Facebook is increasingly used as a marketplace for arms, especially in Libya.

The results suggest that the most common uses for websites on Tor hidden services are criminal, including drugs, illicit finance and pornography involving violence, children and animals.

Moore/Rid 2016: 21.

Categorisation of Hidden Services by Intelliagg/Darksum (Screenshot, p. 10 of report).

Intelliagg/Darksum now corroborate these earlier findings. Depending on how they classify, manual-only or automated, between 52% and 68% of contents of hidden services are illegal, which lies within the range of the 56% that Moore/Rid found.


Ironically, for the dark net to prosper it needs to become lighter.

Intelliagg/Darksum 2016, p. 11.

While both recognise legitimate uses of TOR and the Dark Net they also see the need to confront potential abuses of hidden services by criminals, which threaten reputation of TOR and trust in the Internet as a whole, as Moore/Rid (32) see it, Intelliagg/Darksum give a rather vague recommendation as to what to do with their findings: transparency of the Dark Net should be increased. How, they do not say.

Moore/Rid in turn are a bit more straight-forward: they recommend that

Darknets [...] should be fair game for the most aggressive intelligence and law-enforcement techniques, as well as for invasive academic research (2016: 32).

The difference about what to do might have its root in the different classifications used by the studies: While Intelliagg/Darksum see 57% of the uses of hidden services in provision of leaked data or file sharing, these use cases are not explicitly taken into account by Moore and Rid. 

Since TOR is predominantly, though admittedly not exceedingly so, used for illegal and ethically disturbing purposes, supporters and maintainers of the project should come up with a solution for liberal democratic states, and quickly, otherwise law enforcement and intelligence agencies will make that choice for them.

Update, 29.07.2016

Security researcher Sarah Jamie Lewis, project lead with »OnionScan«, already in April casted doubt on the analysis provided by companies Intelliag and Darksum. Lewis found that about 29% of Hidden Services in the Tor network are in fact duplicates of existing services. While Intelliag/Darksum make no mention of how they treat duplicate webpages, Moore/Rid concede in the technical annex (.pdf) to their paper that duplicate webpages within their data set are indeed possible.