DARPA shows off its tech for indexing the deep web

On Sunday night, 60 Minutes aired a segment about the Defense Advanced Research Projects Agency, or DARPA, and its attempts to secure the internet from hackers, human traffickers and other criminals. One of the DARPA efforts the program highlighted — and did so even more in an unaired segment for the web — is a project called Memex, which is essentially a search engine for the deep web and the dark web.

The technology looks pretty amazing in a number of ways, including its scale, its speed and its interface. Of course, it’s also tackling a horrible and often under-appreciated problem, which is the illegal trafficking of women and girls as sex objects. Asked why DARPA is concerned with sex trafficking, Memex inventor Chris White explained that people willing to take part in that endeavor are often more likely to take part in other endeavors — including things like weapons or drug trafficking — that could have national security implications.

A Memex-generated map of sex trafficking.
A Memex-generated map of sex trafficking.

I wrote briefly about Memex last month, as part of a post about DARPA-funded research into machine learning algorithms — including computer vision and text analysis algorithms — for extracting even more info from deep web content.

The work DARPA is doing is part of a larger effort, which also includes tech companies like Google and Palantir, to identify and map instances of human trafficking around the world. It’s one of many problems that has existed for a long time, but that the internet has made easier to engage in. However, these efforts and others also show how the internet is making it easier for law-enforcement agencies to track and prosecute these crimes, provided the right analytical techniques are in place.

The 60 Minutes segment also featured DARPA innovation head Dan Kaufman, who spoke about web security at our Structure conference last June.