The assessment of Roxanne Network’s effectiveness

Università Cattolica del Sacro Cuore is the leading partner taking care of developing the assessment framework and applying it to the criminal networks analyzed within the project, to investigate whether or not the results match the intended purpose.

To do so, the assessment exercise compares the “real network”, i.e. the actual interaction among individuals hereinafter called the Ground Truth Network, with two different networks:

• the “Roxanne Network”, which is derived from the application of the tools developed by Roxanne
• the “Telephone Network”, a “classic” network based only on the contacts among telephone numbers.

The general objective of the assessment is evaluating how well the Roxanne Network represents the Ground Truth Network, considering the classic Telephone Network as a benchmark (Figure below).

The assessment analyses three main elements:

• the similarity of networks compared to the real, Ground Truth network;
• the accuracy in the identification of the nodes (actors) compared to the Ground Truth;
• the accuracy in the identification of the edges (conversations) compared to the Ground Truth.

The data used for the systematic assessment are the Roxanne simulated data (ROXSD), and ongoing effort by the consortium to develop a detailed simulation of data collected by law enforcement agencies during investigations. The ROXSD dataset was entirely designed and developed by ROXANNE to emulate a criminal scenario of average complexity: the Prague anti-drug unit of the Czech police is simultaneously investigating three different cases that are believed to be connected. This investigation focuses in particular on two cases of international drug trafficking and one case on a drug laboratory. As the project progresses, the consortium keeps expanding the ROXSD data set to make it as realistic as possible. At the moment, Roxanne has released two versions of ROXSD (ROXSD0 and ROXSD1) while a third version (ROXSD2) is currently being finalized.  ROXSD0 original data comprise 100 simulated, recorded conversations among suspects on the criminal activities. In ROXSD1 the dataset was further expanded by adding non-criminally-relevant conversations among suspects and other individuals (e.g., private conversations with family) totaling over 230 calls. With the release of ROXSD2 the consortium will include additional conversations and other media (audio messages, videos, images).                                                                        

To assess the similarity to the Ground Truth network, researchers from Università Cattolica del Sacro Cuore considered a whole network approach that focused on the overall structure of the Roxanne and Telephone networks and compared it to the topology of the Ground Truth Network. The Figure below shows the graphs of the three networks in the ROXSD0 data set: from a visual inspection, it is possible to see that the Roxanne Network identifies one connected network, as in the Ground Truth Network, whereas the Telephone Network generates two separate components. Comparison metrics included information on connectivity, number of components, number of nodes, number of edges, clustering metrics and distance measures.

The results of the analysis are reported in the table below. We can observe that the Roxanne-improved networks outperform the traditional Telephone networks in regards to the overall topology and the connectivity of the structures. In other words, the general structure of the Roxanne networks is more similar to the Ground Truth networks than the Telephone networks.

These results suggest that the tools developed by Roxanne are providing a realistic and reliable reconstruction of the real connections among the individuals.

To assess the prediction accuracy of the nodes (i.e., the actors involved in the calls) and the edges (i.e., the conversations), we also examined whether the speakers identified by the Roxanne and Telephone networks matched the real speakers as observed in the Ground Truth network. For the Roxanne Network, the identification relies on speaker identification tools, whereas for the Telephone Network it relies only on the telephone numbers included in the scenario.

The results, summarized in the table below, report the percentage of correct predictions for any speaker, callers, receivers and conversations, respectively. The different approaches enable to examine the performance of the predictions in detail, e.g., by focusing on the caller, the receiver, or on both. The findings show that Roxanne networks outperform the Telephone networks in regard to almost all accuracy metrics, particularly when predicting caller and conversations. For example, for ROXSD version 1, the accuracy in the identification of the participants in a conversation in the Roxanne network (i.e., correctly identifying both caller and receiver) is 94% (i.e., the identification is correct for 94% of conversations), whereas a classic approach (just relying on the telephone numbers) only achieves an accuracy of 85.4%. Receiver predictions is almost identical when comparing Roxanne and telephone networks.

In summary, it can be said that the systematic assessment exercise on ROXSD data set shows a significant improvement in the understanding and analysis of criminal networks thanks to Roxanne’s technologies. In other words, the results demonstrate that the tools developed by Roxanne can more accurately reconstruct the topology and actor-level interactions of a real criminal network compared to a more classic approach relying exclusively on the telephone numbers of the suspects. The findings also indicate that Roxanne tools can reliably contribute to correctly identify the structure and characteristics of a criminal network and thus support intelligence and law enforcement agencies in their activities against criminal groups.

The assessment framework developed in this activity can be easily replicated on any other criminal network data, provided that it includes a ‘ground truth’ element. As soon as Roxanne analyses additional networks, the researchers at Università Cattolica will apply the assessment framework and replicate on them the same analysis.