Mindsheet Threat Detection Algorithms

Computer assisted threat assessment was weak in many, if not all, all of the MoD Grand Challenge entries. Most of the systems relied heavily on human intervention. In an operational application, this manual approach:

• fails when communications with the unmanned vehicle fails;
• fails when the operator is overloaded, or, distracted by other events;
• fails when the operator loses concentration after a prolonged period of time;
• overloads the operator at the base station with vast amounts of incoming information.

However, a task, that is so deceptively effortless for alert people, has dogged computer vision researchers for forty years. The problems are immense:

• six degrees of viewpoint variance on the threat object;
• object occlusions;
• object shape and colour variances;
• elastic objects with no fixed shape;
• sensor system noise;
• environmental variations (fog, rain, mist, …).

A straight forward correlation system will quickly get bogged down with a combinatorial explosion of threat correlations. Worse still, most threats will not have been seen before and yet humans effortlessly generalise and correctly classify these unknown threats. In many circumstances, humans can even sense threats, merely by the absence of normality.

And yet, analysis of video imagery could bring many benefits to the Mindsheet surveillance robot platform (Testudo):

• Detection of suspicious behaviour eg. Peculiar manoeuvring pattern of bomb car during the collision with Glasgow airport
• Greater stand-off range as explosive molecules do not have to be present in the detector
• Lower cost sensors which are already present in the form of CCTV cameras
• Detection of background anomalies that indicate an impending threat
• Detection of changes, eg. Bag or Case left behind
• Wide area coverage

Mindsheet have created threat detection algorithms which are based on the recent work of Jeff Hawkins: hierarchical temporal memory (“HTM”) systems.  This new class of algorithm is biologically inspired by the workings of the neo-cortex.  The conceptual framework for this approach is laid out in Jeff Hawkins’ paper: Hierarchical Temporal Memory Systems.

HTMs are organised as a tree-shaped hierarchy of nodes, where each node implements a common learning and memory function. They are ideally suited to spatial and temporal pattern recognition tasks and the algorithm efficiently exploits the fact that features that are close together in space and time are more likely to be more correlated than features that are not.

However, Mindsheet’s original implementation discovered many hurdles with this approach that include:

• restricted input to binary and 2D patterns;
• very slow training to allow generalisation to all viewpoint variances;
• no generalisation to different scales and translations;
• no clear guidance for setting the right hierarchical topology.

Mindsheet has developed its own version of these recognition algorithms that overcome some of these problems. The new techniques now include:

• a holographic property that repeats all learnt vectors at every node in the hierarchy for rapid translational and scale generalisation (a variant of object oriented inheritance);
• faster training by automatically generating rotational variance vector sets;
• grey scale input by enabling Euclidean distance measurement to learnt grey scale vectors .

The figure below shows preliminary results from this approach, and shows promise. In particular, the initial training very interestingly allowed the subtle discrimination between look-out behaviour and normal civilian behaviour.

Mindsheet proposes to take these algorithms to the next level by collecting training and test sets derived from CCTV feeds and the data that Mindsheet gathered at the MoD Grand Challenge finale. Mindsheet will then evaluate and refine the algorithms for use in the interpretation of images from static sensors, air and ground vehicles.

• Investigation of the development of a fractal index map of an image, where the inverse of the fractal index of an image is displayed, in false colour, as an overlay on the original image. Man-made objects, such as IEDs, tend to have a low fractal index (low image complexity), which would be displayed as red in an image. This is a promising technology for the display of man-made objects that are partially camouflaged.
• Difference detection software, to highlight changes that have occurred over time. Any changes detected could be forwarded for more in-depth investigation by the threat detection software, or, by an operator.
• Automatic detection of circles and ellipses in an image. These objects are frequently associated with man-made objects, such as weapons and IEDs.