ISCANIT: Recognising Intention in Real-Time
for Visually Mediated Interaction

The aim of the ISCANIT project is to undertake research supporting Visually Mediated Interaction (VMI) and to advance the state of the art by developing generic view-based head and body behavioural models. These models will be used to recognise intentions for active camera control. We propose to develop active computer vision systems which perform dynamic scene interpretation in terms of subjects' behaviour and intention to mediate interaction. A prototype system will be built to perform real-time tracking of multiple people and behavioural analysis of several individuals (at most three simultaneously) within typical indoor office or home environments. The main objectives for evaluation of the project are, therefore:

1. The investigation and development of a computational paradigm based upon active cameras that will enable on-line capture and interpretation of dynamic scenes containing live actions, which can also be used to capture the behaviour of users.

2. The development of body movement and gesture models and the use of such models for user-specific interpretation of behaviour and intention.

3. The development of intentional tracking techniques that exploit the behavioural models above by estimating and predicting expected head and body movement and primitive gestures in real-time.

4. The development of switching of attention mechanisms for active tracking of several individuals with fewer cameras than individuals.

Resources

An overview of Visually-Mediated Interaction research and projects.

A Postscript version of the ISCANIT Project Final Report.

People Involved in the ISCANIT Project

This EPSRC-funded 2-year project is a collaboration between the School of Cognitive and Computing Sciences at the University of Sussex (COGS) and the Department of Computer Science, Queen Mary and Westfield College, University of London.

University of Sussex

Hilary Buxton (PI)
Jonathan Howell

QMW

Shaogang Gong (PI)
Jamie Sherrah

Associates:

University of Dundee

Stephen McKenna

Publications arising out of the ISCANIT Project

2000

A. J Howell and H. Buxton.S. RBF Network Methods for Face Detection and Attentional Frames. Submitted to Neural Processing Letters. 2000.
J. Sherrah, S. Gong, A. J Howell and H. Buxton `Interpretation of Group Behaviour in Visually Mediated Interaction', in Proc. 15th International Conference on Pattern Recognition (ICPR'2000), Barcelona, Spain, September 2000.

1999

J. Sherrah and S. Gong. `Fusion of 2D face alignment and 3D head pose estimation for robust and real-time performance', Proc. IEEE International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems (RATFG-RTS '99), pp.24-31, Corfu, Greece, September 1999
J. Sherrah and S. Gong. Fusion of perceptual cues using covariance estimation. In Proc. British Machine Vision Conference, Nottingham, September 1999.
J. Sherrah, S. Gong and E-J. Ong. Understanding pose discrimination in similarity space. In Proc. British Machine Vision Conference, Nottingham, September 1999.
J. Sherrah and S. Gong. Exploiting context in gesture recognition. In Proc. 2nd International Interdisciplinary Conference on Modelling and Using Context, Trento, Italy, 9-11 September 1999.
S. McKenna, Y. Raja and S. Gong. Tracking colour objects using adaptive mixture models. Image and Vision Computing, Vol. 17, pp. 225-231, 1999.
A. J Howell and H. Buxton. `Gesture Recognition for Visually Mediated Interaction', Proc. of GW'99 - The 3rd Gesture Workshop, Gif-sur-Yvette, France, March 1999.

1998

A. J Howell and H. Buxton. `Learning Gestures for Visually Mediated Interaction', Proc. British Machine Vision Conference (BMVC'98), pp.508-517, Southampton, UK, 1998.
S. Gong, Eng-Jon Ong and S. McKenna. Learning to associate faces across views in vector space of similarities to prototypes. In Proc. British Machine Vision Conference, Southampton, England, September 1998.
S. McKenna and S. Gong. Gesture recognition for visually mediated interaction using probabilistic event trajectories. In Proc. British Machine Vision Conference, Southampton, England, September 1998.
A. J Howell and H. Buxton. `Learning Gestures with Time-Delay RBF Networks', Proc. International Conference on Artificial Neural Networks (ICANN'98), pp.239-244, Skövde, Sweden, 1998.
A. J Howell and H. Buxton. Buxton, H. (1998) `Recognising People and Behaviours', in Wechsler, H. and Philips, P.J. and Bruce, V. and Fogelman-Soulié, F. and Huang, T.S. (Eds.) Face Recognition: From Theory to Applications, NATO ASI Series F, pp.480-491, Springer-Verlag, 1998.
S. McKenna and S. Gong. Recognising moving faces. In Wechsler, Philips, Bruce, Fogelman-Soulie, and Huang (Eds.) Face Recognition: From Theory to Applications, NATO ASI Series F, Springer-Verlag, July 1998.
S. Gong. Understanding head pose and body gestures for visually mediated interaction. In IEE Colloquium on Computer Vision for Virtual Human Modelling. Savoy Place, London, 9 July 1998.
S. McKenna, S. Gong and Y. Raja. Modelling facial colour and identity with Gaussian mixtures. Pattern Recognition. Vol. 31, No. 12, pp. 1883-1892, 1998.
Y. Raja, S. McKenna and S. Gong. Using colour for robust object tracking and segmentation. In Proc. European Workshop on Nonlinear Model Based Image Analysis, Glasgow, Scotland, 1-3 July 1998.
A. J Howell and H. Buxton. `Towards Visually Mediated Interaction using Appearance-Based Models', ECCV'98 Workshop on the Perception of Human Action, Freiburg, Germany, June 1998.
E-J. Ong, S. McKenna and S. Gong. Tracking head pose for inferring intention. In Proc. European Workshop on Perception of Human Action, Freiburg, Germany, 6 June 1998.
Y. Raja, S. McKenna and S. Gong. Colour model selection and adaptation in dynamic scenes. In Proc. European Conference on Computer Vision, Freiburg, Germany, 2-6 June 1998.
S. Duvdevani-Bar, S. Edelman, A. J Howell and H. Buxton. `A Similarity-Based Method for the Generalization of Face Recognition over Pose and Expression', Proc. 3rd International Conference on Automatic Face & Gesture Recognition (FG'98), pp.118-123, Nara. Japan, 1998.
Y. Raja, S. McKenna and S. Gong. Tracking and segmenting people in varying lighting conditions using colour. In Proc. IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, 14-16 April 1998.
A. J Howell and H. Buxton. `Learning Identity and Behaviour with Neural Networks', Proc. 3rd Asian Conference on Computer Vision (ACCV'98), Vol. I, pp.523-530, Hong Kong, 1998.
S. McKenna and S. Gong. Face recognition from sequences using models of identity. In Proc. Asian Conference on Computer Vision, Hong Kong, 8-11 January 1998.
S. McKenna, Y. Raja and S. Gong. Object tracking using adaptive colour mixture models. In Proc. Asian Conference on Computer Vision, Hong Kong, 8-11 January 1998.
Y. Raja, S. McKenna and S. Gong. Segmentation and tracking using colour mixture models. In Proc. Asian Conference on Computer Vision, Hong Kong, 8-11 January 1998.
S. Gong and S. McKenna. Real-time face pose estimation. International Journal on Real Time Imaging, Special Issue on Real-time Visual Monitoring and Inspection. Vol. 4, pp. 333-347, 1998.