eTraining for the Interpretation
of Man-made Scenes (eTRIMS)

Project Data

2006 - 2009
Supported by the European Community under grant IST 027113

University of Bonn, Institute for Photogrammetry - Wolfgang Förstner (coordinator)
University of Hamburg - Bernd Neumann
Czech Technical University in Prague, Center for Machine Perception - Radim Sára
Imperial College London, Department of Electrical and Electronic Engineering - Maria Petrou
HITeC, Hamburg Informatics Technology Center - Lothar Hotz

eTRIMS home page

The eTRIMS Team in Hamburg:

Bernd Neumann
Johannes Hartz
Arne Kreutzmann
Kasim Terzic
Department of Informatics, University of Hamburg
Lothar Hotz
HITeC Hamburg

eTRIMS Research in Hamburg

SCENIC Example
Scene interpretation using learnt concepts

The system SCENIC ("SCENe Interpretation by Configuration") has been extended in several ways to handle learnt concepts in low-level and high-level scene interpretation processes.

  • A middle layer has been developed which mediates between low-level image analysis and high-level interpretation. It maps low-level data to specific object views and allows high-level hypotheses to control low-level processing.
  • A standardised interpretation process has been defined which combines bottom-up part-whole reasoning with  goal-dependent top-down processing and can cope with incomplete evidence.
  • Learnt concepts can be integrated in the SCENIC knowledge base using the Semantic Web language OWL.
  • A GUI has been implemented to control and visualise the interpretation process.

Version Space Learning of spatial structures

Conceptual descriptions of aggregates play an essential role in model-based scene interpretation. An aggregate specifies a set of objects with certain properties and relations which together constitute a meaningful scene entity. Aggregate concepts for spatially related objects can be learnt from positive and negative examples using Version Space Learning, introduced by Mitchell. Our approach features a rich representation language encompassing quantitative and qualitative spatial attributes and relations. Using examples from the buildings domain, we have shown that aggregate concepts for window arrays, balconies and other structures can in fact be learnt from annotated images and successfully employed in the conceptual knowledge base of a scene interpretation system.

Hierarchy Example

Probabilistic models for compositional hierarchies

In the scene interpretation system SCENIC, high-level knowledge about visual scenes is currently represented by means of a logic-based knowledge representation language, using taxonomical and compositional hierarchies. We are developing a probabilistic framework which can be combined with our hierarchical knowledge structures. It will support probabilistic learning methods for high-level structures such as building facades, and provide probabilistic guidance for stepwise scene interpretation. By imposing intuitive abstraction properties on compositional hierarchies, evidence propagation during the interpretation process may become computationally feasible even in large knowledge bases.



Context-aware Classification

Appearance-based classification is a difficult task in many domains due to ambiguous evidence. Knowledge about the relationships between objects in the scene can help to resolve this problem. We have developed a new probabilistic classification framework based on the cooperation of decision trees and Bayesian Compositional Hierarchies, and have shown that introducing contextual knowledge in the form of dynamic priors significantly improves classification performance in the facade domain. For example, in storey images (left), balcony doors and windows could be much better distinguished once a probabilistic context was established.


Learning Probabilistic Structure Graphs

A novel domain-independent method for learning the probabilistic structure of 2D object configuration has been developed. The approach is graph-based and uses Maximum Common Subgraph Isomorphism to generalise a model graph over a set of training examples. A full probabilistic model, called a Probabilistic Structure Graph (PSG), is obtained by generalising the structural probabilities learnt from the examples. Models learnt from the eTRIMS image database have been used for classifying unknown object configurations, and an average classification rate of ca. 80% has been obtained. Additionally, bottom-up scene interpretation based solely on learnt models has been performed with promising results.

Terziç, K.; Hotz, L.; Sochman, J.: Interpreting Structures in Man-Made Scenes: Combining Low-Level and High-Level Structure Sources. In: Proc. International Conference on Agents and Artificial Intelligence (ICAART 2010), Valencia, January 2010   PDF
Terziç, K.; Neumann, B.: Integrating Context Priors  into a Decision Tree Classification Scheme. In: Proc. International Conference on Machine Vision, Image Processing, and Pattern Analysis (MVIPPA 2009), Bangkok, December 2009   PDF
Hartz, J.: Learning Probabilistic Structure Graphs for Classification and Detection of Object Structures. To appear in: IEEE Proceedings of the International Conference on Machine Learning and Applications, Miami (Florida, USA), December 2009   PDF
Terzic, K., Neumann, B.: Decision Trees for Probabilistic Top-down and Bottom-up Integration. Report FBI-HH-B-288/09, Department of Informatics, University of Hamburg, 2009   PDF
Kreutzmann, A.; Terziç, K.; Neumann, B.: Context-aware Classification for Incremental Scene Interpretation. In: Proc.  Workshop on Use of Context in Vision Processing (UCVP 2009), Boston, Nov 2009   PDF
Hartz, J.; Hotz, L.; Neumann, B.; Terzic, K.: Automatic Incremental Model Learning for Scene Interpretation. Proceedings of the International Conference on Computational Intelligence (IASTED CI-2009), Honolulu (Hawaii, USA), August 2009   PDF
Hotz, L.; Neumann, B.; Terzic, K.: High-level Expectations for Low-level Image Processing. Proc. KI-2008, Springer, 2008   PDF
Neumann, B.: Bayesian Compositional Hierarchies - A Probabilistic Structure for Scene Interpretation. Report FBI-HH-B-282/08, Department of Informatics, Hamburg University, 2008. Revised (Section 6) Dec. 2008   PDF
Möller, R.; Neumann, B.: Ontology-Based Reasoning Techniques for Multimedia Interpretation and Retrieval. In: Y. Kompatsiaris, P. Hobson (Eds.): Semantic Multimedia and Ontologies: Theory and Applications, Springer 2008, 55-98
PDF (final draft)
Hartz, J.; Neumann, B.: Learning a Knowledge Base of Ontological Concepts for High-Level Scene Interpretation. International Conference on Machine Learning and Applications, Cincinnati (Ohio, USA), December 2007   PDF
Terzic, K.; Hotz, L.; Neumann, B. : Division of Work During Behaviour Recognition - The SCENIC Approach. Workshop on Behaviour Monitoring and Interpretation, KI-2007  PDF
Hartz, J.; Neumann, B.: Version Space Learning of Spatial Structures for High-Level Scene Interpretation. TR FBI-B-277/07, Department of Informatics, University of Hamburg, 2007  PDF
Hotz, L.; Neumann, B.; Terzic, K.; Sochman, J.: Feedback between Low-Level and High-Level Image Processing. TR FBI-B-278/07, Department of Informatics, University of Hamburg, 2007  PDF


Prague Facade