Content based Image Retrieval (at SVCL) Nikhil Rasiwasia, Nuno Vasconcelos Stati

Content based Image Retrieval (at SVCL) Nikhil Rasiwasia, Nuno Vasconcelos Stati www.phwiki.com

Content based Image Retrieval (at SVCL) Nikhil Rasiwasia, Nuno Vasconcelos Stati

Parker, Jane, Lifestyle Editor has reference to this Academic Journal, PHwiki organized this Journal Content based Image Retrieval (at SVCL) Nikhil Rasiwasia, Nuno Vasconcelos Statistical Visual Computing Laboratory University of Cali as long as nia, San Diego ECE271A – Fall 2007 Why image retrieval Help in finding you the images you want. Source: http://www.bspcn.com/2007/11/02/25-photographs-taken-at-the-exact-right-time/ But there is Google right Metadata based retrieval systems text, click-rates, etc. Google Images Clearly not sufficient what if computers understood images Content based image retrieval (early 90’s) search based on the image content Top 12 retrieval results as long as the query ‘Mountain’ Metadata based retrieval systems

Lappeenranta University of Technology FI www.phwiki.com

This Particular University is Related to this Particular Journal

Early underst in addition to ing of images. Query by Visual Example(QBVE) user provides query image system extracts image features (texture, color, shape) returns nearest neighbors using suitable similarity measure Texture similarity Color similarity Shape similarity This is a graduate class, so! Details Bag of features representation No spatial in as long as mation () Yet per as long as ms good () Each Feature represented by DCT coefficients Other people use SIFT, Gabor filters etc dct( ) + Image representation Bag of DCT vectors GMM

Query by visual example Query Image C in addition to idate Images Probability under various models Ranking p1 > p2 . > pn Query by visual example (QBVE) What can go wrong

This can go wrong! visual similarity does not always correlate with “semantic” similarity Both have visually dissimilar sky Disagreement of the semantic notions of train with the visual notions of arch. Intelligent Researchers (like u) Semantic Retrieval (SR) User provided a query text (keywords) find images that contains the associated semantic concept. around the year 2000, model semantic classes, learn to annotate images Provides higher level of abstraction, in addition to supports natural language queries abc Semantic Class Modeling Bag of DCT vectors GMM wi = mountain mountain Semantic Class Model Efficient Hierarchical Estimation “Formulating Semantics Image Annotation as a Supervised Learning Problem” [G. Carneiro, IEEE Trans. PAMI, 2007]

Semantic Retrieval Query Image C in addition to idate Words Probability under various models Ranking p1 > p2 . > pn Mountian Sky Sexy Girl so on house

First Five Ranked Results Query: mountain Query: pool Query: tiger First Five Ranked Results Query: horses Query: plants Query: blooms First Five Ranked Results Query: clouds Query: field Query: flowers

First Five Ranked Results Query: jet Query: leaf Query: sea But: Semantic Retrieval (SR) Problem of lexical ambiguity multiple meaning of the same word Anchor – TV anchor or as long as Ship Bank – Financial Institution or River bank Multiple semantic interpretations of an image Boating or Fishing or People Limited by Vocabulary size What if the system was not trained as long as ‘Fishing’ In other words, it is outside the space of trained semantic concepts Lake Fishing Boating People Fishing! what if not in the vocabulary abc In Summary SR Higher level of abstraction Better generalization inside the space of trained semantic concepts But problem of Lexical ambiguity Multiple semantic interpretations Vocabulary size QBVE is unrestricted by language. Better Generalization outside the space of trained semantic concepts a query image of ‘Fishing’ would retrieve visually similar images. But weakly correlated with human notion of similarity VS abc Fishing! what if not in the vocabulary Lake Fishing Boating People The two systems in many respects are complementary!

Query by Semantic Example (QBSE) Suggests an alternate query by example paradigm. The user provides an image. The image is mapped to vector of weights of all the semantic concepts in the vocabulary, using a semantic labeling system. Can be thought as an projection to an abstract space, called as the semantic space To retrieve an image, this weight vector is matched to database, using a suitable similarity function Lake Water People Sky Boat .2 .3 .2 .1 Semantic Space Mapping to an abstract space of semantic concepts Query by Semantic Example (QBSE) As an extension of SR Query specification not as set of few words. But a vector of weights of all the semantic concept in the vocabulary. Can eliminat Problem of lexical ambiguity- Bank+’more’ Multiple semantic interpretation– Boating, People Outside the ‘semantic space’ – Fishing. As an enrichment of QBVE The query is still by an example paradigm. But feature space is Semantic. A mapping of the image to an abstract space. Similarity measure at a higher level of abstraction.1 .2 .1 .3 .2 Lake Water People Boating Boat 0 .5 0 .5 0 (SR) query: water, boating = ¹ (QBVE) query: image Lake Water People Boating Boat Semantic Space Boating Water Lake QBSE System Concept 1 Query Image Any Semantic Labeling System Concept 2 Concept 3 Concept L Database Weight Vector 1 Weight Vector 2 Weight Vector 3 Weight Vector 4 Weight Vector 5 Weight Vector N Suitable Similarity Measure Ranked Retrieval Posterior probability Weight Vector

Parker, Jane Tallassee Tribune Lifestyle Editor www.phwiki.com

QBSE System Concept 1 Query Image Any Semantic Labeling System Concept 2 Concept 3 Concept L Database Weight Vector 1 Weight Vector 2 Weight Vector 3 Weight Vector 4 Weight Vector 5 Weight Vector N Suitable Similarity Measure Ranked Retrieval Posterior probability Weight Vector Semantic Class Modeling Bag of DCT vectors Gaussian Mixture Model wi = mountain mountain Semantic Class Model Efficient Hierarchical Estimation “Formulating Semantics Image Annotation as a Supervised Learning Problem” [G. Carneiro, CVPR 2005] QBSE System Concept 1 Query Image Any Semantic Labeling System Concept 2 Concept 3 Concept L Database Weight Vector 1 Weight Vector 2 Weight Vector 3 Weight Vector 4 Weight Vector 5 Weight Vector N Suitable Similarity Measure Ranked Retrieval Posterior probability Weight Vector

Semantic Multinomial Posterior Probabilities under series of L independent class models Semantic Multinomial QBSE System Concept 1 Query Image Any Semantic Labeling System Concept 2 Concept 3 Concept L Database Weight Vector 1 Weight Vector 2 Weight Vector 3 Weight Vector 4 Weight Vector 5 Weight Vector N Suitable Similarity Measure Ranked Retrieval Posterior probability Weight Vector

Content based image retrieval Query by Visual Example (QBVE) Color, Shape, Texture, Spatial Layout. Image is represented as multidimensional feature vector Suitable similarity measure Semantic Retrieval (SR) Given keyword w, find images that contains the associated semantic concept. abc

Parker, Jane Lifestyle Editor

Parker, Jane is from United States and they belong to Tallassee Tribune and they are from  Tallassee, United States got related to this Particular Journal. and Parker, Jane deal with the subjects like Calendar News; Features/Lifestyle; Marriage/Weddings

Journal Ratings by Lappeenranta University of Technology

This Particular Journal got reviewed and rated by Lappeenranta University of Technology and short form of this particular Institution is FI and gave this Journal an Excellent Rating.