Multimedia Data Mining an Overview to Image Processing and Machine Learning by Zaheer Ahmad

Multimedia Data Mining: An Overview to Image Processing and Machine Learning Zaheer Ahmad

PhD Scholar [email protected]

Department of Computer Science University of Peshawar Peshawar 2/16/2011

1

Agenda •

Multimedia Data Mining

•

Image Data Mining and Image Processing

•

Machine Learning

•

Learning Techniques and tools

•

Neural Networks Networks and its types

•

Training (Learning) of Neural Network

2/16/2011

2

Multimedia Data mining •

•

Multimedia Data Mining is i s an interdisciplinary and multidisciplinary field, used to intelligently intelligently retrieve retrieve and search multimedia contents. A variety of techniques, from machine learning, statistics, statistics, databases, knowledge acquisition, data visualization, image analysis, an alysis, high performance computing, and knowledgebased systems are used in MMM

2/16/2011

3

2/16/2011

4

MACHINE LEARNING

2/16/2011

5

Data for MMM Data a database ? No ----- mostly Web Image, Audio, Video Live Streaming Geo Sensors data But yes…. video database Image or audio database d atabase

• • • • • • •

2/16/2011

6

•

•

The word multimedia refers to a combination of multiple media types together Multimedia Data Type  –

 –

2/16/2011

Any Type of information medium that can be represented, processed, stored and transmitted over network in digital form Multi-lingual text, numeric, images, videos, audio, graphical, temporal, relational and categorical categorical data 7

Definition •

MMM is a subfield of data mining that deals with an extraction of implicit knowledge, multimedia data relashionships, or other patterns patterns not explicitly stored stored in multimedia databases  –

2/16/2011

Used for multimedia information system and retrieval retrieval of content based image/audio/video and provide search and efficient storage organization

8

Media Types •

•

•

•

0-dimensional data: This type of the data is the regular, regular, alphanumeric data. A typical example is the text data. 1-dimensional data: This type of the data has one dimension of a space imposed into them. A typical example of this type of the data is the audio data 2-dimensional data: This type of the data has two dimensions of a space imposed into them. Imagery data and graphics data are the two common examples of this type of data 3-dimensional data: This type of the data has three dimensions of a space imposed into them. Video data and animation data are the two common examples of this type of  data

2/16/2011

9

Multimeimedia Data •

Spatial Data  –

•

Image Data  –

•

Generalize detailed geographic points into clusterd regions, such as business, residential, industrial, or agricultural areas, according to land usage Size, color, shape, texture, orientation, and relative postions and structure of the contained objects or regions in the image

Music data  –

 –

2/16/2011

Summarize its melody: based on the approximate pattern pattern that repeateldly occure in the segment Summarized its type: based on its tone, tempo, or the major musical insturment played 10

How Multimedia Data Mining System Works

2/16/2011

11

Similarity Search in Multimedia data •

Description based retrieval systems  –

 –

 –

•

•

Build indices and perform object retrieval based on image descriptions, such as keywords, captions, size and time of creation Labor-intensive if performed manually Results are typically of poor quality if automated

Content Based Retrieval Systems Support retrieval based on the image content, such as color, histogram, texture, shape, objects and wavelet transforms

2/16/2011

12

Multidimensional Analysis of  Multimedia Data •

Multimedia data Cube  –

 –

•

Design and construct similar to that traditional data cubes from relational data Contain additional dimensions and measures for multimedia information such as color, texture, and shape

The database doesn’t store images but their descriptors  –

Feature Descriptor: a set of vectors for each visual characteristics • • •

 –

2/16/2011

Color Vector: contains the color histogram MFC(Most Frequent Color) Vector: Vector: Five color centroids MFO(Most Frequent Orientation) Vector: Five edge orientation centroid

Layout Descriptor: Contains a color layout vector and an edge layout vector

13

Typical Architecture of MMM

2/16/2011

14

Image Data Mining Image and Machine Learning

2/16/2011

15

What is an image? •

An image is a two dimensional function, f(x,y), where x and y are spatial coordinates, coordinates, and the amplitude of f at any pair of  coordinates coordinates (x,y) is called the intensity or grey level of the image at that point.

Image Processing Stages Image Acquisition

Image Processing

Analog to digital conversion 

Remove noise, improve contrast …

Image Segmentation

Find regions (objects) in the image 

Image Analysis

Take measurements of  objects/relationships 

Pattern Recognition

Match the description with  similar description of known  objects (models) 17

Image Analysis Image Analysis Input Image Regions, objects

Measurements

Measurements: -Size -Position -Orientation -Spatial relationship -Gray scale or color intensity 19

Image segmentation The operation of distinguishing important objects from the background (or from unimportant objects) object s) based on different feature feature of the image

Area B

Dark objects, bright background

Area A

Image Segmentation Segmentation Regions Objects

Input Image

-Clasify pixels into into groups having similar characteristics -Two -Two techniques: techniq ues:

Region segmentation segmentat ion

Color/smoothness

—

Edge detection

21

Region Detection

2/16/2011

22

Histogram The data contained in a digital image can be displayed as a histogram histogram which is a plot of the pixel values ranging from black to white versus the number of  pixels that have that particular value.

Edge through Gradient Information Neighborhood pixels Sharpness Change / Contrast change

Edge Location

( xi , yi )

Edge Direction

 i

Patt Pa ttern ern Recognition (PR) Pattern Recognition - Measurements - Stuctural descriptions

Class identifier

feature vector  set of information data 

25

Content Based Image Retrieval

26

Fingerprint recognition system Enrollment  Fingerprint sensor

Feature Extractor

Template database

Identification  Fingerprint sensor

Feature Extractor Feature Matcher

ID 27

Machine Learning A computer program is said to learn from experience ‘E’ with respect to some class of  tasks ‘ T ’ and performance measure ‘P’, If its performance at tasks in T, as measured by P, improves improves with experience E.

Mitchell (1997): 2/16/2011

28

Machine Learning Things learn when they change their behavior in a way that makes them perform better in the future.

From Witten and Frank (2000)

2/16/2011

29

Machine Learning •

•

2/16/2011

ML is a scientific discipline that is concerned with the design and development of algorithms that allow computers to evolve behaviors based on empirical data, such as from sensor data or databases. A major focus of machine learning l earning research is to automatically learn to recognize complex patterns and make intelligent decisions based on data.

30

•

•

the difficulty lies in the fact that the set of all possible behaviors given given all possible inputs inp uts is too large to be covered by the set of observed examples (training data). Hence the learner must generalize from the given examples, so as to be able to produce a useful output in new cases

2/16/2011

31

Types of Learning •

•

•

Supervised Learning Learning a mapping between an input x and a desired output y Unsupervised Learning Understanding the relationships between data components Reinforcement Reinforcement Learning Learning to act in the environment environment based on the delayed rewards

2/16/2011

32

Classes of Learning Machine learning is not only about classification. classification. Classification learning: learn to put instances into pre-defined classes-----competitive network: selects one unit in the output layer layer (target class)--(Supervised Learning) Learning) Association learning: learn relationships between the Attributes------ new response becomes associated with a particular stimulus ---pattern associator: recalls input patterns based on similarity Clustering: discover classes of instances that belong Together------- (Unsupervised (Unsupervised))self-organizing map (SOMs) 2/16/2011

33

Learning Tools and Techniques in Short

2/16/2011

34

Learning Rules •

•

•

•

•

if outlook = sunny sunny and humidity = high then play = no if outlook = rainy and windy = true then play = no if outlook = overcast then play = yes if humidity = normal then play = yes if none of the above then play = yes BEST But LABOURUS , HARD TO CODE AND COVER in Large Domains

2/16/2011

35

Learning Decision Trees •

Example: XOR (familiar from connectionist networks).

Nodes represent decisions on attributes, leaves represent classifications .

Some how like Learning Rules 2/16/2011

36

Principal component analysis •

•

•

PCA is applied as a data reduction reduction or structure detection method combining two correlated variables into one factor PCA defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by any projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate

2/16/2011

37

Support Vector Machine •

•

•

•

Support Vector Machine is a classifier derived from statistic statistical al learning theory by Vladimir Vladim ir Vapnik and his co-workers Used for large data set Good for text classification Work as multilayer perceptron

2/16/2011

38

Hidden Markov Model

2/16/2011

39

Genetic Algorithms

2/16/2011

40

Neural Networks

41

NN A Brain-Inspired Model

Inputs

Outputs

Connection between cells

out in

42

Physical Structure of biological neuron •

•

•

•

•

Nerve cells are main processing element in our central nervous system. Humans generally have about 100 billion nerve cells in the entire nervous system. system. Axon and dandroid are signal carrier away and toward cell body respectively Synapse is the point at which the axon of one cell interconnects interconnects with a dendrite of another cell cel l A basic nerve cell is thought as a black box box

2/16/2011

43

NN A Brain-Inspired Model •

•

•

A neural network acquires knowledge through learning. A neural network's knowledge is stored within inter-neuron inter-neuron connection strengths strengths known as synaptic weights.

The largest modern neural networks achieve the complexity comparable to a nervous system of a fly. 44

Historical Background •

•

•

•

•

•

1943 McCulloch and Pitts proposed the first computational models of neuron. 1949 Hebb proposed the first learning rule. 1958 Rosenblatt’s work in perceptrons. 1969 Minsky and Papert’s exposed limitation of the theory. 1970s Decade of dormancy for neural networks. 1980-90s Neural network return (self-organization, back-propagation back-propagation algorithms, etc)

45

NN Applica Applications tions •

Process Modeling and Control- Creating a neural network model for a physical plant then using that model to determine the best control settings for the plant.

•

Machine Diagnosis- Detect when a machine has failed so that the system can automatically shut down the machine when this occurs.

•

Target Recognition Reco gnition- Military application which uses video and/or infrared image data to determine if an enemy target is present.

•

Medical Diagnosis- Assisting doctors with their diagnosis by analyzing the reported symptoms and/or image data such as MRIs or X-rays.

•

Target Marketing- Finding the set of demographics which have the highest response rate for a particular marketing campaign.

• •

Voice Recogntion- Transcribing spoken words into ASCII text. Financial Forecas Forecasting ting( Stock predication) - Using the historical data of a security to predict the future movement of that security.

•

Quality Control - Attaching a camera or sensor to the end of a production process to automatically inspect for defects.

•

Intelligent Search - An internet search engine that provides the most relevant content and banner ads based on the users' past behavior.

•

Fraud Detection - Detect fraudulen fraudulentt credit card transactions and automatically decline the charge.

46

How NN Work ( Mathematically) •

Linear and Non Linear Pa Patt ttern ern / Classification

•

Regression Regression / Function Estimation

•

Curve Fitting

Why to USE NN Parallel Processing Fault tolerance Self-organization Generalization Generalization ability Continuous adaptivity • • • • •

47

Artificial Neurons •

•

•

•

•

Neural networks are made up of nodes which have  –

Input edges, each with some weight 

 –

Output edges (with weights)

 –

An activation level (a function of the inputs)

Weights of edges can be positive or negative and may change over time (learning) The output function is the weighted sum of the activation levels of inputs The activation level is a linear or non-linear transfer transfer function “a” of the input : Some nodes are inputs, some are outputs. 48

Artificial Neural Networks Block Diagram

2/16/2011

49

Artificial Neural Networks Process

2/16/2011

50

The Perceptron x1 x2

. . x. n

w1 w2

Bias xn+1=-1 wn+1

q=wn+1 y

 wn

a=  bias+w x i

i

y=

{

1 if a  0 0 if a