Image and Video Processing Using MATLAB
April 12, 2011 Rochester, NY
© 2011 The MathWorks, Inc.1
Agenda
9:00 a.m.
Welcome and MathWorks Overview
9:15 a.m.
Image Processing with MATLAB
10:30 a.m.
Break
10:45 a.m.
Video Processing with MATLAB
11:30 a.m.
System Design and Hardware Targeting
12:00 p.m.
Summary and Wrap-up
2
From MathWorks
Sr. Account Manager – Ed Donovan –
[email protected] – 508-647-4251
Application Engineer – Stephanie Kwan
3
Agenda
9:00 a.m.
Welcome and MathWorks Overview
9:15 a.m.
Image Processing with MATLAB
10:30 a.m.
Break
10:45 a.m.
Video Processing with MATLAB
11:30 a.m.
System Design and Hardware Targeting
12:00 p.m.
Summary and Wrap-up
4
Where is image processing used?
5
Common Image Processing Tasks
Perform segmentation Deblur and remove noise Measure image features Perform edge detection Apply morphological operations Enhance image contrast Perform image registration Apply transforms
6
Common Image Processing Challenges
Reading and writing to various file formats Visualizing images and intermediate results Testing algorithms with what-if scenarios Identifying causes of algorithm failure Processing large images with limited memory Executing algorithms faster
7
Technical Computing Tasks Access
Explore and Discover
Files Data Analysis and Modeling
Software
Algorithm Development
Share Reporting and Documentation
Outputs for Design
Code and Applications
Hardware
Application Development
Deployment
Automate 8
MATLAB – A Platform for Technical Computing Access
Explore and Discover
Files Data Analysis and Modeling
Software
Algorithm Development
Share Reporting and Documentation
Outputs for Design
Code and Applications
Hardware
Application Development
Deployment
Automate 9
Core MathWorks Products
The leading environment for technical computing
The industry-standard, high-level programming language for algorithm development Numeric computation Data analysis and visualization Toolboxes for signal and image processing, statistics, optimization, symbolic math, and other areas Foundation of MathWorks products
10
Image Processing Toolbox Perform image processing, analysis, visualization, and algorithm development
Image analysis Image enhancement Spatial transformation Image registration Morphological operations ROI-based processing Image display and exploration
11
Demo: Camera Pipeline From Sensor Data to Image
Scene
Image
12
Demo: Camera Pipeline
1. Noise Reduction 2. Demosaic
3. Tone Mapping 4. White Balance 5. Gamma Correction
13
Image Acquisition Toolbox
Acquire images and video directly into MATLAB and Simulink
Configure device properties
Perform background acquisition
Synchronize multimodal devices
Configure, acquire, and preview live video data using a graphical interface
14
Image Acquisition Toolbox Hardware Support
Industry standard support: – Frame grabbers
Analog
Camera Link
– DCAM compatible FireWire (IIDC 1394) – GigE Vision – Common OS webcam interfaces
Operating system support: – Windows – Linux – Macintosh 15
Image Acquisition Toolbox Hardware Support
Manufacturers include: – Allied Vision Technologies
– Basler – Baumer – DALSA – FLIR
– Hamamatsu – Lumenera – Matrox Imaging – National Instruments
– PixeLINK – Point Grey – Qimaging – Sony
See Supported Hardware Pages for more information 16
Where is Large Imagery Used?
Satellite imagery Aerial surveys Super-resolution Image sequences and stacks Volumetric data Multispectral and hyperspectral Mosaics and panoramic imagery
17
Common Large Image Challenges
“Out of memory” errors Slow processing
18
Some Steps to Handle Large Images
Try memory command to see available space
Preallocate space for variables Minimize numerical data types Load small portions of an image Visualize only what is necessary Incorporate more hardware resources
Can lead to complex code 19
blockproc and rsetwrite
Block processing – Automatically divides an image into blocks for processing – Reduces memory usage – Processes arbitrarily large images
Reduced resolution data set – Avoids memory demands in visualizing large images
20
Demo: Cell Segmentation with blockproc
21
Improving Performance
MATLAB – Preallocate space for variables – Identify bottlenecks with Profiler – Vectorize code
Image Processing Toolbox – Many toolbox functions are faster as of R2010a
Parallel Computing Toolbox – Utilize multicore processors, clusters, and GPUs
22
Parallel Computing Toolbox Going Beyond Serial MATLAB Applications
Worker
Worker Worker
TOOLBOXES
BLOCKSETS
Worker Worker
Worker Worker
Worker
23
Parallel Computing enables you to …
Larger Compute Pool
Speed up Computations
Larger Memory Pool
Work with Large Data 11
26
41
12
27
42
13
28
43
14
29
44
15
30
45
16
31
46
17
32
47
17
33
48
19
34
49
20
35
50
21
36
51
22
37
52
24
Processes
Distributing Tasks (Task Parallel)
Time
Time 25
Demo: Camera Pipeline and Parallel Computing
26
Parallel Computing Toolbox Support for GPUs
Automatically use GPU’s with supported built-in MATLAB functions – conv2, filter2
Run supported MATLAB code on the GPU – arrayfun
Create kernals from existing CUDA code and PTX files Run kernals on the GPU from MATLAB
27
Why would you want to use a GPU?
Speed up execution of computationally intensive simulations For example: – Performance: A\b with Double Precision
28
Why use MATLAB for Image Processing?
Read and write many image file formats Visualize and explore images interactively Connect directly to cameras and frame grabbers Use a large library of built-in functions Quickly build custom image processing algorithms Block-process large images to avoid memory issues Process images faster with multiple cores and clusters
29
Agenda
9:00 a.m.
Welcome and MathWorks Overview
9:15 a.m.
Image Processing with MATLAB
10:30 a.m.
Break
10:45 a.m.
Video Processing with MATLAB
11:30 a.m.
System Design and Hardware Targeting
12:00 p.m.
Summary and Wrap-up
30
Agenda
9:00 a.m.
Welcome and MathWorks Overview
9:15 a.m.
Image Processing with MATLAB
10:30 a.m.
Break
10:45 a.m.
Video Processing with MATLAB
11:30 a.m.
System Design and Hardware Targeting
12:00 p.m.
Summary and Wrap-up
31
Common Video Processing Tasks
Convert between color spaces Enhance video contrast Perform segmentation De-interlace video frames Calculate running statistics Overlay text and graphics
32
Common Video Processing Challenges
Reading and writing to various file formats Managing buffers and intermediate storage Testing video algorithms with test cases Accessing standard video algorithms Visualizing results and intermediate steps Displaying text and graphics
33
Demo: Working with Video in MATLAB
Read in and write out data using VideoReader and VideoWriter
What happens if the video file is very long or the frame size is large?
34
System Objects
MATLAB objects that represent algorithms and I/O capabilities that process data frame by frame Instantiate and configure –
Execute within a loop –
reader = vision.MultimediaFileReader(‘viptraffic.avi’) step(reader)
Standard API – – –
reset isDone close
35
Video Processing in MATLAB myVid = VideoReader(‘myvideofile.avi’); numFrames = myVid.NumberOfFrames; numIter = 10; opticalFlowIn = zeros([size(currentFrame) 5]); opticalFlowOutput = zeros([size(currentFrame) numFrames]); i = 1; Explicit state management while i 400 MATLAB operators and functions • > 200 System objects for • Signal processing • Communications • Computer vision
52
Demo: Camera Pipeline Revisited
Generate C code from MATLAB code
53
Targeting DSPs and FPGAs MATLAB and Simulink • Algorithm development • Debugging and profiling • System design Fixed-Point Modeling Generate code
Verify design Link to Embedded Software
Integrated Development Environment • Compiler • Build automation tools • Debugger
DSP/ FPGA 54
Demo: Camera Pipeline System Design From Sensor Data to Image
Scene
Image
55
Why use MATLAB and Simulink for embedded hardware design?
Automatically convert MATLAB to C code Reduce the number of versions of code Simplify the design process Reuse MATLAB and legacy code in models Convert floating-point algorithms to fixed point Optimize algorithms for DSPs and FPGAs Test and validate an algorithm’s implementation
56
Agenda
9:00 a.m.
Welcome and MathWorks Overview
9:15 a.m.
Image Processing with MATLAB
10:30 a.m.
Break
10:45 a.m.
Video Processing with MATLAB
11:30 a.m.
System Design and Hardware Targeting
12:00 p.m.
Summary and Wrap-up
57
What You’ve Seen Today MATLAB for image and video processing Image Processing Toolbox – Visualize, analyze, and process images
Image Acquisition Toolbox – Test image processing algorithms with live data
Computer Vision System Toolbox – Visualize, analyze, and process video – Motion estimation, tracking, and stereo vision
Parallel Computing Toolbox – Process images faster using multiple core and clusters
MATLAB Coder – Convert MATLAB code to C/C++ code 58
Doheny Eye Institute Develops Next-Generation of Retinal Prosthesis with MathWorks Tools
Challenge Develop next-generation, higher-resolution retinal prostheses
Solution
Illustration of a retinal prosthesis prototype.
Use MathWorks tools to develop, simulate, and automatically generate code for real-time image processing algorithms
“With Video and Image Processing
Results
Blockset and Target Support
Development time reduced from months to weeks DSP deployment streamlined Patient testing improved
Package, we rapidly prototype our image and video processing algorithms on the DM642 board. This
can save me days or weeks of time.” Neha Parikh Doheny Eye Institute
59
Given Imaging Develops Camera-in-aCapsule Using MATLAB to Improve the Diagnosis of Gastrointestinal Disorders
Challenge Create an alternative to endoscopy and other invasive gastrointestinal imaging procedures
Solution
The PillCam SB 2 video capsule.
Use MATLAB and companion toolboxes to develop and implement a swallowable video capsule
Results Fast, efficient development Easy access to precise diagnostic information Improved patient care
“With MATLAB, we simulated the intended system and fine-tuned it at the early stages of implementation, enabling us to develop critical engineering programs that met requirements on the first iteration.” Rafi Nave Given Imaging
Link to user story
60
Additional Articles and User Stories
Newsletter: Robot Soccer
User Story: Dutch Epilepsy Clinics Foundation
Newsletter: JPL Mars rover
Newsletter: Parking assistant at BMW 61
From a Leading Textbook Author ... "I have used a number of commercial image processing packages over the years, and prefer the MathWorks Image Processing Toolbox for several reasons: the wide variety of functions it provides, the user’s ability to write additional functions with minimal effort, the quality of the software, and the high level of support." Rafael C. Gonzalez Professor Emeritus University of Tennessee 62
Support and Community
63
Consulting Services Accelerating return on investment A global team of experts supporting every stage of tool and process integration
Continuous Improvement
Process and Technology Automation Process and Technology Standardization Full Application Deployment
Process Assessment Component Deployment Advisory Services
Jumpstart Migration Planning
Research
Advanced Engineering
Product Engineering Teams
Supplier Involvement 64
Training Services Exploit the full potential of MathWorks products
Flexible delivery options:
Public training available worldwide Onsite training with standard or customized courses Web-based training with live, interactive instructor-led courses Self-paced interactive online training
More than 30 course offerings:
Introductory and intermediate training on MATLAB, Simulink, Stateflow, code generation, and Polyspace products Specialized courses in control design, signal processing, parallel computing, code generation, communications, financial analysis, and other areas 65
MATLAB Central
Community for MATLAB and Simulink users Over 1 million visits per month File Exchange –
– –
Newsgroup – –
Upload/download access to free files including MATLAB code, Simulink models, and documents Ability to rate files, comment, and ask questions More than 12,500 contributed files, 300 submissions per month, 50,000 downloads per month
Web forum for technical discussions about MathWorks products More than 300 posts per day
Blogs – –
Based on February 2011
Commentary from engineers who design, build, and support MathWorks products Open conversation at blogs.mathworks.com 66
Connections Program More than 400 add-on products and services that complement and extend MathWorks products:
Specialized third-party toolboxes for MATLAB Interfaces to third-party software and hardware products Specialized training courses and consulting services System integrators and suppliers that incorporate MathWorks products
67
Book Program More than 1200 books for educational and professional use, in 26 languages
Controls Signal processing Image processing Biosciences Communications Mechanical engineering
Mathematics Aerospace engineering Environmental sciences Chemistry Finance Electronics
68
Technical Support
Resources
Over 100 support engineers – All with MS degrees (EE, ME, CS) – Local support in North America, Europe, and Asia Comprehensive, product-specific Web support resources
High customer satisfaction
95% of calls answered within three minutes 70% of issues resolved within 24 hours 80% of customers surveyed rate satisfaction at 80–100%
69
For More Information
Experiment with product by downloading a trial Peruse videos, webinars, user stories, and demos online Contact us – Talk to a sales representative to get answers to your questions – Discuss your projects with MathWorks applications engineers
70
