TAKING ADVANTAGE OF THE EMC CAPTIVA ARCHITECTURE Applying Best Practices to Optimize Performance Christopher Lund EMC
© Copyright 2011 EMC Corporation. All rights reserved.
1
Agenda • How InputAccel works • InputAccel 6.5 Benchmark Results • Tuning – – – – –
InputAccel Server Batches InputAccel Database Client Modules Capture Workflow
• Diagnosing Performance Issues The EMC® Captiva® InputAccel® and Dispatcher™ Version 6.5 Performance Sizing and Tuning Guide – which is available on PowerLink – provided much of the data for this presentation.
© Copyright 2011 EMC Corporation. All rights reserved.
2
How InputAccel Works A multi-machine capture application server • Server is the data tier (memory mapped)
InputAccel System Capture Capture Capture Modules Modules Modules
Processing Processing Processing Modules Modules Modules
Export Export Export Modules Modules Modules
• Server manages task queues • Server is multi-threaded… • VBA execution is single-threaded
InputAccel Servers
• DB writes are queued, but singlethreaded • Server uses asynchronous I/O • Most work done from a thread pool
WIP & Reports Tables
© Copyright 2011 EMC Corporation. All rights reserved.
• Clients are the executing tier (where scaling comes from)
3
How InputAccel Works An execution pipeline • Task queues are not FIFO
Execution Pipeline m1
m2
m3
m4
• Tasks are scheduled based on Priority, then creation date
t1
B4
t2
B3
B4
t3
B2
B3
B4
t4
B1
B2
B3
B4
t5
A4
B1
B2
B3
A4
B1
B2
• Assume short duration tasks
A2
A3
A4
B1
A1
A2
A3
A4
• Work is pushed, no polling
A1
A2
A3
t6 t7 t8 t9
© Copyright 2011 EMC Corporation. All rights reserved.
• Recovery is through reprocessing • Not a repository
• Tasks may be prefetched
4
How InputAccel Works Like a Petri net • A ProcessFlow defines the steps and trigger levels.
Petri Net
• Implicit fire when data is available
AA
BB
CC
• There is no predefined execution order • There is no end state – IADone triggered implies completed
D
© Copyright 2011 EMC Corporation. All rights reserved.
5
InputAccel 6.5 Benchmark Results • One 8-core IA Server, over 300 Client Modules • In-house, ideal conditions, your mileage will vary • Performance similar to IA 6.0 SP1 Performance Level
Overall Task Processing Rate (tasks/hour)
Processing Rate/CPU Core (tasks/hour)
Avg. CPU Utilization/C PU Core
50 active batches w/reporting disabled
2,672,007
324,001
67%
1000 active batches w/reporting disabled
1,990,892
248,862
53%
1000 active batches w/reporting enabled
1,384,910
173,114
32%
VMware ESX 4.x degrades throughput by approx 27%
© Copyright 2011 EMC Corporation. All rights reserved.
6
Tuning – InputAccel Server… • It is a server application. It will use all available resources • Recommendation: use Windows 2008 R2 for best performance • CPU – ias.exe is multi-threaded – Recommendation: use at least a 4-core CPU for optimum throughput, 8+ is better
• RAM – Recommendation: 4-8 GB RAM, no less than 4 GB – ias.exe is a 32-bit app, so 4 GB address space max – BatchMaxAddressSpaceK controls how much RAM IAS uses • 1.5 GB is the default • Set to 2 to 2.5 GB on 32-bit Windows with /3G option • Set to 3 to 3.5 GB on 64-bit Windows
– Only as many batches as will fit in BatchMaxAddressSpaceK are kept in RAM, when there are more, swapping occurs • Have your working set of batches small enough to all fit in RAM • Delete batches when you are done with them
© Copyright 2011 EMC Corporation. All rights reserved.
7
…Tuning – InputAccel Server • Disk – Used heavily for batches and processes – RAID 1+0 is best – RAID 5 usually is not fast enough and is not recommended •
Recommendation: use a caching controller with Read Ahead and Write Back
– SAN is OK, NAS not recommended – Turn off anti-virus scanning on IAS folder – IAS folder should be on a dedicated disk drive or array so it is not shared by other programs or the Windows swap file – #2 cause of slow performance: a slow hard drive
• Network – InputAccel has a “chatty” protocol between client modules and the InputAccel Server – For best performance, client modules, the InputAccel Server, Administration Console, and DB should be on the same sub-net – WANs usually have low bandwidth with high latency, which may make it unsuitable • •
Connecting client modules to InputAccel Server via a WAN is doable with adequate performance – depends on the client modules, # batches on IA server, and IPP InputAccel Server and IADB should not be connected by a WAN
© Copyright 2011 EMC Corporation. All rights reserved.
8
Tuning – Batches… • Batch size (IAB file) 5 – 20 MB is ideal < 100 MB is OK > 100 MB not recommended, but is allowed 10 – 100 pages per batch recommended 1000 pages per batch degrades throughput by 10% or more – < 10 pages per batch leads to too many small batches and too much batch swapping – – – – –
© Copyright 2011 EMC Corporation. All rights reserved.
9
…Tuning – Batches • Number of Batches
– Throughput is best when the working set of batches fit in RAM • Typically 50 – 500 fit depending on batch size and BatchMaxAddressSpaceK
– When there are more batches than fit in RAM, IA Server swaps batches in/out as needed • Swapping decreases IA Server throughput
– Up to 9,000 idle batches is possible with adequate performance • Idle means that all tasks within the batch have been processed and finished
– IA Server startup is slower with 1000’s of batches because IAS must load all batches into RAM to extract data • After startup they are swapped to disk and do not consume too many resources
– #1 cause of slow performance: too many active batches causing excessive swapping
© Copyright 2011 EMC Corporation. All rights reserved.
10
Tuning – InputAccel Database System Requirements • InputAccel Server requires a database to run • Only MSSQL is supported, versions 2005 and 2008 – Express supported only for low volume and no IA Reporting – Standard or Enterprise recommended for medium to high volume
• IADB stores: – – – –
IA configuration data Reporting data on completed batches/tasks Work In Progress (WIP) status Web services data
© Copyright 2011 EMC Corporation. All rights reserved.
11
Tuning – InputAccel Database System Requirements • Recommendation: 64-bit MSSQL server for best performance – 2-4 CPU cores – 4-8 GB RAM – RAID 1+0 with read/write caching controller w/ fast disks (15k RPM)
• InputAccel Server requires fast, uninterrupted access to the DB – If the DB goes offline, IA Server pauses
• Recommendation: put the DB and InputAccel Server on the same low-latency, high-bandwidth subnet • Reporting decreases the InputAccel Server throughput by about 10-30%, although it can be more
© Copyright 2011 EMC Corporation. All rights reserved.
12
Tuning – InputAccel Database Data Volume and Rates • IADB File Size and Growth Rates – Configuration data - typically < 100 MB – WIP data - 1 MB × # batches • WIP is transient and grows and shrinks as needed
– Error/Warning Log data – typically negligible and can be purged as needed – Reporting data • All reporting log rules off – 0 MB • Some or all reporting log rules are on Typically data grows 100 MB – 3 GB per hour Shrinks only when purged But… » Growth rate depends on page volume and which log rules are on » Overall size depends on # days of data retained
– Audit Log data • All audit log rules off – 0 MB • All audit logs rules on – 2× the growth rate of Reporting data
© Copyright 2011 EMC Corporation. All rights reserved.
13
Tuning – InputAccel Database Data Volume and Rates • InputAccel Database transaction rates Log Rules Enabled
Estimated Transaction Rate
None
IAS Tasks / Hour × 0.075
Reports only (no Audit)
IAS Tasks / Hour × 2.5
Audit only (no Reports)
IAS Tasks / Hour × 5.0
Reports + Audit
IAS Tasks / Hour × 7.5
© Copyright 2011 EMC Corporation. All rights reserved.
14
Tuning – InputAccel Database Improving Performance • Defragment and rebuild indexes – up_ReorganizeIndex – defragments all indexes – up_RebuildIndex – rebuilds all indexes
• Purge reporting and auditing data – Reporting and Auditing tables grow continuously – You must schedule purges via the Admin Console
• Recommendation: generate reports during non-peak hours – Generating Reports runs complex queries that place a heavy load on IADB and MSSQL
• Store MSSQL transaction logs and data files on separate hard drives/controllers
© Copyright 2011 EMC Corporation. All rights reserved.
15
Tuning – Client Modules • Each client module has its own unique tuning characteristics – Example: val2xml slows as more IA Values are exported and is slower when triggered at level 1 than level 7 – See “EMC® Captiva® InputAccel® and Dispatcher™ Version 6.5 Performance Sizing and Tuning Guide” for details
• Parameters on the client machines can be modified to optimize performance – Stored in settings.ini located in %ALLUSERSPROFILE%\EMC\InputAccel – PrefetchDefault (default = 2) – the number of additional tasks the InputAccel Server sends to each client module – FileCacheSize (default = 8) – CacheSize (default = 1,048,576) – CacheCount (default = 200,000), previously 20,000 – the number of files and IA Values the client module caches – IAClientDebug (default = 0) – set to 1 to capture the debug log iaclient.log in %ALLUSERSPROFILE%\EMC\InputAccel (previously was created in C:\)
© Copyright 2011 EMC Corporation. All rights reserved.
16
Tuning – Client Modules • CPU intensive client modules like NuanceOCR & ImageEnhancement – Require fast CPUs – Run one instance for each CPU core. • e.g. 4 CPU cores, run 4 instances of NuanceOCR
• Non-CPU intensive client modules like val2xml & Documentum Export – Performance is limited by other resources (disk, network) – Run at least one instance for each CPU core and possibly more
© Copyright 2011 EMC Corporation. All rights reserved.
17
Tuning – Client Modules Using InputAccel over a WAN… • InputAccel Server, Database and Administration Console – Require high-speed, low-latency connections – Must be on the same LAN as each other
• Unattended client modules – Often require high-speed, low-latency connections for best throughput – Generally do not need to be remote from the InputAccel Server – Should be on the same LAN as the InputAccel Server
• Attended client modules – Performance varies by environment and module – Detailed guidance is in the EMC® Captiva® InputAccel® and Dispatcher™ Version 6.5 Performance Sizing and Tuning Guide – Recommendation: do benchmark testing to ensure adequate performance
© Copyright 2011 EMC Corporation. All rights reserved.
18
Tuning – Client Modules …Using InputAccel over a WAN • Attended client module details – ScanPlus • Not recommended, but may perform acceptably with scanner hardware compression or small scanned images • Works best when bandwidth is ≥50 Mbps and round trip latency ≤25 ms
– IndexPlus • Generally performs well over a WAN (except for thumbnail display) • Displaying the batch list takes about 25% longer • Works best when bandwidth is ≥1.5 Mbps and round-trip latency ≤50 ms
– Dispatcher Classification Edit and Dispatcher Validation • Should not be used over a WAN
• Recommendation: consider using VMware View or Citrix for remote operators – Module executes on the LAN, screen display is over the WAN – Supports remote scanning • ScanPlus is on the LAN, the scanner is on the remote machine • Use scanner hardware compression for best results
– Maximizes InputAccel Server-to-client module throughput – EMC OnDemand uses VMware View
© Copyright 2011 EMC Corporation. All rights reserved.
19
Tuning – Capture Workflow Trigger Levels • Triggering at level 0 or 1 – Usually gives better throughput than level 7 – The tasks within a batch can be distributed among many client modules – Which results in faster end-to-end processing of any single batch
• Triggering at level 7 – Is less work for InputAccel Server as it has fewer tasks to manage – Under some circumstances may provide better overall throughput at the expense that any single batch may take more time to process
• Unsupported: accessing external resources in IPP VBA code – – – –
VBA execution with InputAccel Server is single-threaded The external resource may be slow or not present If InputAccel Server needs to wait for the resource, all other tasks block Put more complicated custom code logic on the client through the .NET Code module or client scripting
© Copyright 2011 EMC Corporation. All rights reserved.
20
Diagnosing Performance Issues • IA Server Performance Counter – – – – – – – – –
Batches loaded and loads/second Connections Disk bytes written & read/second VBA calls/second & queue length Processing Message Count Network bytes written & read/second Packets send & received/second Pending I/O (~ # of asynchronous sends in progress) Event (db) queue length
• Data Access Layer Performance Counters – – – –
Data Requests/second % Load Factor Avg. Execution Time Millisec Current connection count
© Copyright 2011 EMC Corporation. All rights reserved.
21
EMC OnDemand Captiva Instant Cloud Implementation
© Copyright 2011 EMC Corporation. All rights reserved.
22
IIG Applications on EMC OnDemand
ENTERPRISE CAPTURE
CONTENT MANAGEMENT
CUSTOMER COMMUNICATIONS
CASE MANAGEMENT
Captiva
Documentum ECM
Document Sciences
Documentum xCP
INFORMATION
GOVERNANCE SourceOne
Cloud Management Virtualization and Security Storage Networks
© Copyright 2011 EMC Corporation. All rights reserved.
23
Summary/Key Takeaways • Focus on system throughput, not per-task time • Use high-speed hardware – disk drives and networks • Minimize disk I/O where possible – Keep active batches in memory – Avoid excessive reporting
• Parallelize – multi-core CPUs and task granularity • Use performance counters to find bottlenecks
© Copyright 2011 EMC Corporation. All rights reserved.
24
Q&A
Chris Lund
[email protected] (858) 320-1215
© Copyright 2011 EMC Corporation. All rights reserved.
25
Learn More About EMC Captiva
Go to: www.EMC.com/Captiva
© Copyright 2011 EMC Corporation. All rights reserved.
26
Captiva @ Momentum 2011 Thursday 10:00 AM
AP Automation: Best Practices for Capturing and Integrating Paper Invoices into your Accounts Payable Processes
© Copyright 2011 EMC Corporation. All rights reserved.
Galileo 705
27
GET SOCIAL
DOWNLOAD Apple: http://bit.ly/MMTM11 Other: http://bit.ly/MMTMemc
Come to the Momentum Lounge today to play, win, learn and more. © Copyright 2011 EMC Corporation. All rights reserved.
28
© Copyright 2011 EMC Corporation. All rights reserved.
29
THANK YOU
© Copyright 2011 EMC Corporation. All rights reserved.
30
