QlikView Publisher Sizing

QlikView Publisher How to size your QlikView Publisher Server

Version Date Author

1.0 08 September 2014 Shane M Spencer

PUBLIC

Contents Introduction ............................................................................................................................................ 3 QlikView Distribution Service (QDS) & QlikView Batch (QVB) ................................................................... 4 Concurrent / Simultaneous Reloads & The Reload Process ...................................................................... 4 Disk, RAM & CPU Requirements .............................................................................................................. 6 Sizing Your (Publisher) Server .................................................................................................................. 7 Memory Metrics...................................................................................................................................... 8 Network Metrics ................................................................................................................................... 10 Process Metrics ..................................................................................................................................... 11 Key QlikView Processes ......................................................................................................................... 12 Web Server Metrics ............................................................................................................................... 13 Other Metrics ........................................................................................................................................ 14 List of Perfmon Counters ....................................................................................................................... 15 Perfmon Data Collector Template.......................................................................................................... 17

PUBLIC - Page 2 of 7

Introduction Correctly sizing the Publisher Server seems to be something that is overlooked, or at best based purely on guess work. However, it is important to understand your requirements so that your server has enough capacity (but not large amounts of excess) and to ascertain if you should have a dedicated Publisher Server or whether it can reside on the same physical machine as QlikView Server. The size of the QlikView Publisher will depend on several factors: the size of the data being loaded; how the data is loaded / document design; the number of Documents being reloaded; the number of concurrent tasks being run.

PUBLIC - Page 3 of 7

QlikView Distribution Service (QDS) The Distribution Service is the component that is responsible for performing the preparation and delivery of the QlikView files. A subcomponent of the QDS is the QlikView Batch (QVB) which is responsible for the actual Reload. In many ways these could be considered to be the very core of the QlikView Publisher.

QlikView Batch (QVB) When a Task is run (i.e. a reload is performed via QMC) QlikView Distribution Service will spawn a QVB processes that consumers RAM and CPU for the duration of the Task then it will disappear. For each running Task there will be an individual QVB process. This can be observed in Windows Task Manager of the Publisher Server. i.e.

PUBLIC - Page 4 of 7

Concurrent / Simultaneous Reloads Although many Tasks can be set to run at the same time it is more likely that only a few tasks will actually be consuming RAM and CPU the rest will be queuing for these to finish. The maximum number of Concurrent Reloads / Simultaneous Distributions can be set in QMC i.e. QlikView environment without Publisher licence.

i.e. QlikView environment with Publisher licence.

The Reload Process In simple terms, when a task is started the QlikView Distribution Service will open the (empty) document in Memory, load in the data (hence consuming RAM and CPU), and then write the reloaded Document to Disk. It is important you have enough resources to cope with your reload concurrency or that you throttle back your reload concurrency to avoid exceeding the capacity of these resources.

PUBLIC - Page 5 of 7

Disk Requirements As a reloaded document is being written to disk it does not over-write the existing document until the process is complete. That means there must be enough free capacity on the disk (where the document is Distributed to) to temporarily write the document to; once the write is complete the space consumed by the older version of the document will be released. An incorrectly sized environment is likely to use the Paging File (Disk that is used as Virtual Memory) so there should be enough Disk space to allow this to grow also.

RAM Requirements Whilst a Task is being executed a corresponding QVB process will be consuming Memory. The amount it will consume will depend not only on the amount of data being loaded but also the design of the document (as temporary tables may be created). For example we have observed the following: A QVW of 6.25GB consumes about 25GB of RAM when loaded in to Memory of the QlikView Server. The reload process (QVB) for this same document (temporarily) consumes about 47GB of RAM at its peak. A pre-requisite task that consolidates about 2 months of data in to QVD(s) consumes well over 100GB (and runs for over 5 hours) – when we streamlined the task to run quicker we shaved 45 minutes off the batch but peak Memory consumption was about10-20GB more. If there is not enough available Memory (RAM) for the reload Task(s) then Windows Server will automatically start using the Paging File (Disk that has been allocated as an extension to the server’s virtual Memory). This is extremely bad, especially for “In Memory” applications such as QlikView, as reading from Disk is 1000x slower than Memory.

CPU Requirements Whilst a Task is being executed a corresponding QVB process will be consuming CPU. Insufficient resources will mean Tasks are queuing for Processor (CPU) time.

PUBLIC - Page 6 of 7

Sizing Your (Publisher) Server It is relatively easy to calculate your Disk sizing requirements; however the CPU and RAM requirements of a Publisher Server are far more difficult to calculate. Yet these are paramount if you are to understand if you need a dedicated Publisher Server (separate from QlikView Server) and also so you can ensure that the Server is sized correctly with appropriate maximum Reload Task concurrency, no matter what type of deployment you go with. Therefore to understand your requirements you must Performance Test each reload task (running each task in isolation) whilst monitoring CPU, Disk and Memory metrics. Windows Perfmon can be used to collect data about CPU, Memory and Disk Utilisation. See the following document. http://community.qlik.com/docs/DOC-7050 Logical Disk space can easily be monitored by either the ‘% Free Space’ or ‘Free Megabytes” counters. It is in the final stages of the Task writes the Document to disk so it is at this point that close attention of disk space is required. If the documents are written to a NAS share rather than a mounted disk then Perfmon cannot monitor this. Processor (CPU) Utilisation can be monitored at Server level using % Process Time (and also System Processor Queue Length) and at a Process level using the % Process Time for QVB. Memory (RAM) Utilisation can be monitored at Server level using ‘% Committed Bytes in Use’ and/or ‘Available Megabytes’ and more accurately at Process level using Private Bytes for QVB (and QVS if the Publisher is on the QlikView Server). Paging File % Usage should also be monitored to spot when/if paging occurs. To ensure you have enough Memory (RAM) you will need 5-10GB for OS and other processes. The rest of the servers physical Memory can be used by QlikView Server (QVS) and/or QlikView Publisher (QVB). If the Memory required to run your tasks (concurrently) and host your documents in Memory exceeds the servers physical Memory (less 5-10GB) then you should consider some or all of the following options: • • •

reducing your max reload concurrency having a separate QlikView Server(s) and Publisher Server installing more memory / migrating to a bigger server

n.b. To estimate your base Memory footprint requirements (of QVS) excluding cached results, multiple the size on disk of you individual documents by 4.25. A more detailed analysis will need to be done to understand the amount of memory QVS requires to support user activity but that’s another story…

PUBLIC - Page 7 of 7