Splunk 4.0.9 Installation Guide

Splunk Installation Manual Version: 4.0.9 Generated: 5/10/2010 03:24 am Copyright Splunk, Inc. All Rights Reserved

Table of Contents Welcome to the Splunk Installation Manual......................................................................................1 What's in the Installation Manual.................................................................................................1 System requirements..................................................................................................................1 Components of a Splunk deployment........................................................................................3 Splunk's architecture and what gets installed............................................................................4 About Splunk licenses................................................................................................................5 More about Splunk Free.............................................................................................................9 Before you install......................................................................................................................10 Step by step installation procedures...............................................................................................11 Choose your platform...............................................................................................................11 Install on Windows...................................................................................................................11 Install on Windows via the commandline.................................................................................15 Install on Linux.........................................................................................................................20 Install on Solaris.......................................................................................................................22 Install on Mac OS.....................................................................................................................25 Install on FreeBSD...................................................................................................................27 Install on AIX............................................................................................................................29 Install on HP-UX.......................................................................................................................31 Install a license.........................................................................................................................32 Start Splunk for the first time............................................................................................................35 Start Splunk for the first time....................................................................................................35 Migrate from 3.4.x or earlier..............................................................................................................37 What to expect when migrating to 4.0......................................................................................37 Migrate on UNIX.......................................................................................................................43 Migrate on Windows.................................................................................................................45 Steps for manual migration to Splunk 4.x.................................................................................47 Upgrade from 4.0 or later..................................................................................................................52 Upgrade Splunk on Windows...................................................................................................52 Upgrade Splunk on Linux, Solaris, FreeBSD, HP-UX, AIX, and MacOS.................................53 Other tasks.........................................................................................................................................55 Run Splunk as a different or non-root user..............................................................................55 Uninstall Splunk.......................................................................................................................56 Correcting the user selected during Windows installation........................................................58 Configure a standalone 3.4.x deployment server.....................................................................59 What comes next?.............................................................................................................................61 Ready to start using Splunk?...................................................................................................61 Estimating your storage requirements......................................................................................61 Hardware capacity planning for your Splunk deployment........................................................62

i

Table of Contents Reference............................................................................................................................................70 PGP Public Key........................................................................................................................70 File manifest.............................................................................................................................70

ii

Welcome to the Splunk Installation Manual What's in the Installation Manual What's in the Installation Manual

Use this guide to find system requirements, licensing information, and procedures for installing or migrating Splunk. Find what you need

You can use the table of contents to the left of this panel, or simply search for what you want in the search box in the upper right. If you're interested in more specific scenarios and best practices, you can visit the Splunk Community Wiki to see how other users Splunk IT. Make a PDF

If you'd like a PDF of any version of this manual, click the pdf version link above the table of contents bar on the left side of this page. A PDF version of the manual is generated on the fly for you, and you can save it or print it out to read later.

System requirements System requirements

Before you download and install the Splunk software, read the following sections for the supported system requirements. If you have ideas or requests for new features to add to future releases, email Splunk Support. Also, you can follow our Product Roadmap. Refer to the download page for the latest version to download. Check the release notes for details on known and resolved issues. For a discussion of hardware planning for deployment, check out the topic on capacity planning in this manual. Supported OSes

Splunk is supported on the following platforms. • Solaris 9, 10 (x86, SPARC) • Linux Kernel vers 2.6.x and above (x86: 32 and 64-bit) • FreeBSD 6.1 and 6.2 (x86: 32 and 64-bit) • Windows 2003 (x86: 64-bit, supported but not recommended on 32-bit) • Windows 2008 (x86: 64-bit, supported but not recommended on 32-bit) • WindowsXP (x86: 32-bit) • Vista (x86: 32-bit, 64-bit)

1

• MacOSX 10.5 and 10.6 (32-bit & 64bit in one download, 10.6 is only supported in 32-bit mode) • AIX 5.2 and 5.3 • HP-UX 11iv2 (11.22) and 11iv3 (11.31) (PA-RISC or Itanium) FreeBSD 7.x

To run Splunk 4.x on 32-bit FreeBSD 7.x, install the compat6x libraries. Splunk Support will supply "best effort" support for users running on FreeBSD 7.x. For more information, refer to this Community wiki topic. Creating and editing configuration files on non-UTF-8 OSes

Splunk expects configuration files to be in ASCII/UTF-8. If you are editing or creating a configuration file on an OS that is non-UTF-8, you must ensure that the editor you are using is configured to save in ASCII/UTF-8. Supported browsers

• Firefox 2 and 3.0.x • Firefox 3.5 (with Splunk version 4.0.6 and later) • Internet Explorer 6, 7 and 8 • Safari 3 Recommended hardware

Splunk is a high-performance application. If you are performing a comprehensive evaluation of Splunk for production deployment, we recommend that you use hardware typical of your production environment; this hardware should meet or exceed the recommended hardware capacity specifications below. For a discussion of hardware planning for deployment, check out the topic on capacity planning in this manual. Note: Running Splunk in virtual machine (VM) mode on any platform will degrade performance. Recommended and minimum hardware capacity

Platform

Recommended hardware capacity/configuration

Minimum supported hardware capacity

Non-Windows platforms

2x quad-core Xeon, 3GHz, 8GB RAM, RAID 0 or 1+0, with a 64 bit OS installed.

1x1.4 GHz CPU, 1 GB RAM

Windows platforms

2x quad-core Xeon, 3GHz, 8GB RAM, RAID 0 or 1+0, with a 64 bit OS installed.

Pentium 4 or equivalent at 2Ghz, 2GB RAM

• All configurations other than Splunk light forwarder instances require at least the recommended hardware configuration. • The minimum supported hardware guidelines are designed for personal use of Splunk.

2

Important: For all installations including forwarders, a minimum of 2GB hard disk space for your Splunk installation is required in addition to the space required for your index. Refer to this topic on estimating your index size requirements in this manual for some planning information. Hardware requirements for Splunk light forwarders

Recommended Dual Core 1.5Ghz+ processor, 1GB+ RAM Minimum 1.0 Ghz processor, 512MB RAM For more information on deployment planning, refer to the Deployment section of the Splunk community KnowledgeBase. Supported file systems

Platform

File systems

Linux

ext2/3, reiser3, XFS, NFS 3/4

Solaris

UFS, ZFS, VXFS, NFS 3/4

FreeBSD

FFS, UFS, NFS 3/4

Mac OS X HFS, NFS 3/4 AIX

JFS, JFS2, NFS 3/4

HP-UX

VXFS, NFS 3/4

Windows

NTFS, FAT32

Considerations regarding NFS

NFS is usually a poor choice for Splunk indexing activity, for reasons of performance, resilience, and semantics. In environments with very high bandwidth, very low latency links, that are kept highly reliable, it can be an appropriate choice. Typically, this is a NAS accessed via the NFS protocol. Note: Several other file systems are supported. If you run Splunk on a filesystem that is not listed above, Splunk may run a startup utility named locktest. Locktest is a program that tests the start up process. If locktest runs and fails, the filesystem is not suitable for running Splunk. Note: On FreeBSD, mounting as nullfs is not supported. Supported server hardware architectures

32 and 64-bit architectures are supported for some platforms. download page page for details.

Components of a Splunk deployment Components of a Splunk deployment

Splunk is simple to deploy by design. By using a single software component and easy to understand configurations, Splunk can coexist with existing infrastructure or be deployed as a universal platform for accessing IT data.

3

The simplest deployment is the one you get by default when you install Splunk: indexing and searching on the same server. Data comes in from the sources you've configured, and you log into Splunk Web or the CLI on this same server to search, monitor, alert, and report on your IT data. Depending on your needs, you can also deploy components of Splunk on different servers to address your load and availability requirements. This section covers these potential components: Indexer

In this mode, indexers, or index servers, provide indexing capability for local and remote data and host the primary Splunk datastore, as well as Splunk Web. Refer to "How indexing works" in the Admin Manual for more information. Search head

In this mode, a Splunk instance is configured to direct user search requests to one or more indexers. Use distributed search to configure a search head to search across a pool of indexers. Forwarder

Forwarders use the same Splunk software package but do not store indexed data locally. All indexed data is forwarded to remote index servers. To reduce operational footprint, Splunk Web is not used. Refer to the documentation on setting up a Splunk instance as a forwarder. Deployment server

Both indexers and forwarders can also act as deployment servers. A deployment server distributes configuration information to running instances of Splunk via a push mechanism which is enabled through configuration. Refer to the documentation on setting up a Splunk instance as a deployment server. Functions at a glance

Functions Indexing Web Direct search Forward to indexer Deploy configurations

Indexer

Search head

Forwarder

Deployment server

x x x x x

x

x

Splunk's architecture and what gets installed Splunk's architecture and what gets installed

This topic discusses Splunk's internal architecture and processes at a high level. If you're looking for information about third-party components used in Splunk, refer to the credits section in the Release notes.

4

Processes

A Splunk server runs two processes on your host, splunkd and splunkweb: • splunkd is a distributed C/C++ server that accesses, processes and indexes streaming IT data. It also handles search requests. splunkd processes and indexes your data by streaming it through a series of pipelines, each made up of a series of processors. ♦ Pipelines are single threads inside the splunkd process, each configured with a single snippet of XML. ♦ Processors are individual, reusable C or C++ functions that act on the stream of IT data passing through a pipeline. Pipelines can pass data to one another via queues. splunkd supports a command line interface for searching and viewing results. • splunkweb is a Python-based application server based on CherryPy that provides the Splunk Web user interface. It allows users to search and navigate IT data stored by Splunk servers and to manage your Splunk deployment through a Web interface. splunkweb and splunkd can both communicate with your Web browser via REST: • splunkd also runs a Web server on port 8089 with SSL/HTTPS turned on by default. • splunkweb runs a Web server on port 8000 without SSL/HTTPS by default. Architecture diagram

About Splunk licenses About Splunk licenses

Each instance of Splunk must have its own license. This topic discusses the different Splunk licenses, how to install or update a license, and what to do when you have a violation on your license. Note: When running Splunk Enterprise, you must purchase a separate license for every instance of Splunk that you deploy.

5

Enterprise vs. Free license

Splunk provides two standard types of licenses, an Enterprise license, and a Free license. When you download Splunk for the first time, you are asked to register. Your registration authorizes you to receive an Enterprise trial license, which allows a maximum indexing volume of 500 MB/day. The Enterprise trial license expires 60 days from download. If you are running with a Enterprise trial license and your license expires, Splunk continues to index your data. However, you will not be able to search until you install a new license. Once you have installed Splunk, you can choose to run Splunk with the Enterprise trial license until it expires, purchase an Enterprise license, or switch to the Free license, which is included. The Free license is not a trial license and does not have an expiration date. It also allows 500MB/day of indexing volume, but the following features that are available with the Enterprise license are disabled: • Multiple user accounts and role-based access controls • Distributed search • Forwarding in TCP/HTTP formats (you can forward data to other Splunk instances, but not to non-Splunk instances) • Deployment management (including for clients) • Scheduled saved searches (including summary indexing) and alerting/monitoring Learn more about Splunk with a Free license Find more information about the different license features here. Also, read Splunk's Free license agreement. Forwarding license

A license isn't required to enable forwarding, but enables security on the forwarder so that users must supply username and password to access it. Splunk includes a forwarder license that you can install on each Splunk forwarder. This 1 MB/day forward-only license is not subtracted from your existing license(s) and can be applied to multiple forwarders. Each instance of Splunk that does any indexing must have its own license. 1. Stop Splunk: ./splunk stop 2. Copy $SPLUNK_HOME/etc/splunk-forwarder.license to $SPLUNK_HOME/etc/splunk.license 3. Start Splunk: ./splunk start This license does not limit how much data you can forward from that machine.

6

Search head license

A search head does not normally index any data, but you don't want unrestricted access on there either. Apply a forwarder license to enable authentication on your search head instance. Summary index searches count towards your indexed data volume. If you run these searches on your search head, it will violate the forwarder license and search functionality will be disabled. To avoid this, configure your summary index searches on your indexing instances instead. Custom trial Enterprise license

You can request trial Enterprise licenses of varying size and duration. The default evaluation period is 60 days. If you are preparing a pilot for a large deployment and have requirements for a longer duration or higher indexing volumes during your trial, contact Splunk Sales with your request. Preview license

Splunk's Preview releases require a different license that is not compatible with other Splunk releases. Also, if you are evaluating a Preview release of Splunk, it will not run with a Free or Enterprise license. Preview licenses typically enable Enterprise features, they are just restricted to Preview releases. If you are evaluating a Preview version of Splunk, it will come with its own license. View your license and usage details

You can view details about your license from Splunk Web or the command line interface (CLI). The details about your license include general information, such as the type of license, the indexing level, and the expiration date of the license. View license information in Splunk Web

To view your license details in Splunk Web, go to Manager > License. Under "License & Usage". In addition to the general information about your license, the details include: the count of days until your license expires, the peak indexed amount (in MB), and the count of violations against your license. Note: You can also install and update your license from this page. You can also use search to learn information about indexing volumes. For a report on the daily indexed volume, use this search: index=_internal todaysBytesIndexed LicenseManager-Audit NOT source=*web_service.log | eval Daily_Indexing_Volume_in_MBs = todaysBytesIndexed/1024/1024 | timechart avg(Daily_Indexing_Volume_in_MBs) by host Show license information in the CLI

If you have access to the CLI, you can view details about your license with: ./splunk show license

The CLI displays the same general details, but also includes information about the state of your license, such as the maximum number of violations (Max Violations) permitted by the license within 7

the grace period moving window (Violation Period). Splunk also displays an "Expiration State", which tells you when you license is either "7" days or "1" day from expiring; otherwise, it is "ok". Install or update your license

All Splunk servers have a license located in $SPLUNK_HOME/etc/, whether it is a Free license (splunk-free.license) or an Enterprise license (splunk.license). You can install and update your licenses with the CLI or from Splunk Web's Manager > License page. Refer to the Admin Manual for instructions to install or update your Splunk license. Migrating to 4.0

If you migrated a 3.x Splunk instance directly to 4.0, remove the $SPLUNK_HOME/etc/splunk.license file before you run Splunk. The instance will then pick up the 60-day Enterprise trial license that is included with 4.0. If you have a current Enterprise license/support contract for 3.x, an updated license is waiting for you. Log into splunk.com and go to http://www.splunk.com/store/myorders to pick it up. Replace your existing $SPLUNK_HOME/etc/splunk.license file with the new file, or use the instructions for updating your Splunk license. Licenses for distributed deployments

If you have Splunk running on multiple hosts in a distributed environment, you must have a separate unique license key for each host. If you have been using a single license key for all your hosts, this will not work in versions 4.0 and later (except with an Enterprise Trial license, starting with version 4.0.3). Contact Splunk Support to have your license broken up into multiple keys for this purpose, or email Splunk Sales to purchase additional keys. License violations

Violations occur when you exceed the maximum indexing volume allowed for your license. If you exceed your licensed daily volume on any one calendar day, you will get a violation warning. The message persists for 14 days. If you have 5 or more violations on an Enterprise license or 3 violations on a Free license in a rolling 30-day period, search will be disabled. Search capabilities return when you have fewer than 5 (Enterprise) or 3 (Free) violations in the previous 30 days or when you apply a new license with a larger volume limit. Note: During a license violation period, Splunk does not stop indexing your data. Splunk only blocks access while you exceed your license. Note: Searches to the _internal index are not disabled even during a licensing-enforcement period, so you can still access the Indexing Status dashboard, or run searches against _internal to diagnose the licensing problem. Got License Violations? Click here to troubleshoot.

8

More about Splunk Free More about Splunk Free

Splunk Free is a totally free (as in beer) version of Splunk. It allows you to index up to 500MB/day and will never expire. If you go over 500MB/day more than 3 times in a 30 day period, Splunk will continue to index your data, but search will be disabled until you are back down to 3 or fewer times in the 30 day period. What's it for?

Splunk Free is designed for personal, ad-hoc search and visualization of IT data. You can use Splunk Free for ongoing indexing of small volumes (Programs>Splunk or • Open a Web browser and navigate to http://localhost:8000. Log in using the default credentials: username: admin and password: changeme . Be sure to change the admin password as soon as possible and make a note of what you changed it to. Now that you're ready to use Splunk, refer to the User Manual and begin using Splunk! Avoid IE Enhanced Security pop-ups

To avoid IE Enhanced Security pop-ups, add the following URLs to the allowed Intranet group or fully trusted group in IE: • quickdraw.splunk.com • the URL of your Splunk instance 19

Install or upgrade license

If you are performing a new installation of Splunk or switching from one license type to another, you must install or update your license. Uninstall Splunk

To uninstall Splunk, use the Add or Remove Programs option in the Control Panel. You can also use msiexec from the commandline.

Install on Linux Install on Linux

You can install Splunk on Linux using RPM or DEB packages, or a tarball. RedHat RPM install

To install the Splunk RPM in the default directory /opt/splunk: rpm -i splunk_package_name.rpm

To install Splunk in a different directory, use the --prefix flag: rpm -i --prefix=/opt/new_directory splunk_package_name.rpm

To upgrade an existing Splunk installation using the RPM: rpm -U splunk_package_name.rpm

To upgrade an existing Splunk installation that was done in a different directory, use the --prefix flag: rpm -U --prefix=/opt/new_directory splunk_package_name.rpm

If you want to automate your RPM install with kickstart, add the following to your kickstart file: ./splunk start --accept-license ./splunk enable boot-start

Note: The second line is optional for the kickstart file. Debian DEB install

To install the Splunk DEB package: dpkg -i splunk_package_name.deb

Note: You can only install the Splunk DEB package in the default location, /opt/splunk.

20

Tarball install

To install Splunk on a Linux system, expand the tarball into an appropriate directory. The default install directory is /opt/splunk. When installing with the tarball: • Splunk does not create the splunk user automatically. If you want Splunk to run as a specific user, you must create the user manually. • Be sure the disk partition has enough space to hold the uncompressed volume of the data you plan to keep indexed. What gets installed

Splunk package status: dpkg --status splunk

List all packages: dpkg --list Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure that Splunk has the appropriate permissions to read the inputs that you specify. Refer to the instructions for running Splunk as a non-root user for more information. To start Splunk from the command line interface, run the following command from $SPLUNK_HOME/bin directory (where $SPLUNK_HOME is the directory into which you installed Splunk): ./splunk start

By convention, this document uses: • $SPLUNK_HOME to identify the path to your Splunk installation. • $SPLUNK_HOME/bin/ to indicate the location of the command line interface. Startup options

The first time you start Splunk after a new installation, you must accept the license agreement. To start Splunk and accept the license in one step: $SPLUNK_HOME/bin/splunk start --accept-license

Note: There are two dashes before the accept-license option.

21

Launch Splunk Web and log in

After you start Splunk and accept the license agreement, 1. In a browser window, access Splunk Web at http://:port. • hostname is the host machine. • port is the port you specified during the installation (the default port is 8000). 2. Splunk Web prompts you for login information (default, username admin and password changeme) before it launches. If you switch to Splunk Free, you will bypass this logon page in future sessions. Now that you've installed Splunk, what comes next? Uninstall Splunk

Use your local package management commands to uninstall Splunk. In most cases, files that were not originally installed by the package will be retained. These files include your configuration and index files which are under your installation directory. If you can't use package management commands, follow the instructions for manually uninstalling Splunk components. RedHat Linux

To uninstall from RedHat Linux rpm -e splunk_product_name Debian Linux

To uninstall from Debian Linux: dpkg -r splunk

To purge (delete everything, including configuration files): dpkg -P splunk

Install on Solaris Install on Solaris

This topic provides the procedures for installing Splunk on Solaris. Install Splunk

Splunk for Solaris is available as a PKG file or a tarball.

22

PKG file install

The PKG installation package includes a request file that prompts you to answer a few questions before Splunk installs. pkgadd -d ./splunk_product_name.pkg

A list of the available packages is displayed. • Select the packages you wish to process (the default is "all"). The installer then prompts you to specify a base installation directory. • To install into the default directory, /opt/splunk, leave this blank. PKG file upgrade

To upgrade an existing Splunk installation using a PKG file, use the same command line as you would for a fresh install. pkgadd -d

./splunk_product_name.pkg

You will be prompted to overwrite any changed files, answer yes to every one. To run the upgrade silently (and not have to answer yes for every file overwrite), type: pkgadd -n -d

./splunk_product_name.pkg

Tarball install

To install Splunk on a Solaris system, expand the tarball into an appropriate directory. By default, Splunk installs into /opt/splunk/. When installing with the tarball: • Splunk does not create the splunk user automatically. If you want Splunk to run as a specific user, you must create the user manually. • Be sure the disk partition has enough space to hold the uncompressed volume of the data you plan to keep indexed. What gets installed

Splunk package info: pkginfo -l splunk

List all packages: pkginfo

23

Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure that Splunk has the appropriate permissions to read the inputs that you specify. For more information, refer to the instructions on running Splunk as a non-root user. To start Splunk from the command line interface, run the following command from $SPLUNK_HOME/bin directory (where $SPLUNK_HOME is the directory into which you installed Splunk): ./splunk start

By convention, Splunk's documentation uses: • $SPLUNK_HOME to identify the path to your Splunk installation. • $SPLUNK_HOME/bin/ to indicate the location of the command line interface. Startup options

The first time you start Splunk after a new installation, you must accept the license agreement. To start Splunk and accept the license in one step: $SPLUNK_HOME/bin/splunk start --accept-license

Note: There are two dashes before the accept-license option. Launch Splunk Web and log in

After you start Splunk and accept the license agreement, 1. In a browser window, access Splunk Web at http://mysplunkhost:port, where: • mysplunkhost is the host machine. • port is the port you specified during the installation (8000). 2. Splunk Web prompts you for login information (default, username admin and password changeme) before it launches. If you switch to Splunk Free, you will bypass this logon page in future sessions. Now that you've installed Splunk, what comes next? Uninstall Splunk

Use your local package management commands to uninstall Splunk. In most cases, files that were not originally installed by the package are retained. These files include your configuration and index files which are under your installation directory. pkgrm splunk

24

Install on Mac OS Install on Mac OS

The Mac OS build comes in two forms: a DMG package and a tarball. Below are instructions for the: • Graphical (basic) and command line installs using the DMG file. • Tarball install. Graphical install

1. Double-click on the DMG file. A Finder window containing splunk.pkg opens. 2. In the FInder window, double-click on splunk.pkg. The Splunk installer opens and displays the Introduction, which lists version and copyright information. 3. Click Continue. The Select a Destination window opens. 4. Choose a location to install Splunk. • To install in the default directory, /Applications/splunk, click on the harddrive icon. • To select a different location, click Choose Folder... 5. Click Continue. The pre-installation summary displays. If you need to make changes, • Click Change Install Location to choose a new folder, or • Click Back to go back a step. 6. Click Install. Your installation will begin. It may take a few minutes. 7. When your install completes, click Finish. Command line install

1. To mount the dmg: hdid splunk_package_name.dmg

2. To Install

25

• To the root volume: installer -pkg splunk.pkg -target /

• To a different disk of partition: installer -pkg splunk.pkg -target /Volumes\ Disk

-target specifies a target volume, such as another disk, where Splunk will be installed in /Applications/splunk. To install into a directory other than /Applications/splunk on any volume, use the graphical installer as described above. Tarball install

To install Splunk on a Mac OS, expand the tarball into an appropriate directory. The default install directory is /Applications/splunk. When installing with the tarball: • Splunk does not create the splunk user automatically. If you want Splunk to run as a specific user, you must create the user manually. • Be sure the disk partition has enough space to hold the uncompressed volume of the data you plan to keep indexed. Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure that Splunk has the appropriate permissions to read the inputs that you specify. To start Splunk from the command line interface, run the following command from $SPLUNK_HOME/bin directory (where $SPLUNK_HOME is the directory into which you installed Splunk): ./splunk start

By convention, this document uses: • $SPLUNK_HOME to identify the path to your Splunk installation. • $SPLUNK_HOME/bin/ to indicate the location of the command line interface. Startup options

The first time you start Splunk after a new installation, you must accept the license agreement. To start Splunk and accept the license in one step: $SPLUNK_HOME/bin/splunk start --accept-license

Note: There are two dashes before the accept-license option.

26

Launch Splunk Web and log in

After you start Splunk and accept the license agreement, 1. In a browser window, access Splunk Web at http://:port. • hostname is the host machine. • port is the port you specified during the installation (the default port is 8000). 2. Splunk Web prompts you for login information (default, username admin and password changeme) before it launches. If you switch to Splunk Free, you will bypass this logon page in future sessions. Now that you've installed Splunk, what comes next? Manage your license

If you are performing a new installation of Splunk or switching from one license type to another, you must install or update your license. Uninstall Splunk

Use your local package management commands to uninstall Splunk. In most cases, files that were not originally installed by the package will be retained. These files include your configuration and index files which are under your installation directory. You can also simply go to $SPLUNK_HOME/bin, type ./splunk stop on the commandline and then delete the $SPLUNK_HOME directory and everything under it.

Install on FreeBSD Install on FreeBSD

The FreeBSD builds comes in two forms: an installer (5.4-intel) and a tarball (i386). Both are TGZ files. Basic install

To install FreeBSD using the intel installer: pkg_add splunk_package_name-5.4-intel.tgz

This installs Splunk in the default directory, /opt/splunk/ To install Splunk in a different directory: pkg_add -v -p /usr/splunk splunk_package_name-5.4-intel.tgz

27

Tarball install

To install Splunk on a FreeBSD system, expand the tarball into an appropriate directory. The default install directory is /opt/splunk. When installing with the tarball: • Splunk does not create the splunk user automatically. If you want Splunk to run as a specific user, you must create the user manually. • Be sure the disk partition has enough space to hold the uncompressed volume of the data you plan to keep indexed. After you install

To ensure that Splunk functions properly on FreeBSD, you must: 1. Add the following to /boot/loader.conf kern.maxdsiz="2147483648" # 2GB kern.dfldsiz="2147483648" # 2GB machdep.hlt_cpus=0

2. Add the following to /etc/sysctl.conf: vm.max_proc_mmap=2147483647

A restart of the OS is required for the changes to effect. What gets installed

To see the list of Splunk packages: pkg_info -L splunk

To list all packages: pkg_info Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure that Splunk has the appropriate permissions to read the inputs that you specify. To start Splunk from the command line interface, run the following command from $SPLUNK_HOME/bin directory (where $SPLUNK_HOME is the directory into which you installed Splunk): ./splunk start

By convention, this document uses:

28

• $SPLUNK_HOME to identify the path to your Splunk installation. • $SPLUNK_HOME/bin/ to indicate the location of the command line interface. Startup options

The first time you start Splunk after a new installation, you must accept the license agreement. To start Splunk and accept the license in one step: $SPLUNK_HOME/bin/splunk start --accept-license

Note: There are two dashes before the accept-license option. Launch Splunk Web and log in

After you start Splunk and accept the license agreement, 1. In a browser window, access Splunk Web at http://:port. • hostname is the host machine. • port is the port you specified during the installation (the default port is 8000). 2. Splunk Web prompts you for login information (default, username admin and password changeme) before it launches. If you switch to Splunk Free, you will bypass this logon page in future sessions. Now that you've installed Splunk, what comes next? Manage your license

If you are performing a new installation of Splunk or switching from one license type to another, you must install or update your license. Uninstall Splunk

Use your local package management commands to uninstall Splunk. In most cases, files that were not originally installed by the package will be retained. These files include your configuration and index files which are under your installation directory. To uninstall Splunk from the default location: pkg_delete splunk

To uninstall Splunk from a different location: pkg_delete -p /usr/splunk splunk

Install on AIX

29

Install on AIX

This topic will guide you through installing Splunk on the AIX platform. Note: If you are upgrading, review the upgrade documentation later in this manual and check the migration documentation for any migation considerations before proceeding. Install Splunk

The AIX install comes in tarball form. When installing with the tarball: • Splunk does not create the splunk user automatically. If you want Splunk to run as a specific user, you must create the user manually. • Be sure the disk partition has enough space to hold the uncompressed volume of the data you plan to keep indexed. To install Splunk on an AIX system, expand the tarball into an appropriate directory. The default install directory is /opt/splunk. For AIX 5.3, check to make sure your service packs are up to date. Splunk requires the following service level: $ oslevel -r 5300-005 Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure that Splunk has the appropriate permissions to read the inputs that you specify. Refer to the instructions for running Splunk as a non-root user for more information. To start Splunk from the command line interface, run the following command from $SPLUNK_HOME/bin directory (where $SPLUNK_HOME is the directory into which you installed Splunk): ./splunk start

By convention, this document uses: • $SPLUNK_HOME to identify the path to your Splunk installation. • $SPLUNK_HOME/bin/ to indicate the location of the command line interface. Note: The AIX version of Splunk does not register itself to auto-start on reboot. Startup options

The first time you start Splunk after a new installation, you must accept the license agreement. To start Splunk and accept the license in one step:

30

$SPLUNK_HOME/bin/splunk start --accept-license

Note: There are two dashes before the accept-license option. For more information, refer to "Splunk startup options" in this manual. Launch Splunk Web and log in

After you start Splunk and accept the license agreement, 1. In a browser window, access Splunk Web at http://:port. • hostname is the host machine. • port is the port you specified during the installation (the default port is 8000). 2. Splunk Web prompts you for login information (default, username admin and password changeme) before it launches. If you switch to Splunk Free, you will bypass this logon page in future sessions. Now that you've installed Splunk, what comes next? Manage your license

If you are performing a new installation of Splunk or switching from one license type to another, you must update your license. Uninstall Splunk

Use your local package management commands to uninstall Splunk. In most cases, files that were not originally installed by the package will be retained. These files include your configuration and index files which are under your installation directory.

Install on HP-UX Install on HP-UX

To install Splunk on an HP-UX system, expand the tarball into an appropriate directory. The default install directory is /opt/splunk. When installing with the tarball: • Splunk does not create the splunk user automatically. If you want Splunk to run as a specific user, you must create the user manually. • Be sure the disk partition has enough space to hold the uncompressed volume of the data you plan to keep indexed. Start Splunk

Splunk can run as any user on the local system. If you run Splunk as a non-root user, make sure that Splunk has the appropriate permissions to read the inputs that you specify. 31

To start Splunk from the command line interface, run the following command from $SPLUNK_HOME/bin directory (where $SPLUNK_HOME is the directory into which you installed Splunk): ./splunk start

By convention, this document uses: • $SPLUNK_HOME to identify the path to your Splunk installation. • $SPLUNK_HOME/bin/ to indicate the location of the command line interface. Note: The HP-UX version of Splunk does not register itself to auto-start on reboot. Startup options

The first time you start Splunk after a new installation, you must accept the license agreement. To start Splunk and accept the license in one step: $SPLUNK_HOME/bin/splunk start --accept-license

Note: There are two dashes before the accept-license option. Launch Splunk Web and log in

After you start Splunk and accept the license agreement, 1. In a browser window, access Splunk Web at http://:port. • hostname is the host machine. • port is the port you specified during the installation (the default port is 8000). 2. Splunk Web prompts you for login information (default, username admin and password changeme) before it launches. If you switch to Splunk Free, you will bypass this logon page in future sessions. Now that you've installed Splunk, what comes next? Manage your license

If you are performing a new installation of Splunk or switching from one license type to another, you must install or update your license.

Install a license Install a license

Each instance of Splunk server must have its own license. This topic discusses how to install or update a license, and what to do when you have a violation on your license.

32

For more details about license types and policies, refer to "About Splunk licenses" earlier in this manual. Note: You must purchase a separate license for every instance of Splunk with an Enterprise license that you deploy. Install your license

All Splunk servers have a license located in $SPLUNK_HOME/etc/, whether it is a Free license (splunk-free.license) or an Enterprise license (splunk.license). You can install and update your licenses with the CLI or using Splunk Web. Install a license using Splunk Web

1. Log into Splunk Web as the admin user. 2. Click Manager>License. 3. Click Change License. 4. Paste in your license key and click Save. 5. Return to the main Manager tab and click Restart Splunk. Note - Sometimes, you may get an 'Invalid License' error when you apply a license via Splunk Web. This is usually caused by hidden characters getting picked up and added to the license string when using copy&paste. To resolve this, you can update the new splunk.license file manually in $SPLUNK_HOME/etc. Make sure you use a plain text editor to do this. Pre-seeding your license before first time run

Starting with 4.0.2, by default when you start Splunk for the first time, it moves aside any existing 3.x license and replaces it with a temporary Enterprise trial license. This allows you to bring up the new version of Splunk without having your license be expired until you get your new one copied in. If you are migrating to Splunk 4.0.2 or later and have a valid 4.x license, you can pre-seed the license file so that it pulls in and installs your new license the first time you start Splunk 4. This is useful if you have to deploy multiple instances and don't want to have to manually copy the new license in after starting Splunk on each machine. • Migrate to 4.0.2 or later, but don't start Splunk yet. • Copy your new license into $SPLUNK_HOME/etc/splunk-user.license. • Start Splunk. If you're making a deployable package, you can include the splunk-user.license file with your updated license in it before you tar/zip it up for deployment to other systems.

33

License violations

Violations occur when you exceed the maximum indexing volume allowed for your license. If you exceed your licensed daily volume on any one calendar day, you will get a violation warning. The message persists for 14 days. If you have more than 5 violations in a rolling 30-day period, search will be disabled. Search capabilities return when you have fewer than 5 violations in the previous 30 days or when you apply a new license with a larger volume limit. Got license violations? Click here to troubleshoot Note: During a license violation period, Splunk does not stop indexing your data. Splunk only blocks search access while you exceed the allowed number of license violations.

34

Start Splunk for the first time Start Splunk for the first time Start Splunk for the first time

To start Splunk: On Windows You can start Splunk on Windows using either the commandline, or the Windows Services Manager. Using the commandline offers more options, described later in this section. In a cmd window, go to C:\Program Files\Splunk\bin and type: splunk start

(For Windows users: in the subsequent examples and information, replace $SPLUNK_HOME with C:\Program Files\ if you have installed Splunk in the default location.) On UNIX Use the Splunk command line interface (CLI): $SPLUNK_HOME/bin/splunk start

Splunk then displays the license agreement and prompts you to accept the before the startup sequence continues. Other start options

To automatically accept the license when you start Splunk for the first time, add the accept-license option to the start command: $SPLUNK_HOME/bin/splunk start --accept-license

The startup sequence displays:

Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Verifying configuration. This may take a while... Finished verifying configuration. Checking index directory... Verifying databases... Verified databases: _audit, _blocksignature, _internal, _thefishbucket, history, main, sampleda Checking index files All index checks passed. All preliminary checks passed. Starting splunkd... Starting splunkweb... Splunk Server started. The Splunk web interface is at http://:8000 If you get stuck, we're here to help. Feel free to email us at '[email protected]'.

Note: If the default ports are already in use (or are otherwise not available), Splunk will offer to use the next available port. You can either accept this option or specify a port for Splunk to use. 35

There are two other start options: no-prompt and answer-yes: • If you run $SPLUNK_HOME/bin/splunk start --no-prompt, Splunk proceeds with startup until it requires you to answer a question. Then, it displays the question, why it is quitting, and quits. • If you run SPLUNK_HOME/bin/splunk start --answer-yes, Splunk proceeds with startup and automatically answers "yes" to all yes/no questions. Splunk displays the question and answer as it continues. If you run start with all three options in one line, for example: $SPLUNK_HOME/bin/splunk start --answer-yes --no-prompt --accept-license

• Splunk does not ask you to accept the license. • Splunk answers yes to any yes/no question. • Splunk quits when it encounters a non-yes/no question. Start and disable individual processes

You can start and stop individual Splunk processes by adding the process as an object to the start command. The objects include: • splunkd, the Splunk server daemon. • splunkweb, Splunk's Web interface process. For example, to start only splunkd: $SPLUNK_HOME/bin/splunk start splunkd

To disable splunkweb: $SPLUNK_HOME/bin/splunk disable webserver

For more information about start, refer to the CLI help page: $SPLUNK_HOME/bin/splunk help start Launch Splunk Web

Navigate to: http://mysplunkhost:8000 Use whatever host and port you chose during installation. The first time you log in to Splunk Enterprise, the default login details are: Username - admin. Password - changeme. Splunk Free does not have access controls. 36

Migrate from 3.4.x or earlier What to expect when migrating to 4.0 What to expect when migrating to 4.0

This topic discusses the various issues and considerations you should review before migrating to 4.0 from 3.4.x or earlier. You'll find information about aspects of your deployment that cannot be migrated automatically, some changes you will need to make manually, and other changes that the migration script will handle. Note: If you're migrating from versions earlier than 3.4.x, please follow the documentation for that version to migrate to 3.4.x and then proceed from there. Migration to 4.x is only supported from 3.4.x. Before you proceed with migration, you should also review the Known Issues for additional information. Share your migration experiences

Did you migrate an existing Splunk installation to 4.x and run into some issues? What led to your success? Share your tips with other users on the Splunk Community Wiki "Migration experiences" page. Considerations and support for users of Splunk 3.4.x

Splunk 4 is a huge stride forward in performance and flexibility, but there are a few interaction changes vs. 3.4.x which upgraders should be aware of, and even some reasons why you might want to wait for a future release before upgrading. Below are some capabilities that have changed with the introduction of Splunk 4: Live tail • With Splunk 4's dramatically improved search and indexing speed, along with the ability to provide intermediate search results, you don't really need a separate live event console to see data in near real-time. However, if your use case relies on version 3.4.x's "Live tail" feature, you may want to wait on upgrading to Splunk 4. Future roadmap plans involve re-architecting the live tail functionality to scale across much larger data flows, and across distributed environments. Additionally, look out for improve real-time alerting and dashboard updates down the road as a result of these upcoming architectural changes. Custom field actions • Based on customer feedback, we decided to re-architect this feature to improve flexibility and allow for event actions based on multiple fields. Expect this functionality to be reintroduced in a near term 4.x release. If you rely on this functionality, but still want to upgrade, you may want to consider Splunk 4's new "Dynamic field lookups" as an alternative which allows you to map data from external databases and lists into Splunk. Snapshots 37

• In Splunk 4, we've improved upon 3.x's ability to take a timeline snapshots of individual searches. Try out Splunk 4's new job manager which allows you to retrieve the entire cached search result, including reports, from existing searches. Event scrolling • In Splunk 4, the new page selector allows you to hop between results with greater flexibility, even as a search runs. However, for those who still prefer a scroll bar, expect this capability to be re-introduced as an option in a future 4.x release. Timeline and timestamp interaction • In Splunk 4, we improved the timeline to allow users to quickly view any time range within search results, without having to rerun a search. Also try clicking "zoom-in" on the timeline, which now allows you to lock-in a time range, and specify follow on search. • We're also planning to improve the usability of some related 3.4.x functionality including clicking on timestamps, and double clicking on timeline bars in future versions of 4.x. Crawl • Crawl is no longer configurable via Splunk Web, but is still available as a search command. Based on customer feedback, we have decided to re-architect this feature to make it easier and more effective. Expect improved functionality, along with a new user interface to be introduced in a future release. FIFO inputs • This input type has been deprecated with Splunk 4, and we do not recommend using it as a best practice due to data loss considerations. If you currently rely on this input type consider writing the output to a flat file that Splunk is monitoring. Persistent Queue • This feature has been deprecated with Splunk 4. When a forwarder loses its connection to the indexer, it will block instead of continuously writing events to disk. If you are concerned about data loss, consider using TCP or file inputs. RSS alerts • RSS alerts are not supported for the initial realease of the 4.x line. Let us know when you're ready to migrate, and we'll be here to help. Migrating your license

Splunk 4.x does not work with licenses from older releases. • If you're migrating to 4.0 or 4.0.1, AFTER YOU INSTALL, YOU WILL SEE A MESSAGE STATING THAT YOUR LICENSE IS NO LONGER VALID. 38

• If you are an current Enterprise customer, check your splunk.com orders page for an updated license. • If you're migrating to 4.0.2, your 3.x license is backed up and replaced with an Enterprise trial license when you start Splunk 4. • If you are running with a 3.x Free or Enterprise trial license, delete the $SPLUNK_HOME/etc/splunk.license file before you start Splunk 4.x. The instance will then pick up the 60-day Enterprise trial license. Pre-seeding your license before first time run

Starting with 4.0.2, by default when you start Splunk for the first time, it moves aside any existing 3.x license and replaces it with a temporary Enterprise trial license. This allows you to bring up the new version of Splunk without having your license be expired until you get your new one copied in. If you are migrating to Splunk 4.0.2 or later and have a valid 4.x license, you can pre-seed the license file so that it pulls in and installs your new license the first time you start Splunk 4. This is useful if you have to deploy multiple instances and don't want to have to manually copy the new license in after starting Splunk on each machine. • Migrate to 4.0.2 or later, but don't start Splunk yet. • Copy your new license into $SPLUNK_HOME/etc/splunk-user.license. • Start Splunk. If you're making a deployable package, you can include the splunk-user.license file with your updated license in it before you tar/zip it up for deployment to other systems. Which migration is right for you?

If you have not customized your 3.x deployment extensively, you may be able to use the automatic migration process described in either Migrate on Windows or Migrate on UNIX. If you have customized your 3.x deployment to any degree, (in particular, if you have customized deployment.conf), you may have to consider migrating your deployment manually. In this case, follow the Steps for manual migration to Splunk 4.x. What cannot be migrated

Splunk 4.x is significantly different from earlier versions. Some configuration files from your 3.x deployment can be copied over to your new 4.x install without any migration required. However, some aspects of your existing deployment cannot be migrated, and must be rebuilt; this is most relevant for 3.x deployments and configurations that have been extensively customized. Splunk Web display customizations

All things to do with Splunk Web have been completely re-architected and rebuilt from the ground up. As a result, any customizations you've made that affect the display of Splunk Web cannot be migrated; you must rebuild them using the 4.x architecture and tools. These include: • Form searches • Field actions (field_actions.conf) 39

• Dashboards • UI preferences (prefs.conf) • Report chart and table preferences on saved searches • Changes to UI strings (literals.conf) Considerations for apps

In general, apps do not get migrated; you should re-architect your apps to follow the new 4.0 architecture. For more information about apps in 4.0, refer to the Developer Manual. For more guidance on migrating your apps, consult this topic on migrating your 3.x apps in the Developer Manual. Not all Splunk 3.4.x apps will be immediately available for 4.0. New apps will be added to the Splunk App Store as they become available. Splunk for Change Management has not yet been made available for Splunk 4, If you have this app, contact Support before upgrading, or consider installing a parallel instance of Splunk to maintain it. Considerations for deployment server and forwarders

4.x deployment server is not backwards compatible with 3.x deployment clients. If you use the Splunk deployment server, you must rebuild your deployment configuration using the new deployment server architecture in 4.x; it cannot be migrated. If you've made changes to deployment.conf in version 3.x, you should not copy it over into your new Splunk 4.x installation. Set it aside to use as a basis for building your new deployment server configuration. In the meantime, if you have 3.4.x deployment server and clients and do not want to migrate all the clients at this time, you can use the instructions in this topic to configure a stripped-down 3.4.x deployment server to continue managing your deployment clients. If you have 3.4.x forwarders, you can delay migrating them to 4.x; 3.4.x forwarders will work with a 4.x version of Splunk. This is especially useful for large forwarder deployments. You can wait until you are comfortable with your new deployment server configuration before migrating forwarders to 4.x. In summary: • 3.x forwarders will work with a 4.x indexer. • 3.x deployment clients will not work with 4.x deployment server. Considerations for distributed search

4.0.x distributed search is not backwards compatible with 3.x distributed search. In addition, the deployment server classes are not directly applicable to 4.x deployment server configuration. 4.x deployment servers use the concept of "apps" to push configuration to clients, as opposed to classes. Admins must build an application from their classes configuration to push to clients. It is recommended to do this first, before losing configuration parity at upgrade time, for clients running 4.x. 40

If you use distributed search, all your participating distributed search nodes will need to run Splunk 4.x in order to successfully communicate and return results. Using a 4.0 or 4.0.1 Splunk deployment derver with 3.x deployment clients will crash the clients

This issue was resolved in 4.0.2. To ensure your 4.x server doesn't contact your existing 3.x deployment clients, change the management port on the 4.x instance. Add the following to $SPLUNK_HOME/etc/system/local/web.conf to change it from the default 8089 port to 8090: [settings] mgmtHostPort = 127.0.0.1:8090 What you must migrate manually

Splunk 4.x differs significantly from version 3.x, and the migration script cannot convert the content of some configuration files. This section describes some of the changes that you must make manually. Considerations for scheduled searches and alerts

Scheduled searches and alerts do not migrate automatically. In the 4.0 knowledge model, a search cannot execute outside of an app context. Thus, saved searches are considered part of the Search app; they are not global and viewable across all apps. To migrate your saved searches: 1. Stop Splunk; execute the command: $SPLUNK_HOME/bin/splunk stop 2. Move $SPLUNK_HOME/etc/system/local/savedsearches.conf to $SPLUNK_HOME/etc/apps/search/local/ 3. Move any stanzas whose name starts with "savedsearches/" from $SPLUNK_HOME/etc/system/metadata/local.meta to $SPLUNK_HOME/etc/apps/search/metadata/local.meta 4. Start Splunk; execute the command: $SPLUNK_HOME/bin/splunk start

Note: Scheduled searches will not run unless these 2 steps are completed. Considerations for tags and aliases

Sourcetype renaming replaces "sourcetype aliasing" and isn't implemented by tags. For more information, see "About tags and aliases" in the Knowledge Manager manual. Saved searches that rely on aliased sourcetypes won't work without migration. For steps to migrate these saved searches, see migration concerns for scheduled searches and alerts. 41

Considerations for crawl.conf

The crawl.conf stanza [file_crawler] should be renamed [files]. This does not happen automatically. No migration concerns for CLI

The command line interface, CLI, has been completely rewritten for Splunk 4.x. There are no known migration or backwards compatibility issues. Refer to release notes for the list of changes to the CLI, which include deprecated CLI command options and new functionality. Changes made during automatic migration

During migration, many configuration files cannot simply be copied over. Splunk provides scripts for converting and migrating the content of these files so that they will function correctly in 4.x. You can run the migration preview utility to see what will be changed before you actually upgrade and migrate. The following is a list of some of the changes that take place during migration. alert_actions.conf

This configuration file is migrated with savedsearches.conf. indexes.conf

During migration, some attributes are added to indexes.conf while other local attributes are either removed or changed to global parameters. For more details about these parameters, refer to indexes.conf in the Admin manual. The following attributes are no longer supported and have no effect: • _actions • maxTermChars • maxTerms • maxPostings • maxValues • waitForOptimize The following global attributes are added with these defaults: • indexThreads = auto • maxMemMB = 5 • memPoolMB = auto • maxHotSpanSecs = 7776000 • maxHotIdleSecs = 0 • maxHotBuckets = 1 • quarantinePastSecs = 77760000 • quarantineFutureSecs = 2592000 For high-volume indexes (such as main), the following defaults are used: • maxHotBuckets = 10 42

• maxHotIdleSecs = 86400 • maxMemMB = 20 savedsearches.conf

During migration, form searches are disabled, a default global view state is set, and owner attributes are moved so that they follow the 4.0 roles and permissions model. • if the search attribute contains macros ($$), disabled = 1 is added to the stanza. • if the attribute defines a view state &mdash, which is any line that begins with "viewstate.", the line is commented out. • if the stanza contains 'userid' and/or 'role' attributes, a corresponding metadata owner/ACL is added. server.conf

During migration, the following attributes are moved from splunkd.xml to server.conf: • minFreeMb • pollingFrequency • serverName The following attributes, for the server certificate and key file and password, are renamed: • keysfile is replaced with sslkeysfile • keysfilepassword is replaced with sslkeysfilePassword Windows specific conf files

During migration, Windows-specific files (regmon-filters.conf, sysmon.conf, and wmi.conf) and knowledge objects are moved out of default and into the Windows app architecture. Conf files that are removed

During migration, the following conf files are removed: • typedefs.conf • searchdata.conf Index data compatibility

Data indexed by version 3.x is completely compatible with 4.x and does not need to be re-indexed. However, note that it may have been indexed in a less efficient form, so searches performed over data that was originally indexed and migrated from 3.x may not be as fast as searches over newly-indexed 4.x data.

Migrate on UNIX

43

Migrate on UNIX

When you migrate to 4.0, your configuration files will be updated and changed. You can run the migration preview utility to see what will be changed before you actually upgrade and migrate. When you do this, a file containing the changes that the script proposes to make is written to: $SPLUNK_HOME/var/log/splunk/migration.log. Before you migrate

Before you perform the migration, we strongly recommend that you review these migration considerations and back up all of your files, including Splunk configurations, data and binaries. Splunk does not provide a means of downgrading to previous versions; if you need to revert to an older Splunk release, just reinstall it. Migration instructions

1. Execute the $SPLUNK_HOME/bin/splunk stop command. 2. Disable any 'coldToFrozen' scripts in $SPLUNK_HOME/etc/system/local/indexes.conf. If you leave them in place they could potentially run when you start 4.0 for the first time and cause your instance to appear unresponsive. 3. Install the Splunk package over your existing Splunk deployment. If you are using a TAR file, expand it into the same directory as your existing Splunk instance. This overwrites and replaces matching files but does not remove unique files. If you are using a package manager, such as an RPM: rpm -U splunk_package_name.rpm

4. Execute the $SPLUNK_HOME/bin/splunk start command. The following output is displayed: This appears to be an upgrade of Splunk. -------------------------------------------------------------------------------Splunk has detected an older version of Splunk installed on this machine. To finish upgrading to the new version, Splunk's installer will automatically update and alter your current configuration files. Deprecated configuration files will be renamed with a .deprecated extension. You can choose to preview the changes that will be made to your configuration files before proceeding with the migration and upgrade: If you want to migrate and upgrade without previewing the changes that will be made to your existing configuration files, choose 'y'. If you want to see what changes will be made before you proceed with the upgrade, choose 'n'.

44

Perform migration and upgrade without previewing configuration changes? [y/n]

4. You're given the choice of running the migration preview script to see what changes will be made to your existing configuration files, or proceeding with the migration and upgrade right away. 5. If you choose to view the expected changes, the script provides a list. 6. Once you've reviewed these changes and are ready to proceed with migration and upgrade, run $SPLUNK_HOME/bin/splunk start again. Note: You can complete Steps 3 to 5 in one line: To accept the license and view the expected changes (answer 'n') before continuing the upgrade: $SPLUNK_HOME/bin/splunk start --accept-license --answer-no

To accept the license and begin the upgrade without viewing the changes (answer 'y'): $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes

Migrate on Windows Migrate on Windows

When you migrate, your configuration files are updated and changed to support the new functionality. You can run the migration preview utility to see what will be changed before you actually upgrade and migrate. When you do this, a file containing the changes that the script proposes to make is written to: $SPLUNK_HOME/var/log/splunk/migration.log. Before you migrate

• Review the migration concerns in "What to expect when upgrading to 4.0". • Back up your files, including Splunk configurations, data and binaries. • Stop Splunk either using the Windows Start menu option or by executing the $SPLUNK_HOME\bin\splunk stop command. • Be aware that you cannot change the user Splunk runs as during an upgrade. Do not change the user from the Windows Service Control panel; Splunk will stop working. If you must change the user, you must uninstall and reinstall Splunk. • The Windows App was enabled by default in versions 4.0-4.0.2. Starting in version 4.0.3, it is disabled by default unless you enable it explicitly via the commandline/MSI installation process. • Disable any 'coldToFrozen' scripts in $SPLUNK_HOME\etc\system\local\indexes.conf. If you leave them in place they could potentially run when you start 4.0 for the first time and cause your instance to appear unresponsive.

45

Upgrade instructions

1. Download the new MSI file from the Splunk download page. 2. Double-click the MSI file. The Welcome panel is displayed. Follow the onscreen instructions to upgrade Splunk. For information about each panel, refer to the installation instructions. When you reach the Install step, you have the option to preview changes that will be made for this upgrade. 3. Preview your upgrade and migration if desired. The following text is displayed: This appears to be an upgrade of Splunk. -------------------------------------------------------------------------------Splunk has detected an older version of Splunk installed on this machine. To finish upgrading to the new version, Splunk's installer will automatically update and alter your current configuration files. Deprecated configuration files will be renamed with a .deprecated extension. You can choose to preview the changes that will be made to your configuration files before proceeding with the migration and upgrade: If you want to migrate and upgrade without previewing the changes that will be made to your existing configuration files, choose 'y'. If you want to see what changes will be made before you proceed with the upgrade, choose 'n'. Perform migration and upgrade without previewing configuration changes? [y/n]

4. You're given the choice of running the migration preview script to see what changes will be made to your existing configuration files, or proceeding with the migration and upgrade right away. 5. If you choose to view the expected changes (select N), the script provides a list. You can scroll up to review the changes or look at them in $SPLUNK_HOME\var\log\splunk\migration.log.. At the end of the list, you will see an error message, which you can ignore. 6. Press Enter to return to step 3 and finish your upgrade by typing Y. Important: When migrating or upgrading, you must re-specify the user you want Splunk to run as--this information is not automatically maintained from release to release.

46

Start Splunk

On Windows, Splunk is installed by default into \Program Files\Splunk You can start and stop the following Splunk processes via the Windows Services Manager: • Server daemon: splunkd • Web interface: splunkweb You can also start, stop, and restart both processes at once by going to \Program Files\Splunk\bin and typing #

splunk.exe [start|stop|restart]

Note: If you do not select Start Splunk Services now, they will be set to manual startup and therefore will not start after a reboot. You must start them from the Windows Service Manager MMC, and optionally configure auto-start if you want them to start automatically at boot time. Important: After upgrading, Splunk may start reading some files incorrectly as binaries. You can override this behavior in props.conf by adding the following line to a source stanza: NO_BINARY_CHECK = true

Steps for manual migration to Splunk 4.x Steps for manual migration to Splunk 4.x

This topic describes the procedure for migrating a 3.x Splunk installation to version 4.x manually. Before you proceed, be sure to review "What to expect when migrating to 4.0" in this manual. If you have not customized your 3.x deployment extensively, you may be able to use the automatic "in-place" migration process described in the Migrate on Windows or Migrate on UNIX topics. In particular, if you have not customized deployment.conf, you can likely use the automatic migration process. Some custom configurations cannot be migrated at all, and must be rebuilt within the new Splunk framework. Refer to the section titled "What cannot be migrated" in this topic and "What to expect when migrating to 4.0" in this manual for more details. Share your migration experiences

Did you migrate an existing Splunk installation to 4.x and run into some issues? What led to your success? Share your tips with other users on the Splunk Community Wiki "Migration experiences" page. Overview of the manual migration process

During the manual procedure, you will make a backup copy of your existing deployment, install Splunk 4.x in a separate path from 3.x, and then migrate your data and configuration files across from your 3.x deployment. This procedure is designed to allow for a minimum of downtime.

47

It's important to follow this procedure carefully--Splunk 4.x differs significantly from version 3.x, and many configuration files cannot simply be copied over. Splunk provides scripts for converting and migrating the content of these files so that they will function correctly in 4.x. Read through this entire topic before beginning the procedure. This way, you will know what to expect, and can plan accordingly. What cannot be migrated

As mentioned earlier, Splunk 4.x is significantly different from earlier versions. Some aspects of your existing deployment cannot be migrated, and must be rebuilt. Splunk Web display customizations

All things to do with Splunk Web have been completely re-architected and rebuilt from the ground up. As a result, any customizations you've made that affect the display of Splunk Web cannot be migrated; you must rebuild them using the 4.x architecture and tools. These include: • Form searches • Field actions (field_actions.conf) • Dashboards - visit here for instructions on how to rebuild them • UI preferences (prefs.conf) • Report chart and table preferences on saved searches • Changes to UI strings (literals.conf) • 3.x Apps - read this Developer Manual topic for guidance on reworking them. Considerations for deployment server and forwarders

• If you use the Splunk deployment server, you must rebuild your deployment configuration using the new deployment server architecture in 4.x; it cannot be migrated. If you've made changes to deployment.conf in version 3.x, you should not copy it over into your new Splunk 4.x installation. Set it aside to use as a basis for building your new deployment server configuration. • If you have deployed forwarders, you can delay migrating them to 4.x; 3.x forwarders will work with a 4.x version of Splunk. This is especially useful for large forwarder deployments. You can wait until you are comfortable with your new deployment server configuration before migrating forwarders to 4.x. • Version 4.x forwarders do not work with version 3.x deployment server. Before beginning the migration

Before you begin migrating your Splunk data and configuration files, make a backup copy of your 3.x installation. The easiest way to do this is to run the Splunk CLI diag utility, which will create a compressed (.tar) file containing your entire installation. To do this, from $SPLUNK_HOME/bin type: On UNIX: ./splunk diag

On Windows: 48

splunk diag

If you have difficultly running diag in your environment, you can also run the python script directly using the cmd command: ./splunk cmd python /opt/splunk/lib/python2.5/site-packages/splunk/clilib/info_gather.py

This produces splunk-diag.tar.gz|zip that you can preserve as a backup of your 3.x deployment. Once you have done this, use the instructions in this manual to install version 4.0 for your platform in a different location from your existing 3.x install but do not start it yet. Do not start your 4.x Splunk instance yet. Import configuration files that do not require manual migration

Some of the configuration files from your 3.x deployment can be copied over to your new 4.x install without any migration required. Remember to copy these files from/to $SPLUNK_HOME/etc/system/local or from your custom config directory--do not copy over the files in /default! • Copy over all configuration files except deployment.conf. • Edit the copy of inputs.conf you put on your new 4.x Splunk instance to set all the inputs to disabled=true. • Copy over the etc/myinstall/splunkd.xml file Import user and password information

You can bring your user data, password, and other authentication information across at this point: • authentication.conf • authorize.conf • splunk.secret • passwd file Do not start your 4.x Splunk install yet. Migrate your Splunk index data

This section explains how to configure your new 4.x install to read your existing Splunk index data. Data indexed by version 3.x is completely compatible with 4.x and does not need to be re-indexed. However, note that it may have been indexed in a less efficient form, so searches performed over data that was originally indexed and migrated from 3.x may not be as fast as searches over newly-indexed 4.x data. Again, be sure to follow these instructions carefully to minimize downtime and data loss. Order is important! Additionally, be prepared to perform these steps in rapid succession. Once you roll the 'hot' index data directory to 'warm' on your existing Splunk deployment, you'll want to immediately 49

move on to the next steps as quickly as possible to ensure there is a minimum of downtime in the transition to indexing with your new 4.x Splunk install. To understand the implications of "hot" and "warm" index directories, review the "Best practices for backing up" topic in the Splunk Community Wiki. 1. In the root directory of your 4.x install, edit splunk-launch.conf to point the value of SPLUNK_DB to point to the index data directories of your 3.x deployment. By default, this value is /opt/splunk/var/lib/splunk, but you can check the splunk-launch.conf in your 3.x deployment to be sure you've got the right path. 2. Then, in your version 3.x deployment, use the CLI to force a roll of any data in your hot index data directory to the warm' directory. To do this, type: • ./splunk search '| oldsearch !++cmd++::roll' -auth admin: ♦ You'll see an error about "Search Execute failed because Hot db rolled out to warm" right afterward; you can safely ignore it. You'll also need to provide the admin password to execute this CLI command. ♦ To roll any other indexes you are currently writing to, use: ♦ ./splunk search ' | oldsearch index= !++cmd++::roll' -auth admin: 3. As soon as you've rolled all your indexes from hot to warm, immediately shut down your 3.x deployment and start your 4.x install up. When you start your 4.x, some of the configuration files you copied over may undergo an auto-migration process. Splunk will display information about this process when you start 4.x for the first time. 4. As quickly as you can, log into your new 4.x install and search for data you know is in your 3.x index, to validate all the migration steps. Ensure that your new deployment is able to search and find data in your 3.x indexes. If the validation fails, immediately restart your 3.x deployment before proceeding with any troubleshooting. 5. Once you have successfully validated that your 4.x deployment can search your 3.x index data, immediately edit inputs.conf and set your inputs to disabled=false and restart your 4.x Splunk deployment to enable them. Verify that data is coming into your new deployment. Now you're ready to begin using your new 4.x Splunk deployment! If you use distributed search

If you have configured distributed search, you must distribute new keys to all search peers to use distributed search in 4.x. These keys are generated at startup if distributed search is enabled. You can do this via the UI on the search head. For each of your search peers, re-enter the remote username/password through the Manager Distributed search pages.

Or you can do this manually. The keys will be generated in 50

$SPLUNK_HOME/etc/auth/distServerKeys/. Distribute the files $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem and private.pem from one host to any others distributed search peers.

51

Upgrade from 4.0 or later Upgrade Splunk on Windows Upgrade Splunk on Windows

This topic describes the procedure for upgrading your Windows Splunk instance from version 4.x to a later version. You can upgrade using the GUI installer, or by running msiexec on the commandline as described in "Install on Windows via the commandline". Note: The Windows App was enabled by default in its app.conf file in versions 4.0-4.0.2. Starting in version 4.0.3, it is disabled in this file by default. Read on for important details: • If you're upgrading from 4.0-4.0.2 to 4.0.3 or later, the Windows App will be disabled, even if it was enabled in the version you're upgrading from. • If you're doing a fresh installation of 4.0.3 or later, the Windows App is enabled by default via the MSI and if you want to install it in a disabled state, you must specify this using the SPLUNK_APP msiexec command as described in "Install on Windows via the commandline". Before you upgrade

Important: When upgrading, you must explicitly specify the same domain user that you specified during first time install. If you do not specify the same user, Splunk will default to using the Local System User. If you accidentally specify the wrong user during your installation, use the instructions in these instructions to switch to the correct user before starting Splunk. Important: Before you perform the upgrade, we strongly recommend that you back up all of your files, including Splunk configurations, data and binaries. Splunk does not provide a means of downgrading to previous versions; if you need to revert to an older Splunk release, just reinstall it. Upgrading using the GUI installer

1. Stop Splunk either using the Windows Start menu option or by executing the $SPLUNK_HOME/bin/splunk stop command. 2. Download the new MSI file from the Splunk download page. 3. Double-click the MSI file. The Welcome panel is displayed. Follow the onscreen instructions to upgrade Splunk. For information about each panel, refer to the installation instructions. 4. Splunk will start up by default when you complete the installation. A log of the changes made to your configuration files during the upgrade is placed in $TEMP$. Upgrading using the commandline

1. Stop Splunk either using the Windows Start menu option or by executing the $SPLUNK_HOME/bin/splunk stop command.

52

2. Download the new MSI file from the Splunk download page. 3. Use the instructions in "Install on Windows via the commandline". If Splunk is running as a user other than the Local System user, you must explicitly specify this user in your commandline. You can change the ports (SPLUNKD_PORT and WEB_PORT) at this time, and also use the LAUNCHSPLUNK option to specify whether Splunk should start up automatically or not when you're finished, but you cannot change any other settings. 4. Depending on your specification, Splunk may start automatically when you complete the installation. A log of the changes made to your configuration files during the upgrade is placed in $TEMP$. Start Splunk

On Windows, Splunk is installed by default into \Program Files\Splunk and is started by default. You can start and stop the following Splunk processes via the Windows Services Manager: • Server process: splunkd • Web interface process: splunkweb You can also start, stop, and restart both processes at once by going to \Program Files\Splunk\bin and typing #

splunk [start|stop|restart]

Upgrade Splunk on Linux, Solaris, FreeBSD, HP-UX, AIX, and MacOS Upgrade Splunk on Linux, Solaris, FreeBSD, HP-UX, AIX, and MacOS

This topic describes the procedure for upgrading your Splunk instance from version 4.x to a later version. How upgrading works

When you upgrade your configuration files are not actually changed until you start Splunk after performing the installation of the new version. You can run the migration preview utility at that time to see what will be changed before the files are updated. If you choose to view the changes before proceeding, a file containing the changes that the upgrade script proposes to make is written to $SPLUNK_HOME/var/log/splunk/migration.log. Important: Before you perform the upgrade, we strongly recommend that you back up all of your files, including Splunk configurations, data and binaries. Splunk does not provide a means of downgrading to previous versions; if you need to revert to an older Splunk release, just reinstall it.

53

Steps for upgrading

1. Execute the $SPLUNK_HOME/bin/splunk stop command. 2. To upgrade and migrate from version 4.0 and later, install the Splunk package over your existing Splunk deployment: • If you are using a .tar file, expand it into the same directory as your existing Splunk instance. This overwrites and replaces matching files but does not remove unique files. • If you are using a package manager, such as an RPM, type rpm -U splunk_package_name.rpm • If you are using a .dmg file (on MacOS), double-click it and follow the instructions. Be sure specify the the same installation directory as your existing installation. 3. Execute the $SPLUNK_HOME/bin/splunk start command. The following output is displayed: This appears to be an upgrade of Splunk. -------------------------------------------------------------------------------Splunk has detected an older version of Splunk installed on this machine. To finish upgrading to the new version, Splunk's installer will automatically update and alter your current configuration files. Deprecated configuration files will be renamed with a .deprecated extension. You can choose to preview the changes that will be made to your configuration files before proceeding with the migration and upgrade: If you want to migrate and upgrade without previewing the changes that will be made to your existing configuration files, choose 'y'. If you want to see what changes will be made before you proceed with the upgrade, choose 'n'. Perform migration and upgrade without previewing configuration changes? [y/n]

4. Choose whether you want to run the migration preview script to see what changes will be made to your existing configuration files, or proceed with the migration and upgrade right away. 5. If you choose to view the expected changes, the script provides a list. 6. Once you've reviewed these changes and are ready to proceed with migration and upgrade, run $SPLUNK_HOME/bin/splunk start again. Note: You can complete Steps 3 to 5 in one line: To accept the license and view the expected changes (answer 'n') before continuing the upgrade: $SPLUNK_HOME/bin/splunk start --accept-license --answer-no

To accept the license and begin the upgrade without viewing the changes (answer 'y'): $SPLUNK_HOME/bin/splunk start --accept-license --answer-yes

54

Other tasks Run Splunk as a different or non-root user Run Splunk as a different or non-root user

You can run Splunk as any user on the local system. If you run Splunk as a non-root user, make sure Splunk has the appropriate permissions to: • Read the files and directories it is configured to watch. Some log files and directories may require root or superuser access to be indexed. • Write to Splunk's directory and execute any scripts configured to work with your alerts or scripted input. • Bind to the network ports it is listening on (ports below 1024 are reserved ports that only root can bind to). Note: Because ports below 1024 are reserved for root access only, Splunk will only be able to listen on port 514 (the default listening port for syslog) if it is running as root. You can, however install another utility (such as syslog-ng) to write your syslog data to a file and have Splunk monitor that file instead. Instructions

To run Splunk as a non-root user, you need to first install Splunk as root. Then, before you start Splunk for the first time, change the ownership of the splunk directory to the desired user. The following are instructions to install Splunk and run it as a non-root user, splunk. 1. Create the user and group, splunk. For Linux, Solaris, and FreeBSD: useradd splunk groupadd splunk

For Mac OS: You can use the System Preferences > Accounts panel to add users and groups. 2. As root and using one of the packages (not a tarball), run the installation. Important: Do not start Splunk yet. 3. Use the chown command to change the ownership of the splunk directory and everything under it to the desired user. chown -R splunk $SPLUNK_HOME/

Note: $SPLUNK_HOME refers to installation directory of Splunk.

55

4. Start Splunk. $SPLUNK_HOME/bin/splunk start

Also, if you want to start Splunk as the splunk user while you are logged in as a different user, you can use the sudo command: sudo -H -u splunk $SPLUNK_HOME/bin/splunk start

This example command assumes: • If Splunk is installed in an alternate location, update the path in the command accordingly. • Your system may not have sudo installed. If this is the case, you can use su. • If you are installing using a tarball and want Splunk to run as a particular user (such as splunk), you must create that user manually. • The splunk user will need access to /dev/urandom to generate the certs for the product. Solaris 10 privileges

When installing on Solaris 10 as the splunk user, you must set additional privileges to start splunkd and bind to reserved ports. To start splunkd as the splunk user on Solaris 10, run: # usermod -K defaultpriv=basic,net_privaddr,proc_exec,proc_fork splunk

To allow the splunk user to bind to reserved ports on Solaris 10, run (as root): # usermod -K defaultpriv=basic,net_privaddr splunk

Uninstall Splunk Uninstall Splunk

Before you uninstall, stop Splunk. Navigate to $SPLUNK_HOME/bin and type ./splunk stop (or just splunk stop on Windows). Use your local package management commands to uninstall Splunk. In most cases, files that were not originally installed by the package will be retained. These files include your configuration and index files which are under your installation directory. Note: $SPLUNK_HOME refers to the Splunk installation directory. On Windows, this is C:\Program Files\Splunk by default. For most Unix platforms, the default installation directory is /opt/splunk; for Mac OS, it is /Applications/splunk. RedHat Linux

To uninstall Splunk on RedHat: rpm -e splunk_product_name

56

Debian Linux

To uninstall Splunk on Debian: dpkg -r splunk

To purge (delete everything, including configuration files) on Debian: dpkg -P splunk FreeBSD

To uninstall Splunk from the default location on FreeBSD: pkg_delete splunk

To uninstall Splunk from a different location on FreeBSD: pkg_delete -p /usr/splunk splunk Solaris

To uninstall Splunk on Solaris: pkgrm splunk Windows

To uninstall Splunk on Windows: Use the Add or Remove Programs option in the Control Panel. Uninstall Splunk manually

If you can't use package management commands, use these instructions to uninstall Splunk. Note: These instructions will not remove any init scripts that have been created. 1. Stop Splunk. $SPLUNK_HOME/bin/splunk stop

2. Find and kill any lingering processes that contain "splunk" in its name. For Linux and Solaris: kill -9 `ps -ef | grep splunk | grep -v grep | awk '{print $2;}'`

For FreeBSD and Mac OS kill -9 `ps ax | grep splunk | grep -v grep | awk '{print $1;}'`

57

3. Remove the Splunk installation directory, $SPLUNK_HOME. For example: rm -rf /opt/splunk

Note: For Mac OS, you can also remove the installation directory by dragging the folder into the trash. 3. Remove any Splunk datastore or indexes outside the top-level directory, if they exist. rm -rf /opt/splunkdata

4. Delete the splunk user and group, if they exist. For Linux, Solaris, and FreeBSD: userdel splunk groupdel splunk

For Mac OS: You can use the System Preferences > Accounts panel to manage users and groups. For Windows: Open a command prompt and run the command msiexec /x against the msi package that you installed.

Correcting the user selected during Windows installation Correcting the user selected during Windows installation

If you have selected "other user" during the Windows GUI installation, and that user does not exist or perhaps you mistyped the information, you can go into the Windows Service Control Manager and specify the correct information, as long as you have not started Splunk yet. If you specified an invalid user during the Windows GUI installation process, you will see two popup error windows. To change the user: 1. In Control Panel > Administrative Tools > Services, find the Splunkd and SplunkWeb services. You'll note they are not started and are currently owned by the Local System User. 2. Right click on each service and choose Properties. The properties dialog for that service is displayed. 3. Select the Log On tab. 4. Select the This account radio button and fill in the correct domain\username and password. 5. Click Apply. 6. Click OK. 58

6. Repeat for the second service (you must do this for both Splunkd and Splunk Web. 7. You can now either start both services from the Service Manager or from the Splunk command line interface.

Configure a standalone 3.4.x deployment server Configure a standalone 3.4.x deployment server

If you are planning to migrate to Splunk 4.x, but do not want to migrate your deployment clients until a later time, you can set up a stripped-down, standalone 3.4.x deployment server to serve your deployment clients until you're ready to migrate them (Splunk 4.x deployment server is incompatible with clients older than 4.x). This procedure assumes the following: • You have an existing deployment server at fflanda.splunk.com listening on port 8089 for deployment clients. • Your deployment clients are all 3.x and are all polling this deployment server. • The deployment classes are in $SPLUNK_HOME/etc/modules/distributedDeployment/classes. • This Splunk instance is also an index server that must be upgraded. Given the above, the procedure is as following: 1. Download the latest Splunk 3.4.x build for your architecture. 2. Back up the existing $SPLUNK_HOME/etc using tar -zxvf $SPLUNK_HOME/etc > /tmp/splunk_old_etc.tgz 3. Stop Splunk, remove deployment.conf and deployment classes. 4. If $SPLUNK_HOME = /opt/splunk, mv to /opt/splunk_old. Otherwise, install the 3.4.x tarball or rpm in the default location 5. Extract the splunk_old_etc.tgz over top of the fresh installation. 6. Remove/rename any inputs.conf/outputs.conf files in /opt/splunk_depserver/etc/system/local or /opt/splunk_depserver/etc/apps/. You will probably want to keep authentication.conf, server.conf, /opt/splunk/etc/passwd, /opt/splunk/etc/auth/* - pretty much anything but inputs.conf, outputs.conf and the unchanged splunk-launch.conf. 7. Disable Splunk Web on the newly installed instance using the CLI or web.conf. 8. Execute mv /opt/splunk /opt/splunk_depserver 9. Edit /opt/splunk_depserver/etc/splunk-launch.conf to change $SPLUNK_HOME to /opt/splunk_depserver. If $SPLUNK_DB is also set, comment out this variable so that the new instance does not try to write to the old data store.

59

10. To ensure that this deployment server remains functional, switch its license out for a 3.x forwarder license. Copy $SPLUNK_HOME/etc/splunk-forwarder.license to $SPLUNK_HOME/etc/splunk.license . 11. Execute /opt/splunk_depserver/bin/splunk start 12. Execute mv /opt/splunk_old /opt/splunk, then perform migration. 13. During post-migration start up, Splunk will notice that the old management port is bound, and will prompt the admin to change the management port. Keep track of this new port as you must update it in any distributed search or REST configurations. 14. Execute /opt/splunk_depserver/bin/splunk list deploy-clients -auth admin:changeme and verify that the deployment clients have been in touch with the deployment server. 14. Review /opt/splunk_depserver/var/log/splunk/splunkd.log and /opt/splunk/var/log/splunk/splunkd.log for errors.

60

What comes next? Ready to start using Splunk? Ready to start using Splunk?

Now that you've got Splunk installed on one server, here're some links to get you started: • Learn what Splunk is, what it does, and how it's different. • Learn how to add your data to Splunk. • Add and manage users. • Learn how to search, monitor, report, and more • Learn about Splunk knowledge and how you can use it to maximize Splunk's value. • Plan your deployment, from gigabytes to terabytes per day.

Estimating your storage requirements Estimating your storage requirements

This topic describes how to estimate the size of your Splunk index on disk and associated data so that you can plan your storage capacity requirements. When Splunk indexes your data, the resulting data falls into two basic categories: the compressed raw data that is persisted and the indexes that point to this data. With a little experimentation, you can estimate how much disk space you will need. Typically, the compressed, persisted data that Splunk extracts from your data inputs amounts to approximately 10% of the raw data that comes into Splunk. The indexes that are created to access this data can be anywhere from 10% to 110% of the data that comes in. This value is affected strongly by how many unique terms occur in your data. Depending on the characteristics of your data, you might want to tune your segmentation settings later on. For an introduction to how segmentation works and how it affects your index size, you can also watch this video on segmentation by one of Splunk's lead developers. The best way to get an idea of your index size is to experiment by installing a test copy of Splunk somewhere and indexing a representative sample of your data, and then checking the sizes of the resulting directories defaultdb. Here's how: Once you've indexed your sample: 1. Go to $SPLUNK_HOME/var/lib/splunk/defaultdb/db/db-hot. 2. Run du -shc hot_v*/rawdata to determine how large the compressed persisted raw data is. 3. Run du -ch hot_v* and look at the last total line to see the size of the index.

61

This is the persisted data to which the items in the index point. Typically, this file's size is about 10% of the size of the sample data set you indexed. 4. Add the values you get together. This is the total size of the index and associated data for the sample you have indexed. You can now use this to extrapolate the size requirements of your Splunk index and rawdata directories over time.

Hardware capacity planning for your Splunk deployment Hardware capacity planning for your Splunk deployment

Splunk is a very flexible product that can be deployed to meet almost any scale and redundancy requirement. However, that doesn't remove the need for care and planning. This article discusses high level considerations for Splunk deployments, including sizing and availability. After you've worked through the general layout of your Splunk search topology, the other sections in this document can explain more thoroughly how to implement them, along with the formal Admin guide for Splunk. Reference Hardware

Let's consider a common, commodity hardware server as our standard: • Intel 64-bit chip architecture • Standard Linux or Windows 64-bit distribution • 2 CPU, 4 core per CPU, 2.5-3Ghz per core • 8GB RAM • 4x300GB SAS hard disks at 10,000 rpm each in RAID 10 ♦ capable of 800 IO operations / second (IOPS) • standard 1Gb Ethernet NIC, optional 2nd NIC for a management network For the purposes of this discussion this will be our single server unit. Note that the only exceptional item here is the disk array. Splunk is often constrained by disk I/O first, so always consider that first when selecting your hardware. Performance Checklist

The first step to deciding on a reference architecture is sizing - can your Splunk handle the load? For the purposes of this guide we assume that managing forwarder connections and configurations (but not their data!) to be free. Therefore we need to look at index volume and search load. Question 1: Do you need to index more than 2GB per day? Question 2: Do you need more than 2 concurrent users?

If the answer to both questions is 'NO' then your Splunk instance can safely share one of the above servers with other services, with the caveat that Splunk be allowed sufficient disk I/O on the shared box. If you answered yes, continue. 62

Question 3: Do you need to index more than 100GB per day? Question 4: Do you need to have more than 4 concurrent users?

If the answer to both questions is 'NO', then a single dedicated Splunk server of our reference architecture should be able to handle your workload. Question 5: Do you need more than 500GB of storage?

At a high level, total storage is calculated as follows: daily average rate x retention policy x 1/2

You can generally safely use this simple calculation method. If you want to base your calculation on the specific type(s) of data that you'll be feeding into Splunk, you can use the method described in "Estimating your storage requirements" in this manual. Splunk can generally, including indexes, store raw data at approximately half the original size thanks to compression. Given allowances for operating system and disk partitioning, that suggests about 500GB of usable space. In practical terms, that's ~6 months of fast storage at 5GB/day, or 10 days at 100GB/day. If you need more storage, you can either opt for more local disks for fast access (required for frequent searching) or consider attached or network storage (acceptable for occasional searching). Low-latency connections over NFS or CIFS are acceptable for searches over long time periods where instant search returns can be compromised to lower cost per GB. Shares mounted over WAN connections and standby storage such as tape are never acceptable. Beyond 100GB/Day

If you have requirements greater than 100GB/day or 4 concurrent users, you'll want to leverage Splunk's scale-out capabilities. That involves using distributed search to run searches in parallel across multiple indexers at once, and possibly load balancing the incoming data with auto load balanced Splunk forwarders. Also, at this scale it is very likely that you'll have high availability or redundancy requirements, covered in greater detail below. Question 6: Do you need more than 300GB/day of daily indexed volume?

If you do not - i.e. you are between 100GB/day and 300GB/day - you should be able to have multiple dual-purpose Splunk boxes that are searching across each other.

63

example of a search user searching on one Splunk instance and having their search distributed to other instances Additional Considerations

• you can use a third party load balancer to assign users to different Splunk instances • the recommended best practice for Splunk forwarder management is to use auto load balancing • you can use Splunk deployment server to propagate Splunk apps and user preferences between instances • if you need more than 4 concurrent search users 'per server' this deployment is not appropriate ♦ for example if you have 2 reference servers but need more than 8 search users Beyond 300GB/Day

For deployments of 300GB/day or larger, consider a three tier Splunk deployment. In this model, search is separated from index by creating Splunk search heads, or instances of Splunk that only do searching. That allows for more efficient use of hardware, and to scale search usage (mostly) independently of index volume.

Example Splunk distributed topology. This example could handle up to 400GB/day and 8 concurrent search users for common use cases. Dividing Up Indexing and Searching

At daily volumes above 300GB/day, it makes sense to slightly modify our reference hardware to reflect the differing needs of indexers and search heads. Search heads do not need disk I/O, nor much local storage. However they are far more CPU bound than indexers. Therefore we can change our recommendations to: Search Head 64

• Intel 64-bit chip architecture • Standard Linux or Windows 64-bit distribution • 4 CPU, 4 core per CPU, 2.5-3Ghz per core • 4GB RAM • 2 300GB SAS hard disks at 10,000 rpm each in RAID 0 • standard 1Gb Ethernet NIC, optional 2nd NIC for a management network Given that a search head will be CPU bound, if fewer, more performant servers are desired, adding more and faster CPU cores is best. Note: The guideline of 1 core per active user still applies. Don't forget to account for scheduled searches in your CPU allowance as well. Indexer • Intel 64-bit chip architecture • Standard Linux or Windows 64-bit distribution • 2 CPU, 4 core per CPU, 2.5-3Ghz per core • 8GB RAM • 8 300GB SAS hard disks at 10,000 rpm each in RAID 10 ♦ capable of 1200 IO operations / second (Iopps) • standard 1Gb Ethernet NIC, optional 2nd NIC for a management network The indexers will be busy both writing new data and servicing the remote requests of search heads. Therefore disk I/O is the primary bottleneck. At these daily volumes, likely local disk will not provide cost effective storage for the time frames that speedy search is desired, suggesting fast attached storage or networked storage. While there are too many types of storage to be prescriptive, here are guidelines to consider: • indexers do many bulk reads • indexers do many disk seeks Therefore... • more disks (specifically, more spindles) are better • total throughput of the entire system is important, but... • disk to controller ratio should be higher, similar to a database Ratio of indexers to search heads

Technically, there is no practical Splunk limitation on the number of search heads an indexer can support, or the number of indexers a search head can search against. However systems limitations suggest a ratio of approximately 8 to 1 for most use cases. That is a rough guideline however; if you have many searchers compared to your total data volume, more search heads make sense, for example. In general, the best use of a separate search head is to populate summary indexes. This search head will then act like an indexer to the primary search head that users log into.

65

Accommodating many simultaneous searches

A common question for a large deployment is: how do I account for many concurrent users? Let's take as an example a system that may have at peak times 48 concurrent searches. The short answer is that we can accommodate 48 simultaneous searches on a cluster of indexers and search heads where each machine has enough RAM to prevent swapping. Assuming that each search takes 200MB of RAM per system, that is roughly 10GB additional RAM (beyond indexing requirements). This is because CPU will degrade gracefully with more concurrent jobs but once the working set of memory for all processes exceeds the physical RAM, performance drops catastrophically with swapping. The caveat here is that a search's run time will be longer in proportion to the number of free cores when no searches were running. For example, suppose the indexers were doing nothing before the searches arrived and have 8 cores each. Suppose the first (of identical searches) takes 10s to complete. Then the first 8 searches will each take 10s to complete since there is no contention. However, since there are only 8 cores, if there are 48 searches running, each search will take 48/8 = 6x longer than if only 1-8 searches were running. So now, every search takes ~1 minute to complete. This leads to the observation that the most important thing to do here is add indexers. Indexers do the bulk of the work in search (reading data off disk, decompressing it, extracting knowledge and reporting). If we want to return to the world of 10s searches, we use 6 indexers (one search head is probably still fine, though it may be appropriate to set aside a search head for summary index creation) and searches 1-8 now take 10/6 = 1.6s and with 48 searches, each takes 10s. Unfortunately, the system isn't typically idle before searches arrive. If we are indexing 150 GB/day, at peak times, we probably are using 4 of the 8 cores doing indexing. That means that the first 4 searches take 10s, and having 48 searches running takes 48/4 = 12x longer, or 2 min to complete each. Now one might say: let me put sixteen cores per indexer rather than eight and avoid buying some machines. That makes a little bit of sense, but is not the best choice. The number of cores doesn't help searches 1-16 in this case; they still take 10s. With 48 searches, each search will take 48/16 = 3x longer, which is indeed better than 6x. However, it's usually not too much more expensive to buy two 8 core machines, which has advantages: the first few searches will now just take 5s (which is the most common case) and we now have more aggregate I/O capacity (doubling the number of cores does nothing for I/O, adding servers does). The lesson here is to add indexers. Doing so reduces the load on any system from indexing, to free cores for search. Also, since the performance of almost all types of search scale with the number of indexers, searches will be faster, which mitigates the effect of slowness from resource sharing. Additionally making every search faster, we will often avoid the case of concurrent searches with concurrent users. In realistic situations, with hundreds of users, each user will run a search every few minutes, though not at the exact same time as other users. By reducing the search time by a factor of 6 (by adding more indexers), the concurrency factor will be reduced (not necessarily by 6x, but by some meaningful factor). This in turn, lowers the concurrency related I/O and memory contention.

66

Summary of Performance Recommendations

Daily Volume

Number of Search Users

Recommended Indexers

Recommended Search Heads

< 2GB/day