201403 Tampa Performance3

Performance Tuning and Analysis of Red Hat Enterprise Linux 6 and 7 Jeremy Eder Principal Software Engineer March 13, 2014

1

RED HAT CONFIDENTIAL | JEREMY EDER

Agenda: Performance Analysis of RHEL6/7

●

Performance Engineering Overview

●

Performance Analysis Utilities

●

Tuned

●

NUMA Tuning

●

Power Management

2

RED HAT CONFIDENTIAL | JEREMY EDER

Performance Engineering Overview Micro-Benchmarks Applications/Benchmarks

3

RED HAT CONFIDENTIAL | JEREMY EDER

Red Hat Performance Engineering

●

4

Benchmarks – code path coverage ●

CPU – linpack, lmbench

●

Memory – lmbench, McCalpin Streams

●

Disk IO – Iozone, aiostress – scsi, FC, iSCSI

●

Filesystem – IOzone, postmark– ext3/4, xfs. gfs2,gluster

●

Network – Netperf – 10 Gbit, 40 Gbit IB, PCI3

●

Bare Metal, RHEL6/7 KVM

●

White box AMD/Intel, with our OEM partners

RED HAT CONFIDENTIAL | JEREMY EDER

Red Hat Confidential

Performance Projects / Tooling ●

RHEL6.5 “numad” “tuna”, and “tuned”

●

Tuna used to bind IRQ's / real-time like isolation

●

Profiling challenges −Data address profiling (cache-2-cache detection), providing: • the hottest contended cachelines • the process names, addresses, pids, tids causing that contention • the cpus they ran on, • and how the cacheline is being accessed (read or write)

5

RED HAT CONFIDENTIAL | JEREMY EDER

Performance Optimization Out-of-the-box

6

RED HAT CONFIDENTIAL | JEREMY EDER

Performance Tuning Automation

Automatic Tuning

Manual Tuning

Tuned

N/A

Transparent Hugepages

Static Hugepages

numad

NUMA Pinning (numactl)

irqbalance

IRQ Pinning (tuna)

RHEL7 numa_balancing

8

RED HAT CONFIDENTIAL | JEREMY EDER

But...what if we have a problem ? ●

9

Automatic not enough...

RED HAT CONFIDENTIAL | JEREMY EDER

But...what if we have a problem ? ●

Automatic not enough...

●

Need to eek out the last X percent

10

RED HAT CONFIDENTIAL | JEREMY EDER

But...what if we have a problem ? ●

Automatic not enough...

●

Need to eek out the last X percent

●

Need Determinism

11

RED HAT CONFIDENTIAL | JEREMY EDER

But...what if we have a problem ? ●

Automatic not enough...

●

Need to eek out the last X percent

●

Need Determinism

●

OK...here's where it get's fun!

12

RED HAT CONFIDENTIAL | JEREMY EDER

Overview of Performance Analysis Utilities perf

13

RED HAT CONFIDENTIAL | JEREMY EDER

perf

Userspace tool to read CPU counters and kernel tracepoints

14

RED HAT CONFIDENTIAL | JEREMY EDER

perf list

List counters/tracepoints available on your system

15

RED HAT CONFIDENTIAL | JEREMY EDER

perf list

grep for something interesting, maybe to see what numabalance is doing ? ...more on numabalance later...

16

RED HAT CONFIDENTIAL | JEREMY EDER

perf top

System-wide 'top' view of busy functions

17

RED HAT CONFIDENTIAL | JEREMY EDER

perf record ●

18

Record system-wide (-a)

RED HAT CONFIDENTIAL | JEREMY EDER

perf record

Record system-wide (-a) ● A single command ●

19

RED HAT CONFIDENTIAL | JEREMY EDER

perf record

Record system-wide (-a) ● A single command ● An existing process (-p) ●

20

RED HAT CONFIDENTIAL | JEREMY EDER

perf record

Record system-wide (-a) ● A single command ● An existing process (-p) ● Add call-chain recording (-g) ●

21

RED HAT CONFIDENTIAL | JEREMY EDER

perf record

Record system-wide (-a) ● A single command ● An existing process (-p) ● Add call-chain recording (-g) ● Only specific events (-e) ●

22

RED HAT CONFIDENTIAL | JEREMY EDER

perf record

Record system-wide (-a) ● A single command ● An existing process (-p) ● Add call-chain recording (-g) ● Only specific events (-e) ●

23

RED HAT CONFIDENTIAL | JEREMY EDER

perf report

/dev/zero

24

RED HAT CONFIDENTIAL | JEREMY EDER

perf report

/dev/zero

oflag=direct

25

RED HAT CONFIDENTIAL | JEREMY EDER

perf diff

Compare 2 perf recordings

26

RED HAT CONFIDENTIAL | JEREMY EDER

perf probe (dynamic tracepoints)

Insert a tracepoint on any function... Try 'perf probe -F' to list possibilities

27

RED HAT CONFIDENTIAL | JEREMY EDER

Cool, what else ya got ?

28

RED HAT CONFIDENTIAL | JEREMY EDER

Overview of Performance Analysis Utilities Performance Co-Pilot (PCP)

29

RED HAT CONFIDENTIAL | JEREMY EDER

Performance Co-Pilot (PCP)

(Multi) system-level performance monitoring and management

30

RED HAT CONFIDENTIAL | JEREMY EDER

pmchart – graphical metric plotting tool ●

31

Can plot myriad performance statistics

RED HAT CONFIDENTIAL | JEREMY EDER

pmchart – graphical metric plotting tool ●

Can plot myriad performance statistics

●

Recording mode allows for replay ● ●

32

i.e. on a different system Record in GUI, then # pmafm $recording.folio

RED HAT CONFIDENTIAL | JEREMY EDER

pmchart – graphical metric plotting tool ●

Can plot myriad performance statistics

●

Recording mode allows for replay ● ●

●

33

i.e. on a different system Record in GUI, then # pmafm $recording.folio

Ships with many pre-cooked “views”...for example: ●

ApacheServers: CPU%/Net/Busy/Idle Apache Servers

●

Overview: CPU%/Load/IOPS/Net/Memory

RED HAT CONFIDENTIAL | JEREMY EDER

What do these views look like ?

34

RED HAT CONFIDENTIAL | JEREMY EDER

Performance Co-Pilot Demo Script

●

Tiny script to exercise 4 food groups...

# CPU /root/pig -s 5 # DISK dd if=/dev/zero of=/root/2GB count=2048 bs=1M oflag=direct # NETWORK netperf -H lab7 -l 5 # MEMORY /root/pig -m 16384 -l sleep -s 5

35

RED HAT CONFIDENTIAL | JEREMY EDER

CPU % Load Avg IOPS

Network Memory Allocated 36

RED HAT CONFIDENTIAL | JEREMY EDER

collectl mode

CPU

37

RED HAT CONFIDENTIAL | JEREMY EDER

collectl mode

CPU IOPS

38

RED HAT CONFIDENTIAL | JEREMY EDER

collectl mode

CPU IOPS

NET

39

RED HAT CONFIDENTIAL | JEREMY EDER

collectl mode

CPU IOPS

NET

MEM

40

RED HAT CONFIDENTIAL | JEREMY EDER

atop mode

41

RED HAT CONFIDENTIAL | JEREMY EDER

Questions so far ?

42

RED HAT CONFIDENTIAL | JEREMY EDER

NUMA Tuning Discovery

43

RED HAT CONFIDENTIAL | JEREMY EDER

Visualize NUMA Topology: lstopo NUMA Node 0

How can I visualize my system's NUMA topology in Red Hat Enterprise Linux? https://access.redhat.com/site/solutions/62879

44

RED HAT CONFIDENTIAL | JEREMY EDER

Visualize NUMA Topology: lstopo NUMA Node 0

NUMA Node 1

How can I visualize my system's NUMA topology in Red Hat Enterprise Linux? https://access.redhat.com/site/solutions/62879

45

RED HAT CONFIDENTIAL | JEREMY EDER

Visualize NUMA Topology: lstopo NUMA Node 0

PCI Devices

NUMA Node 1

How can I visualize my system's NUMA topology in Red Hat Enterprise Linux? https://access.redhat.com/site/solutions/62879

46

RED HAT CONFIDENTIAL | JEREMY EDER

Visualize NUMA Topology: lscpu # lscpu Architecture: ... CPU(s): On-line CPU(s) list: Thread(s) per core: Core(s) per socket: Socket(s): NUMA node(s): ... NUMA node0 CPU(s): NUMA node1 CPU(s): 47

x86_64 16 0-15 1 8 2 2 0-7 8-15

RED HAT CONFIDENTIAL | JEREMY EDER

Logical Cores/HT

Visualize NUMA Topology: lscpu # lscpu Architecture: ... CPU(s): On-line CPU(s) list: Thread(s) per core: Core(s) per socket: Socket(s): NUMA node(s): ... NUMA node0 CPU(s): NUMA node1 CPU(s): 48

x86_64 16 0-15 1 8 2 2 0-7 8-15

RED HAT CONFIDENTIAL | JEREMY EDER

Logical Cores/HT

NUMA Node/Core Map

NUMA Topology and PCI Bus ●

●

Install adapters “close” to the CPU that will run the performance critical application. When BIOS reports locality, irqbalance handles NUMA/IRQ affinity automatically. # lstopo-no-graphics |egrep 'NUMA|eth4' NUMANode L#0 (P#0 144GB) NUMANode L#1 (P#1 144GB) Net L#10 "eth4"

49

RED HAT CONFIDENTIAL | JEREMY EDER

RHEL NUMA Scheduler

●

●

RHEL6 ●

numactl, numastat enhancements

●

numad – usermode tool, dynamically monitor, auto-tune

RHEL7 – numabalance ●

Enable / Disable ●

50

sysctl kernel.numabalancing={0,1}

RED HAT CONFIDENTIAL | JEREMY EDER

How to manage NUMA manually - Checklist

Checklist

Tool

 Research Topology

 lstopo/lscpu

 Make a resource plan

 cgroups, numactl

 Consider I/O

 irqbalance/PCI Bus

 Virtualization

 numatune/numad

51

RED HAT CONFIDENTIAL | JEREMY EDER

How to manage NUMA manually - Checklist

Checklist

Tool

 Research Topology

 lstopo/lscpu

 Make a resource plan

 cgroups, numactl

 Consider I/O

 irqbalance/PCI Bus

 Virtualization

 numatune/numad

52

RED HAT CONFIDENTIAL | JEREMY EDER

How to manage NUMA manually - Checklist

Checklist

Tool

 Research Topology

 lstopo/lscpu

 Make a resource plan

 cgroups, numactl

 Consider I/O

 irqbalance/PCI Bus

 Virtualization

 numatune/numad

53

RED HAT CONFIDENTIAL | JEREMY EDER

How to manage NUMA manually - Checklist

Checklist

Tool

 Research Topology

 lstopo/lscpu

 Make a resource plan

 cgroups, numactl

 Consider I/O

 irqbalance/PCI Bus

 Virtualization

 numatune/numad

54

RED HAT CONFIDENTIAL | JEREMY EDER

How to manage NUMA manually - Checklist

Checklist

Tool

 Research Topology

 lstopo/lscpu

 Make a resource plan

 cgroups, numactl

 Consider I/O

 irqbalance/PCI Bus

 Virtualization

 numatune/numad

55

RED HAT CONFIDENTIAL | JEREMY EDER

NUMA Tools: numastat

Completely rewritten for RHEL6.4 Per-node /proc/meminfo Backwards compatible

56

RED HAT CONFIDENTIAL | JEREMY EDER

NUMA Tools: numastat

Completely rewritten for RHEL6.4 Per-node /proc/meminfo Backwards compatible

57

RED HAT CONFIDENTIAL | JEREMY EDER

numastat: compatibility mode (old) # numastat numa_hit numa_miss numa_foreign interleave_hit local_node other_node

58

node0

node1

77587739 0 0 30254 69302710 8285029

131990042 0 0 30099 129511360 2478682

RED HAT CONFIDENTIAL | JEREMY EDER

numastat: per-node meminfo (new) # numastat -mczs MemTotal MemFree MemUsed Active FilePages Active(file) Active(anon) AnonPages Inactive Inactive(file) Slab SReclaimable SUnreclaim AnonHugePages 59

Node 0 -----65491 60366 5124 2650 2021 1686 964 964 341 340 380 208 173 134

RED HAT CONFIDENTIAL | JEREMY EDER

numastat: per-node meminfo (new) # numastat -mczs MemTotal MemFree MemUsed Active FilePages Active(file) Active(anon) AnonPages Inactive Inactive(file) Slab SReclaimable SUnreclaim AnonHugePages 60

Node 0 Node 1 Total ------ ------ -----65491 65536 131027 60366 59733 120099 5124 5803 10927 2650 2827 5477 2021 3216 5238 1686 2277 3963 964 551 1515 964 550 1514 341 946 1287 340 946 1286 380 438 818 208 207 415 173 230 403 134 236 370

RED HAT CONFIDENTIAL | JEREMY EDER

NUMA Tuning numad

61

RED HAT CONFIDENTIAL | JEREMY EDER

NUMA: Process Scheduler Behavior ●

●

Scheduler distributes load evenly across all cores ●

Maintains responsiveness

●

Optimizing for CPU utilization

Tries to use idle CPUs, regardless of where process memory is located

BUT! Using remote memory degrades performance!

62

RED HAT CONFIDENTIAL | JEREMY EDER

How to manage NUMA manually - Checklist

Red Hat has taken 2 approaches for customers: Short Term (RHEL6.4)

Long Term (RHEL7)

 Userspace solution

numad

63

RED HAT CONFIDENTIAL | JEREMY EDER

How to manage NUMA manually - Checklist

Red Hat has taken 2 approaches for customers: Short Term (RHEL6.4)  Userspace solution

numad

64

Long Term (RHEL7)  In-Kernel Solution

numabalance

RED HAT CONFIDENTIAL | JEREMY EDER

What does it look like ?

65

RED HAT CONFIDENTIAL | JEREMY EDER

Effect of Automatic NUMA Balancing Software # numastat -c pig (default scheduler – non-optimal) Before numad

PID 2578 2579 2580 2581

66

(pig) (pig) (pig) (pig)

Node 0 Node 1 2123 11878 1988 12013 14000 1 1981 12020

RED HAT CONFIDENTIAL | JEREMY EDER

Effect of Automatic NUMA Balancing Software # numastat -c pig (default scheduler – non-optimal) Before numad

PID 2578 2579 2580 2581 PID 2578 2579 2580 2581

67

(pig) (pig) (pig) (pig)

(pig) (pig) (pig) (pig)

Node 0 Node 1 2123 11878 1988 12013 14000 1 1981 12020 Node 0 Node 1 14000 0 0 14000 14000 0 0 14000

RED HAT CONFIDENTIAL | JEREMY EDER

After numad

Effect of numad/numabalance Automatic NUMA Balancing - NUMAD 14000

12000

M egabytes Per NUM A Node

10000

8000

6000

numad begins numad done

4000

2000

0 1

2

3

4

5

6

7

8

9

10

11

Tim e (sec onds)

NODE-0-MB

68

NODE-1-MB

RED HAT CONFIDENTIAL | JEREMY EDER

12

13

14

15

16

Questions on NUMA ?

69

RED HAT CONFIDENTIAL | JEREMY EDER

tuned

70

RED HAT CONFIDENTIAL | JEREMY EDER

What is “tuned” ?

Tuning profile delivery mechanism Red Hat ships tuned profiles that improve performance for many workloads...hopefully yours!

71

RED HAT CONFIDENTIAL | JEREMY EDER

Yes, but why do I care ?

72

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Storage Performance Boost

Larger is better 73

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Network Latency Performance Boost C-state lock improves determinism, reduces jitter

Latency (Microseconds)

250 200

C6

C3

C1

150 100 50 0 Max

Time (1-sec intervals) 74

RED HAT CONFIDENTIAL | JEREMY EDER

C0

What do the guts look like ?

75

RED HAT CONFIDENTIAL | JEREMY EDER

tuned Profile Summary: RHEL6 Tunable

default

enterprisestorage

sched_min_ granularity_ns

4ms

10ms

10ms

10ms

10ms

sched_wakeup_granula rity_ns

4ms

15ms

15ms

15ms

15ms

dirty_ratio

20% RAM

40%

10%

40%

40%

dirty_background_ratio

10% RAM

5%

swappiness

60

10

30

I/O Scheduler (Elevator)

CFQ

deadline

deadline

deadline

Filesystem Barriers

On

Off

Off

Off

CPU Governor

ondemand

performance

Disk Read-ahead

virtual-host virtualguest

latencyperformance

deadline

deadline

performance

performance

4x

Disable THP

Yes

CPU C-States

Locked @ 1

76

RED HAT CONFIDENTIAL | JEREMY EDER

throughputperformance

What's happening to Tuned for RHEL7 ?

77

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Updates for RHEL7 ●

78

Installed by default!

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Updates for RHEL7 ●

Installed by default!

●

Profiles automatically set based on install type: ● ●

79

Desktop/Workstation: balanced Server/HPC: throughput-performance

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Updates for RHEL7 ●

80

Re-written for maintainability and extensibility.

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Updates for RHEL7 ●

Re-written for maintainability and extensibility. ●

81

Configuration consolidated to single tuned.conf file

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Updates for RHEL7 ●

Re-written for maintainability and extensibility. ●

●

82

Configuration is now consolidated a single tuned.conf file Optional hook/callout capability

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Updates for RHEL7 ●

Re-written for maintainability and extensibility. ●

● ●

83

Configuration is now consolidated a single tuned.conf file Optional hook/callout capability Adds concept of Inheritance (just like httpd.conf)

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Updates for RHEL7 ●

Re-written for maintainability and extensibility. ●

● ●

●

84

Configuration is now consolidated a single tuned.conf file Optional hook/callout capability Adds concept of Inheritance (just like httpd.conf) Profiles updated for RHEL7 features and characteristics

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Profile Inheritance

Parents throughput-performance

balanced

latency-performance

Children network-throughput

desktop

virtual-host virtual-guest

85

RED HAT CONFIDENTIAL | JEREMY EDER

network-latency

Tuned: Profile Inheritance

Parents throughput-performance

balanced

latency-performance

Children network-throughput

desktop

network-latency

Your-DB

Your-Middleware

virtual-host virtual-guest

Your-Web 86

RED HAT CONFIDENTIAL | JEREMY EDER

What do the guts look like ?

87

RED HAT CONFIDENTIAL | JEREMY EDER

Tuned: Throughput Profiles - RHEL7 Tunable

Units

Balanced

throughput-performance

Inherits From/Notes

throughput-performance

sched_min_ granularity_ns

nanoseconds

auto-scaling

10000000

sched_wakeup_granularity_ns

nanoseconds

3000000

15000000

dirty_ratio

Percent

20

40

dirty_background_ratio

Percent

10

10

swappiness

Weight 1-100

60

10

I/O Scheduler (Elevator) Filesystem Barriers

network-throughput

deadline Boolean

CPU Governor

Enabled ondemand

Disk Read-ahead

KB

128

Disable THP

Boolean

Enabled

Energy Perf Bias

normal

performance 4096

performance

kernel.sched_migration_cost_ns

nanoseconds

500000

min_perf_pct (intel_pstate only)

Percent

auto-scaling

tcp_rmem

Bytes

auto-scaling

Max=16777216

tcp_wmem

Bytes

auto-scaling

Max=16777216

88 udp_mem

PagesRED HAT CONFIDENTIAL auto-scaling | JEREMY EDER

Max=16777216

100

Tuned: Latency Profiles - RHEL7 Tunable

Units

Balanced

latency-performance

network-latency

Inherits From/Notes

latency-performance

sched_min_ granularity_ns

nanoseconds

auto-scaling

sched_wakeup_granularity_ns

nanoseconds

3000000

dirty_ratio

percent

20

10

dirty_background_ratio

percent

10

3

swappiness

Weight 1-100

60

10

I/O Scheduler (Elevator) Filesystem Barriers

10000000 10000000

deadline Boolean

CPU Governor

Enabled ondemand

performance

N/A

No

CPU C-States

N/A

Locked @ 1

Energy Perf Bias

normal

performance

Disable THP

Boolean

kernel.sched_migration_cost_ns

nanoseconds

min_perf_pct (intel_pstate only)

percent

net.core.busy_read

microseconds

50

net.core.busy_poll

microseconds

50

net.ipv4.tcp_fastopen

Boolean

89

kernel.numa_balancing

N/A

Yes

RED HAT CONFIDENTIAL | JEREMY EDER

Boolean

5000000 100

Enabled Disabled

Tuned: Virtualization Profiles - RHEL7 Tunable

Units

throughput-performance

Inherits From/Notes sched_min_ granularity_ns

nanoseconds

10000000

sched_wakeup_granularity_ns

nanoseconds

15000000

dirty_ratio

percent

40

dirty_background_ratio

percent

10

swappiness

Weight 1-100

10

virtual-host

virtual-guest

throughputperformance

throughputperformance

30 5

I/O Scheduler (Elevator) Filesystem Barriers

Boolean

CPU Governor Disk Read-ahead

performance Bytes

Energy Perf Bias

4096 performance

kernel.sched_migration_cost_ns

nanoseconds

5000000

min_perf_pct (intel_pstate only) 90

percentRED HAT CONFIDENTIAL | JEREMY100 EDER

30

RHEL “tuned” package Available profiles: - balanced - desktop - latency-performance - myprofile - network-latency - network-throughput - throughput-performance - virtual-guest - virtual-host Current active profile: myprofile

91

RED HAT CONFIDENTIAL | JEREMY EDER

Power Management

92

RED HAT CONFIDENTIAL | JEREMY EDER

CPU Tuning: C-states (idle states)

Power Saving

Performant C-state Impact on Jitter

Latency (Microseconds)

250 200

C6

C3

C1

150 100 50 0 Max

Time (1-sec intervals) 93

RED HAT CONFIDENTIAL | JEREMY EDER

C0

Power Consumption RHEL6 vs RHEL6@C0 ●

● ● ●

94

C-state lock increases power draw over “out of the box” Test

Efficiency [Wh] % Diff

Kernel Build

+12.5%

Disk Read

+32.2%

Disk Write

+25.6%

Unpack tar.gz

+23.3%

Active Idle

+41%

Use cron to set latency-performance tuned profile when necessary. Or use BUSY_POLL Set tuned profile in application init script

RED HAT CONFIDENTIAL | JEREMY EDER

Turbostat shows P/C-states on Intel CPUs ●

turbostat in RHEL6.4, cpupowerutils package or kernel-tools in Fedora/RHEL7

Default pk cor CPU

%c0

GHz

TSC

%c1

%c3

%c6

%c7

0

0

0

0.24 2.93 2.88

5.72

1.32

0.00

92.72

0

1

1

2.54 3.03 2.88

3.13

0.15

0.00

94.18

0

2

2

2.29 3.08 2.88

1.47

0.00

0.00

96.25

0

3

3

1.75 1.75 2.88

1.21

0.47

0.12

96.44

%c1

%c3

%c6

%c7

latency-performance pk cor CPU

95

%c0

GHz

TSC

0

0

0 0.00

3.30 2.90 100.00

0.00

0.00

0.00

0

1

1 0.00

3.30 2.90 100.00

0.00

0.00

0.00

0

2

2 0.00

3.30 2.90 100.00

0.00

0.00

0.00

0

3

3 0.00

3.30 2.90 100.00

0.00

0.00

0.00

RED HAT CONFIDENTIAL | JEREMY EDER

Profiling cpuidle and cpufreq ●

●

●

●

●

96

Fixed upstream cpuidle regression in June Future CPUs such as Haswell add more C-states, C8,C9,C10 Turbostat display a bit awkward for 40+ cores Direction in future is to shrink the exit-latency between C-states http://www.breakage.org/2012/11/processor-max_cstat e-intel_idle-max_cstate-and-devcpu_dma_latency/

RED HAT CONFIDENTIAL | JEREMY EDER

Take-aways

97

RED HAT CONFIDENTIAL | JEREMY EDER

Helpful Utilities

●

●

●

●

redhat-support-tool sos kdump perf

Networking

NUMA

Supportability ●

hwloc

●

dropwatch

●

Intel PCM

●

ethtool

●

numactl

●

netsniff-ng (EPEL6)

●

numad

●

tcpdump

●

numatop (01.org) Power/Tuning

●

wireshark/tshark Storage

●

psmisc

●

strace

●

cpupowerutils (R6)

●

blktrace

●

sysstat

●

kernel-tools (R7)

●

iotop

●

systemtap

●

powertop

●

iostat

●

trace-cmd

●

tuna

●

util-linux-ng

●

tuned

98

RED HAT CONFIDENTIAL | JEREMY EDER

Helpful Links

99

●

Red Hat Low Latency Performance Tuning Guide

●

Optimizing RHEL Performance by Tuning IRQ Affinity

●

Red Hat Performance Tuning Guide

●

Red Hat Virtualization Tuning Guide

●

STAC Network I/O SIG

●

Finteligent Low Latency Tuning w/KVM

●

Perf

●

Blog: http://www.breakage.org/ or @jeremyeder

RED HAT CONFIDENTIAL | JEREMY EDER

Questions

100

RED HAT CONFIDENTIAL | JEREMY EDER