Oracle troublesoohting vol 2

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Volume II • Student Guide

D61523GC20

Edition 2.0

May 2011

D72554

Oracle University and Sentra inversiones y servicios LTDA use only

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting

Copyright © 2011, Oracle and/or it affiliates. All rights reserved.

Bill Bell

Disclaimer

Technical Contributors and Reviewers Will Lyons TJ Palazzolo Serge Moiseev

Editors Richard Wallis Malavika Jinka

Publishers Jobi Varghese Shaik Mahaboob Basha

This document contains proprietary y information and is protected by y copyright y g and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle. The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not warranted t d tto be b error-free. f Restricted Rights Notice If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract. Trademark Notice Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Author

1

Course Overview Course Objectives 1-2 Target Audience 1-3 Introductions 1-4 Course Schedule 1-5 Course Appendix 1-7 Course Practices 1-8 Classroom Guidelines 1-9 For More Information 1-10 Related Training 1-11 Oracle by Example (OBE) 1-12

2

WLST Monitoring Objectives 2-2 WLS Domains: Review 2-3 Java Management Extension (JMX): Review 2-4 WLS MBean Hierarchies 2-5 WLS MBean Reference Documentation 2-6 Console Monitoring: Review 2-8 WebLogic Scripting Tool (WLST): Review 2-9 WLST MBean Syntax: Review 2-10 Domain Runtime 2-11 Basic Jython Syntax: Review 2-12 Basic WLST Commands 2-13 Variable Declaration 2-14 Password Management 2-15 Error Handling 2-16 File I/O 2-17 Standard Jython Libraries 2-18 WLST Example: Monitor a JMS Server 2-19 Quiz 2-20 Summary 2-23 Practice 2-1 Connecting to the Classroom Grid 2-24 Practice 2-2 Developing a Custom Monitoring Script 2-25

iii

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Contents

Guardian Objectives 3-2 Guardian Capabilities 3-3 Using Guardian 3-4 Guardian Architecture 3-5 Agent Installation 3-6 Collected Data 3-7 Client Installation 3-8 Guardian User Interface 3-9 Activating a Domain 3-10 Creating a Domain Inventory 3-11 Signatures and Bundles 3-12 Updating the Signature Repository 3-13 Signature Annotations 3-14 Evaluating a Domain 3-15 Evaluation Summary 3-16 Generating a Support Request 3-17 Command-Line Interface 3-18 Quiz 3-19 Summary 3-22 Practice 3-1 Using Guardian to Evaluate a Domain 3-23

4

Diagnostic Framework Essentials Objectives 4-2 Road Map 4-3 WebLogic Diagnostic Framework (WLDF) 4-4 WLDF Architecture 4-5 WLS Logging: Review 4-6 Log Severity Thresholds 4-7 Application Logging 4-8 Server Logging Bridge 4-9 WLDF Configuration: Overview 4-10 Diagnostic Images 4-11 Capturing a Server Diagnostic Image 4-12 WLST: Downloading Diagnostic Image Files 4-13 Diagnostic Archives 4-14 Configuring Server Diagnostic Archives 4-15 Archive Retirement Policies 4-16 Archive Database Schema 4-17 Viewing Archive Contents 4-18 Creating a Diagnostic Module 4-19

iv

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

3

v

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

WLDF WLST Examples 4-20 Section Summary 4-22 Road Map 4-23 Harvester Architecture 4-24 Metric Collector Definitions 4-25 Configuring a Metric Collector 4-26 Watches and Notifications 4-28 Configuring a Watch 4-29 Watch Alarms 4-31 Configuring a JMS Notification 4-32 Configuring an Email Notification 4-33 Harvester WLST: Example 4-34 Watch WLST: Example 4-35 WLDF Sample Framework 4-36 Section Summary 4-37 Practice 4-1 Harvesting Diagnostic Metrics 4-38 Road Map 4-39 New Monitoring Dashboard 4-40 Viewing the Dashboard 4-41 Monitoring Dashboard Interface 4-42 Views 4-43 Built-In Views 4-44 Creating a Custom View 4-45 Metric Browser 4-46 Anatomy of a Chart 4-47 Chart and Graph Properties 4-48 Chart Styles 4-49 Current and Historical Data 4-50 Section Summary 4-51 Practice 4-2 Monitoring Diagnostic Metrics 4-52 Road Map 4-53 Subsystem Debugging 4-54 Console Debug Scopes 4-55 Debug Scopes: Examples 4-56 Debug Logging 4-57 WLST Debugging: Examples 4-58 Section Summary 4-59 Quiz 4-60 Summary 4-64

Diagnostic Instrumentation Objectives 5-2 Road Map 5-3 Instrumentation Scenarios 5-4 Instrumentation Architecture 5-5 Monitor Actions 5-6 Application-Scoped Modules 5-8 WLS Monitor Library 5-9 Deployment Plan Review 5-11 WLDF and Deployment Plans 5-12 WLDF Deployment Plan: Example 5-13 WLDF Hot Swap 5-14 Configuring a System-Scoped Monitor 5-15 Configuring an Application-Scoped Monitor 5-17 Aspect-Oriented Programming (AOP) Concepts 5-18 Custom Monitors 5-19 Instrumentation WLST: Example 5-20 Instrumentation and Request Performance 5-21 Section Summary 5-22 Practice 5-1 Configuring and Monitoring Diagnostic Events 5-23 Road Map 5-24 Request Context ID 5-25 Viewing Context IDs 5-26 Request Dying 5-27 Available Dyes 5-28 Configuring a Dye Injection Monitor 5-29 Event Filtering 5-30 Configuring Dye Masks 5-31 Event Throttling 5-32 Configuring Throttle Properties 5-33 Section Summary 5-34 Quiz 5-35 Summary 5-38 Practice 5-2 Tracing a Client Request 5-39

6

JVM Diagnostics Objectives 6-2 Road Map 6-3 Basic Java Concepts 6-4 Java Virtual Machine (JVM): Review 6-5 Oracle JVM Support 6-6

vi

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

5

vii

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JVM Recommendations 6-7 JVM Memory 6-8 Garbage Collection 6-9 Sun HotSpot Garbage Collection 6-10 Garbage Collection (GC) Types 6-11 Setting WLS JVM Arguments 6-12 Basic Sun JVM Arguments 6-13 JRockit Garbage Collection 6-14 Basic JRockit JVM Arguments 6-15 Out of Memory 6-16 Out-of-Memory Response 6-17 Memory Leak 6-18 JVM Crash 6-19 JVM Error Log 6-20 Section Summary 6-21 Road Map 6-22 JVM Tool Varieties 6-23 Java Stack Trace 6-24 Java Thread Dump: Overview 6-25 Thread Dump Signal 6-26 JVM Crash Actions 6-27 Verbose GC 6-28 Sun JVM Profiler Agent 6-29 Sun JVM Diagnostic Tools: Overview 6-30 Sun Diagnostic Tools: Examples 6-31 JVisualVM 6-33 Using JVisualVM 6-34 Section Summary 6-36 Practice 6-1 Troubleshooting a Running JVM 6-37 Road Map 6-38 Console JVM Monitoring 6-39 JVM WLST: Example 6-40 WLS Low Memory Detection 6-41 Configuring Low Memory Detection 6-42 Section Summary 6-43 Road Map 6-44 JRockit Diagnostic Tools: Overview 6-45 JRockit Diagnostic Tools: Examples 6-46 Management Communication 6-47 JRockit Mission Control (JRMC) 6-48 JRockit Discovery Protocol (JDP) 6-49

7

Troubleshooting Java Applications Objectives 7-2 Java Exception-Handling Concepts 7-3 Exception Chains 7-4 “Class Not Found” Errors 7-5 Class Cast Errors 7-6 Classpath: Review 7-7 WebLogic Start Script: Review 7-8 Viewing the WLS Classpath 7-9 Manifest Files and the Classpath 7-10 Domain Libraries 7-11 Java Class Loaders 7-12 Searching Class Loaders 7-13 Searching Class Loaders: Example 7-14 Default WLS Class Loader Hierarchy 7-15 Java EE Packaging: Review 7-16 Prefer Web Application Classes 7-17 Prefer Enterprise Application Classes 7-18 Client Library Errors 7-19 Null Pointer Errors 7-20 Stack Overflow Errors 7-21

viii

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JVM Browser 6-50 Management Console: Features 6-51 Management Console: General > Overview 6-52 Management Console: Runtime > Threads 6-53 Management Console: MBeans > Triggers 6-54 JRockit Flight Recorder (JFR) 6-55 Integration of JRockit Flight Recorder and WLDF 6-56 Starting the Flight Recorder from JRMC 6-57 Flight Recorder Output 6-58 General > Overview 6-59 Memory: Object Statistics 6-60 Code > Overview 6-61 Memory Leak Detector (Memleak): Features 6-62 Memleak: Trend Tab 6-63 Memleak: Type Graph 6-64 Section Summary 6-65 Quiz 6-66 Summary 6-70 Practice 6-2 Troubleshooting Applications on JRockit 6-71

8

Troubleshooting Servers Objectives 8-2 Road Map 8-3 WLS Message Catalog: Review 8-4 Server Startup Errors 8-5 Boot Identity Errors 8-6 WLS Native Libraries 8-7 Setting the Native Library Path 8-8 Causes of Unresponsive Servers 8-9 WLS Threading Architecture 8-10 Execute Thread State 8-11 Work Managers 8-12 Work Manager Architecture 8-13 Creating a Work Manager 8-14 Creating and Using a Request Class 8-15 Assigning Work Managers to Applications 8-16 Monitoring a Server Thread Pool 8-17 Monitoring Individual Server Threads 8-18 Server Monitoring: WLST Examples 8-19 Server WLDF Image Contents 8-20 Java Deadlock Concepts 8-21 Thread Analysis 8-22 Lock Chains 8-23 Stuck Thread Detection 8-24 Overload Protection 8-25 Configuring Overload Protection 8-26 Section Summary 8-27 Practice 8-1 Investigating Server Problems 8-28 Road Map 8-29 WLS Deployment: Review 8-30 Deployment Errors 8-32 Application Staging 8-33 Deployment Memory Errors 8-34 Shared Library: Review 8-35 Library Errors 8-36 Deployment Debug Flags 8-37

ix

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Too Many Open Files Errors 7-22 Quiz 7-23 Summary 7-25 Practice 7-1 Investigating Classpath Problems 7-26

9

Troubleshooting JDBC Objectives 9-2 JDBC: Review 9-3 Data Sources: Review 9-4 JDBC Management: WLST Examples 9-5 JDBC Runtime Attributes 9-6 JDBC Monitoring: WLST Examples 9-7 JDBC WLDF Image Contents 9-8 JDBC WLDF Monitor: Review 9-9 Data Source Diagnostic Profiling 9-10 Configuring Diagnostic Profiling 9-11 JDBC Debug Flags 9-12 Other JDBC Debugging Tools 9-13 Common Configuration Errors 9-14 Configuration Error Examples 9-15 Insufficient Connection Errors 9-16 Connection Leaks 9-17 Database Cursor Considerations 9-18 Common Connection Errors 9-19 Statement Timeout 9-20 Data Sources and Database Availability 9-21 Retry Frequency and Login Timeout 9-22 Connection Testing: Review 9-23 Testing Trusted Connections 9-24 Firewall Considerations 9-25 Multi Data Source: Overview 9-26 Multi Data Source: Architecture 9-27 Java Persistence API (JPA): Overview 9-28 JPA Configuration: Overview 9-29 Troubleshooting JPA: Overview 9-30 Quiz 9-31 Summary 9-34 Practice 9-1 Investigating JDBC Problems 9-35

x

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Application Error Handling 8-38 Application Monitoring: Review 8-39 Application Monitoring: WLST Examples 8-40 Section Summary 8-41 Quiz 8-42 Summary 8-45

xi

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

10 Troubleshooting JMS Objectives 10-2 JMS: Review 10-3 WebLogic JMS Configuration: Review 10-5 JMS Transactions: Review 10-7 JMS Management: Overview 10-8 Console JMS Management 10-9 JMS Management: WLST Examples 10-10 JMS Runtime MBean Hierarchy 10-11 JMS Monitoring: WLST Examples 10-12 JMS Diagnostic Image Contents 10-13 JMS Message Logging 10-14 Configuring JMS Logging 10-15 JMS Debug Flags 10-16 Message Type Considerations 10-17 Common Configuration Errors 10-18 JMS Client Libraries 10-20 Out-of-Memory Errors and Quotas 10-21 Configuring a JMS Server Quota 10-22 Creating a Destination Quota 10-23 Message Paging 10-24 Too Many Pending Messages 10-25 Quota Blocking Policies 10-26 Thresholds and Flow Control 10-27 Configuring Thresholds 10-28 Tuning Flow Control 10-29 Lost Messages 10-30 Time to Live (TTL) 10-31 Expiration Policies 10-32 Delivery Mode 10-33 Message Redelivery 10-34 Time to Deliver (TTD) 10-35 Durable Subscriber Review 10-36 Monitoring and Managing Subscriptions 10-37 Duplicate Messages 10-38 Poison Messages 10-39 Consumer Acknowledgement Modes 10-40 Messages Out of Sequence 10-41 Unit of Order (UOO): Overview 10-42 Unit of Work (UOW): Overview 10-43 Message-Driven Beans (MDBs): Review 10-44

11 Troubleshooting Security Objectives 11-2 Road Map 11-3 Secure Sockets Layer (SSL): Review 11-4 SSL Communication: Review 11-5 WebLogic SSL Scenarios 11-6 Proxy Server SSL Scenarios 11-8 Keystore: Review 11-9 Trust Keystores 11-10 Keytool: Review 11-11 WebLogic SSL Support 11-13 SSL Configuration: Review 11-14 Restarting SSL 11-16 SSL Debug Flags 11-17 SSL Handshake Trace 11-18 Other SSL Traces 11-19 Invalid Format or Cipher Errors 11-20 Certificate Validation Errors 11-21 Host Name Verification Errors 11-22 Certificate Chains 11-23 WLS Chain Validation Utility 11-24 Missing Constraint or Policy Errors 11-25 Section Summary 11-26 Practice 11-1 Investigating SSL Problems 11-27 Road Map 11-28 Security Realm: Review 11-29 Security Provider Stores 11-30 Some Security Providers 11-31 Embedded LDAP: Review 11-32 Embedded LDAP Backups 11-33 Embedded LDAP Synchronization Issues 11-34 Viewing Embedded LDAP Contents 11-35 LDAP Concepts 11-36 LDAP Structure 11-37

xii

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

MDB Capabilities 10-45 MDB Runtime Attributes 10-46 MDB Diagnostics and Debugging 10-47 Quiz 10-48 Summary 10-51 Practice 10-1 Investigating JMS Problems 10-52

12 Troubleshooting Node Manager Objectives 12-2 Node Manager (NM): Review 12-3 Node Manager Types: Review 12-4 Node Manager Configuration: Review 12-5 Basic Java Node Manager Properties 12-6 Java Node Manager Logging 12-7 Java Node Manager Availability 12-8 Basic Script Node Manager Interface 12-9 Node Manager Server Start Parameters 12-11 Configuring Server Start Parameters 12-13 Monitoring Node Managers 12-14 Node Manager: WLST Examples 12-15 Common Configuration Errors 12-16 Generating Template Properties for Java NM 12-17 Node Manager Authentication 12-18 Configuring Node Manager Credentials 12-19 Node Manager Trusted Domains 12-20 Machine Enrollment 12-21 Server Boot Identity 12-22 Configuring Node Manager SSL 12-23

xiii

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

LDAP Search Operations 11-38 Resetting Admin Password in Embedded LDAP 11-39 Database Store Cache Synchronization Issues 11-40 Auditing Provider 11-41 Security Audit Events 11-42 Configuring the Auditing Provider 11-43 Realm Debug Flags 11-44 Typical Authentication Trace 11-45 Typical Role Mapping Trace 11-47 Typical Authorization Trace 11-48 LDAP Trace Log 11-49 Authentication Provider Control Flags 11-50 External LDAP Authentication Providers 11-52 LDAP Provider Configuration: Overview 11-53 Common LDAP Issues 11-56 Section Summary 11-57 Quiz 11-58 Summary 11-62 Practice 11-2 Investigating Security Realm Problems 11-63

13 Troubleshooting Clusters Objectives 13-2 Road Map 13-3 Cluster Review 13-4 Proxy Plug-in Review 13-5 Obtaining and Using Plug-Ins 13-6 Oracle HTTP Server (OHS) Review 13-7 Oracle Process Manager and Notification Server (OPMN) Review 13-9 OPMNCTL Examples 13-10 OHS Logs 13-11 Plug-in Configuration Review 13-12 Basic Plug-in Parameters 13-13 Proxy Connection Architecture 13-14 Dynamic Server List 13-16 Connection Parameters 13-17 Common Connectivity Issues 13-18 Proxy SSL Issues 13-19 Proxy Debug Page 13-20 Proxy Debug Log 13-22 Typical Proxy Trace 13-23 Section Summary 13-24 Practice 13-1 Investigating Proxy Problems 13-25 Road Map 13-26 Cluster Communication Review 13-27 Unicast Architecture 13-28 Session Management Review 13-29 Session Persistence Review 13-30 In-Memory Replication Review 13-31 Cluster Monitoring WLST Examples 13-32 Session Monitoring WLST Examples 13-33 Session Monitoring Attribute 13-34 Session Instrumentation 13-35 Cluster Debug Flags 13-36 Typical Cluster Heartbeat Trace 13-37 Typical Replication Trace: Primary 13-38 Typical Replication Trace: Secondary 13-39 Common Replication Issues 13-40

xiv

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz 12-24 Summary 12-26 Practice 12-1 Investigating Node Manager Problems 12-27

A WebLogic SNMP Simple Network Management Protocol (SNMP) A-2 SNMP Architecture A-3 Object Identifier (OID) A-4 Management Information Base (MIB) A-5 WLS MIB and OIDs A-6 Common SNMP Message Types A-7 WLS SNMP Architecture A-8 Creating an SNMP Agent A-10 Configuring an SNMP Agent A-11 SNMP Channels A-12 WLS SNMP Notifications A-13 Creating Trap Monitors A-14 Creating Trap Destinations A-15 SNMP Security A-16 Configuring Agent Security A-17 Configuring SNMP V3 Credentials A-18 Configuring Trap Destination Security A-19 WLS SNMP Utility A-20

xv

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

HttpSession API Overview 13-41 Serialization Overview 13-42 Serialization Debug Messages 13-43 Section Summary 13-44 Quiz 13-45 Lesson Summary 13-48 Practice 13-2 Investigating Cluster Replication Problems 13-49

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle University and Sentra inversiones y servicios LTDA use only

Troubleshooting JDBC

After completing this lesson, you should be able to: • Use the console, WLST, and WLDF to monitor and troubleshoot data sources • Identify common data source configuration issues • Identify connection leaks • Configure a data source to handle database availability • Describe the multi data source architecture • Explain the relationship between JDBC and JPA

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 2

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Objectives

•

Java Database Connectivity (JDBC): – Allows Java programs to access databases in a uniform way – Requires the use of a JDBC-compliant driver that supports your DB vendor and version

• •

Several drivers are included with WLS and are already in the server classpath. Most other third-party drivers are also supported. Application

1. 2. 3. 4. 5.

Create connection. Create statement. Execute SQL. Process results. Close connection.

JDBC Driver

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The JDBC API is the industry standard for database-independent connectivity between the Java programming language and a wide range of databases. The JDBC API provides a calllevel API for SQL-based database access. The JDBC API makes it possible to do three things: establish a connection with a database, send SQL statements, and process the results. With the JDBC API, no configuration is required on the client side. With a driver written in the Java programming language, all the information needed to make a connection is defined by the JDBC URL and optional connection properties. Type 4 JDBC drivers are installed with WebLogic Server in the \server\lib folder, where is the directory in which you installed WebLogic Server. Driver class files are included in the manifest classpath in weblogic.jar, so the drivers are automatically added to your classpath on the server. If you choose a custom installation, ensure that the WebLogic JDBC Drivers option is selected (checked). If this option is not selected, the drivers are not installed.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 3

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JDBC: Review

Data sources: • Allow database connectivity to be managed by the application server • Are obtained by applications from the server’s JNDI tree • Use a dynamic pool of reusable database connections Data Source App

Connection

App

Connection Connection

App

1. 2. 3. 4.

JNDI lookup Reserve connection. Perform SQL Release connection.

Connection JDBC Driver

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server can manage your database connectivity through JDBC Data Sources and multi data sources. Each data source you configure contains a pool of database connections that are created when the data source instance is created—when it is deployed or targeted, or at server startup. The connection pool can grow or shrink dynamically to accommodate demand. Applications look up a data source on the Java Naming and Directory Interface (JNDI) tree or in the local application context (java:comp/env), depending on how you configure and deploy the object, and then request a database connection. When finished with the connection, the application uses the close operation on the connection, which simply returns the connection to the connection pool in the data source. Oracle WebLogic Server data sources allow connection information such as the JDBC driver, the database location (URL), and the username and password to be managed and maintained in a single location, without requiring the application to worry about these details. In addition, limiting the number of connections is important if you have a licensing limitation on your database or if it can support only a specific capacity.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 4

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Data Sources: Review

• •

Data sources support various management tasks via the console or WLST. Most data source modifications require that the data source be restarted.

Restart a data source using the latest configuration settings: serverRuntime() ds = getMBean('/JDBCServiceRuntime/ServerC/ JDBCDataSourceRuntimeMBeans/NodeXDataSource') ds.shutdown() ds.start() Recreate all data source connections: ... ds.reset()

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Using the Administration Console or WebLogic Scripting Tool (WLST) scripts, you can manage the JDBC data sources in your domain for maintenance or troubleshooting purposes. The JDBCDataSourceRuntimeMBean’s operations include: • forceShutdown(): Destroys all open database connections, including those currently in use by applications. The shutdown() command will fail if there are active connections. • shrink(): When you shrink a data source, WebLogic Server reduces the number of connections in the pool to the greater of either the initial capacity or the number of connections currently in use. • reset(): Closes and re-creates all available database connections in a data source. This may be necessary, for example, after the DBMS has been restarted. Often when one connection in a data source has failed, all of the connections in the pool are bad. • suspend(): When you suspend a data source, the data source is marked as disabled and applications cannot use connections from the pool. Applications that already have a reserved connection from the data source when it is suspended will get an exception when trying to use the connection. • clearStatementCache(): When a prepared statement or callable statement is used on a connection, WebLogic Server caches the statement so that it can be reused. You can manually clear the statement cache for all connections in a data source.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 5

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JDBC Management: WLST Examples

Metric

Description

State

Running, Suspended, Shutdown, Overloaded, or Unhealthy

Capacity

Current number of connections in the pool

Capacity High

Highest number of connections in the pool since server start

Active Connections

Current number of connections reserved by applications

Active Connections High

Highest number of reserved connections since server start

Active Connections Average

Average number of reserved connections at any given time

Number Available

Current number of connections not reserved

Number Unavailable

Current number of connections that are either reserved or being tested

Number Unavailable High

Highest number of connections that were unavailable since server start

Failed Reserve

Cumulative count of connection reservation failures Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Refer to the JDBCDataSourceRuntimeMBean documentation for a complete list of available metrics. If the Connection Reserve Timeout attribute of the data source is not disabled (–1), additional runtime metrics are available that describe how often applications waited for connections and for how much time, as in the following example: WaitingForConnectionCurrentCount A data source can be in one of several states: • Running: The data source is enabled (deployed and not suspended). This is the normal state of the data source. • Suspended: The data source has been administratively disabled, although connections may still remain alive. • Shutdown: The data source is shut down and all database connections have been closed. • Overloaded: All available connections are in use. • Unhealthy: All connections are unavailable (not because they are in use). This state occurs if the database server is unavailable when the data source is created (creation retry must be enabled) or if all connections have failed connection tests (on creation, on reserve, or on periodic testing).

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 6

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JDBC Runtime Attributes

Get data source runtime statistics: serverRuntime() ds = getMBean('/JDBCServiceRuntime/ServerC/ JDBCDataSourceRuntimeMBeans/NodeXDataSource') print 'Active: ' , ds.getActiveConnectionsCurrentCount() print 'Unavailable: ' , ds.getNumUnavailable() Print the status of each connection in a data source: ds = ... ds.dumpPool()

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Refer to the JDBCDataSourceRuntimeMBean documentation for a list of all available attributes and operations.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 7

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JDBC Monitoring: WLST Examples

Diagnostic images include a file named JDBC.img that captures the dumpPool() output for all data sources: • • •

Current capacity Status and general attributes of each connection Reservation stack trace for each active connection

JDBC.img: Resource Pool:NodeX_DS:dumpPool Current Capacity = 19 Resource Pool:NodeX_DS:dumpPool available[0] = autoCommit=true,enabled=true,isXA=true ... Resource Pool:NodeX_DS:dumpPool reserved[3] = ... at weblogic.jdbc.common.internal.ConnectionEnv.setup ... at weblogic.common.resourcepool.ResourcePoolImpl ... at com.mycompany.payroll.EmployeeManager.getEmployee ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

A diagnostic image is a heavyweight artifact meant to serve as a server-level state dump for the purpose of diagnosing significant failures. It enables you to capture a significant amount of important data in a structured format and then to provide that data to support personnel for analysis. Because it is an artifact intended primarily for internal consumption, the image contents are not documented in detail and are subject to change.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 8

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JDBC WLDF Image Contents

Subsystem JDBC

Available Code Points Before/After Connection Reserve/Release Connection Before/After Commit Before/After Rollback Before/After SQL Statement Perform a stack dump when a connection is released from any data source.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

A diagnostic monitor is a dynamically manageable unit of diagnostic code that is inserted into server or application code at specific locations. WLDF provides a library of predefined diagnostic monitors and actions. Diagnostic actions perform some type of data collection intended to help gain insight into the server or application. Each diagnostic action can be used with only those monitor types with which they are compatible. All actions also capture general statistics such as the current time, transaction ID, and user ID, if applicable. A Stack Dump action generates an instrumentation event at the affected location in the program execution to capture a stack dump. It captures the current Java stack trace as an event payload. A Thread Dump action generates an instrumentation event at the affected location in the program execution to capture a JVM thread dump. It captures the thread dump as event payload. When attached to “before” monitors, the Display Arguments instrumentation event captures input arguments to the joinpoint (for example, method arguments). When attached to “after” monitors, the instrumentation event captures the return value from the joinpoint.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 9

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JDBC WLDF Monitor: Review

•

You can also enable more sophisticated diagnostic data collection on individual data sources. Data source connections are profiled periodically and the results are recorded to the WLDF archive as events. These event payloads can be viewed only by using WLST.

• •

Event Type

Description

Usage

A thread that is currently using a specific connection

Last Usage

A thread that received an exception while using a connection

Reservation Wait

A thread that is currently waiting for a connection

Reservation Failed

A thread that received an exception while requesting a connection

Leak

A missing connection along with the reservation stack trace

Statement Usage

A cached SQL statement that is being used Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

You can configure any data source to collect profile information to help you pinpoint the source of a problem. The collected profile information is stored as events in the WLDF event archive. Each available data source event type has the same attribute names, but they will contain different information. When configuring your data source for profiling, you must specify the interval at which profile data is harvested. Enable connection usage profiling to collect information about threads currently using connections from the pool of connections in the data source. This profile information can help determine why applications are unable to get connections from the data source. By default, enabling connection usage profiling on its own will not provide a stack trace of the threads using the connections. To obtain this information, you must enable profiling of connection leaks in addition to this profile type. Enable connection reservation wait profiling to collect information about threads currently waiting to reserve a connection from the data source. This profile information can help determine why applications are unable to get connections from the data source or to wait for connections. Enable connection leak profiling to collect information about threads that have reserved a connection from the data source but have had the connection leaked (the connection was not properly returned to the pool of connections). This profile information can help determine which applications are not properly closing JDBC connections. Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 10

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Data Source Diagnostic Profiling

1 2 Enable or disable diagnostic event types.

3 How often to profile connections?

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

To configure diagnostic profiling for a JDBC data source: 1. Navigate to the data source that you want to modify. 2. Select Configuration > Diagnostics tab. 3. Enable any of the following diagnostic profiling options. Also configure how frequently the selected profiling options should occur by using the Profile Harvest Frequency Seconds field (default is five minutes). Enable connection reservation failure profiling to collect information about threads that attempt to reserve a connection from the data source but fail to get that connection. This profile information can help determine why applications are unable to get connections from the data source even after reserving them. Enable connection last usage profiling to collect information about the previous thread that last used the connection. This information is useful when you are debugging problems with connections infected in pending transactions that cause subsequent XA operations on the connections to fail. Enable statement cache entry profiling to collect information for prepared and callable statements added to the statement cache, and for the threads that originated the cached statements.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 11

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring Diagnostic Profiling

Flag

Description

DebugJDBCConn

Trace all connection reserve and release operations.

DebugJDBCSQL

Trace all JDBC API calls, parameters, and return values.

DebugJDBCInternal

Dump all low-level internal data source activities.

DebugJTAJDBC

Trace transaction management for JDBC resources.

Server log messages with DebugJDBCSQL enabled: Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

DebugJDBCSQL (scope weblogic.jdbc.sql): Prints information about all JDBC methods invoked, including their arguments and return values, and thrown exceptions DebugJDBCConn (scope weblogic.jdbc.connection): Traces all connection reserve and release operations in data sources as well as all application requests to get or close connections DebugJDBCInternal (scope weblogic.jdbc.internal): Low-level debugging related to the data source, the connection environment, and the data source manager DebugJDBCRMI (scope weblogic.jdbc.rmi): Similar to JDBCSQL but at the RMI level. Turning on this flag and JDBCSQL will get two sets of debug messages for each operation called from a remote JDBC client. DebugJDBCDriverLogging (scope weblogic.jdbc.driverlogging): Enables JDBC driver-level logging (this replaces ServerMBean JDBCLoggingEnabled and getJDBCLogFileName). Note that to get driver-level tracing for Oracle, you need to use ojdbc14_g.jar instead of ojdbc14.jar. DebugJTAJDBC (scope weblogic.transaction.jdbc, weblogic.jdbc.transaction): Traces information about reading and writing JTA records DebugJTAXAStackTrace (scope weblogic.transaction.stacktrace): Detailed tracing that prints stack traces at various critical locations

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 12

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

JDBC Debug Flags

• • •

JDBC Spy logging connection properties for WLS drivers Debugging and logging properties for Oracle thin and other drivers Third-party JDBC debugging tools such as P6Spy

Data source URL with JDBC Spy logging attributes: jdbc:weblogic:db2://199.177.1.1:50000;spyAttributes=(log=(file )db2-spy.out;load=weblogic.jdbc.db2.DB2Driver;timestamp=yes) JDBC Spy log file: spy>> spy>> spy>> spy>>

Connection[1].createStatement OK (Statement[1]) Statement[1].executeQuery(String sql) sql = select name, job from employee where empno=7369

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

WebLogic JDBC Spy is a wrapper that wraps a WebLogic Type 4 JDBC driver. It logs detailed information about JDBC calls issued by an application and then passes the calls to the wrapped WebLogic Type 4 JDBC driver. You can use the information in the logs to help troubleshoot problems in your application. Before you start the server, add /server/lib/wlspy.jar to your CLASSPATH, where is the directory in which you installed the WebLogic Server software. Using WLS administrative tools such as the console or WLST, append the WebLogic JDBC Spy options to the data source URL. Enclose all JDBC Spy options in one set of parentheses; separate multiple options with a semicolon. The “load” parameter indicates the Java class name of the JDBC driver whose work should be intercepted by JDBC Spy. P6Spy is an open source framework to support applications that intercept and optionally modify database statements. The P6Spy distribution includes the following modules: P6Log and P6Outage. P6Log intercepts and logs the database statements of any application that uses JDBC. This application is particularly useful for developers to monitor the SQL statements produced by EJB servers, enabling the developer to write code that achieves maximum efficiency on the server. P6Outage detects long-running statements that may be indicative of a database outage problem; it will log any statement that surpasses the configurable time boundary during its execution.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 13

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Other JDBC Debugging Tools

Typical causes include: • Using invalid connection URL, credentials, and/or attributes • Missing driver or wrong version in the system classpath (bundled drivers are included indirectly using MANIFEST entries in JAR files) • Missing native libraries for older, platform-dependent drivers • Specifying invalid connection testing parameters

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The 11g version of the Oracle thin driver (ojdbc6.jar for JDK 6, and ojdbc5.jar for JDK 5) is installed with Oracle WebLogic Server. In addition to the Oracle thin driver, the mySQL 5.0.x (mysql-connector-java-commercial-5.0.x-bin.jar) JDBC driver is installed with WebLogic Server. Drivers are installed in the \server\lib folder (where is the folder where WebLogic Server is installed) by using weblogic.jar. The manifest file found in weblogic.jar lists driver JARs to be loaded when weblogic.jar is loaded (when the server starts). Therefore, you do not need to add these JDBC drivers to your CLASSPATH. If you plan to use a third-party JDBC driver that is not installed with WebLogic Server, you must install the drivers, which includes updating your CLASSPATH with the path to the driver files and may include updating your PATH with the path to database client files. If you plan to use a different version of any of the drivers installed with WebLogic Server, you can replace the driver file in \server\lib with an updated version of the file or add the new file to the front of your CLASSPATH. Copies of the MySQL and Oracle thin drivers installed with WebLogic Server and other supporting files are installed in \server\ext\jdbc\. There is a subdirectory in this folder for each DBMS. If you need to revert to the version of the driver installed with WebLogic Server, you can copy the file from \server\ext\jdbc\DBMS to \server\lib.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 14

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Common Configuration Errors

Invalid URL or missing native libraries: Invalid credentials:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

In the first example in the slide, the data source activation failed (either during server startup or later when the data source is first created) due to the fact that no connections could be established. The message “IO Exception” likely indicates that the database URL could not be reached. Therefore, the database is unavailable or the connection properties of the data source are incorrect. For some type 2 native JDBC drivers, this message can also result if the required native libraries could not be found by the JVM. In the second example, the data source activation failed not because a network connection could not be established, but because the database rejected the connections due to a security violation. As the error message indicates, the supplied connection credentials were not granted access.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 15

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuration Error Examples

• • •

If a Maximum Capacity is not indicated, the default is 25 connections (production mode). If the Reserve Timeout is greater than 0, data source clients block and wait for a specified time before an error occurs. If the Reserve Timeout is –1, clients immediately receive an exception if no connections are available.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When an application requests a connection from a data source, if all connections in the data source are in use and if the data source has expanded to its maximum capacity, the application will get a ConnectionUnavailableSQLException. To avoid this, you can configure the Connection Reserve Timeout value (in seconds) so that connection requests will wait for a connection to become available. After the Connection Reserve Timeout has expired, if no connection becomes available, the request will fail and the application will get a PoolLimitSQLException exception. If you set Connection Reserve Timeout to –1, a connection request will time out immediately if there is no connection available. If you set Connection Reserve Timeout to 0, a connection request will wait indefinitely. The default value is 10 seconds. You may also want to tune this attribute if you have very high network timeout settings between WLS and your database, and you want WLS to be more responsive to connectivity issues. Connection requests that wait for a connection block a thread. If too many connection requests concurrently wait for a connection and block threads, your system performance can degrade. To avoid this, you can set the “Maximum Waiting for Connection” attribute, which limits the number connection requests that can concurrently wait for a connection. If the maximum number of requests has been met, a SQLException is thrown when an application requests a connection.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 16

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Insufficient Connection Errors

•

Poorly implemented applications can starve the data source of connections. – Cache or “hog” connections instead of releasing them. – Do not explicitly release connections using close().

•

To provide a failsafe, WLS can automatically reclaim leaked connections.

Reclaim connection if not released after two minutes.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

A leaked connection is a connection that was not properly returned to the connection pool in the data source. To automatically recover leaked connections, you can specify a value for Inactive Connection Timeout. Find this attribute under the Advanced area of the Connection Pool tab. WebLogic Server will forcibly return a connection to the data source when there is no activity on a reserved connection for the number of seconds that you specify. When set to 0 (the default value), this feature is turned off. Note that the actual timeout could exceed the configured value for Inactive Connection Timeout. The internal data source maintenance thread runs every five seconds. When it reaches the Inactive Connection Timeout (for example 30 seconds), it checks for inactive connections. To avoid timing out a connection that was reserved just before the current check or just after the previous check, the server gives an inactive connection a “second chance.” On the next check, if the connection is still inactive, the server times it out and forcibly returns it to the data source. On average, there could be a delay of 50% more than the configured value.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 17

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Connection Leaks

• • •

Oracle and other databases define a maximum number of concurrent tasks that it can perform, often called “cursors.” A database generates errors when this maximum limit is exceeded. Try the following: – Increase the open cursor limit on the database. – Decrease the Maximum Capacity of the data source. – Decrease the Statement Cache Size of the data source.

Sample database cursor error: java.sql.SQLException: ORA-01000: maximum open cursors exceeded

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Cursors provide a mechanism by which you can iterate over the records in a database. Using cursors, you can get, put, and delete database records. If a database allows duplicate records, cursors are the easiest way you can access anything other than the first record for a given key. The Statement Cache Size attribute determines the total number of prepared and callable JDBC statements to cache for each connection in each instance of the data source. By caching statements, you can increase your system performance. However, you must consider how your DBMS handles open prepared and callable statements. In many cases, the DBMS will maintain a cursor for each open statement. This applies to prepared and callable statements in the statement cache. If you cache too many statements, you may exceed the limit of open cursors on your database server. For example, if you have a data source with 10 connections deployed on two servers, if you set the Statement Cache Size to 10 (the default), you may open 200 (10 x 2 x 10) cursors on your database server for the cached statements.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 18

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Database Cursor Considerations

Exception

Description

ConnectionUnavailable SQLException

Reservation failed because there are no available connections in the pool.

PoolLimitSQLException

Reservation failed due to some threshold such as Reserve Timeout or Maximum Waiting.

PoolDisabledSQL Exception

Reservation failed because data source has been administratively disabled.

ConnectionDeadSQL Exception

Reservation failed because connection was tested and database is unavailable.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

ConnectionUnavailableSQLException: Generated when an application request to get a connection fails because there are currently no connections available in the pool to be allocated. This is a transient failure and is generated if all connections in the pool are currently in use. It can also be thrown when connections are unavailable because they are being tested. PoolLimitSQLException: Generated when an application request to get a connection fails due to a configured threshold of the data source, such as HighestNumWaiters, ConnectionReserveTimeoutSeconds, and so on PoolDisabledSQLException: Generated when an application request to get a connection fails because the JDBC Data Source has been administratively disabled ConnectionDeadSQLException: Generated when an application request to get a connection fails because the connection test on the reserved connection failed. This typically happens when the database server is unavailable. PoolPermissionsSQLException: Generated when an application request to get a connection fails a security authentication or authorization check. This scenario applies only when using the “Client ID on Connection” feature.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 19

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Common Connection Errors

•

Most JDBC drivers support a maximum time limit for SQL statements that can be set: – By using the data source Statement Timeout attribute – Programmatically by an application

•

Increase this value if your application requires complex, long-running database operations.

Refer to your driver documentation for supported values.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

With the Statement Timeout option on a JDBC data source, you can limit the amount of time that a statement takes to execute on a database connection reserved from the data source. When you set a value for Statement Timeout, WebLogic Server passes the time specified to the JDBC driver using the Statement.setQueryTimeout() method. WebLogic Server will make the call, and if the driver throws an exception (it is unsupported, for example), the value will be ignored. In some cases, the driver may silently not support the call or may document limited support. Oracle recommends that you check the driver documentation to verify the expected behavior. When Statement Timeout is set to –1, (the default) statements do not time out. Find the Statement Timeout attribute in the Advanced area of the Connection Pool tab.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 20

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Statement Timeout

•

If a database is not available when the data source is first initialized: – A pool is created with 0 connections – Data source clients will block and/or receive exceptions – Connections are automatically created after the DB becomes available

•

WLS data sources support several database availability features: – Retry frequency – Connection testing – Multi data sources

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Ultimately, even if WebLogic does its best, a connection may fail in the instant after WebLogic successfully tested it and just before the application uses it. Therefore, every application should be written to respond appropriately in the case of unexpected exceptions from a dead connection.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 21

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Data Sources and Database Availability

• •

The Retry Frequency data source attribute specifies how often to attempt creating the initial pool of connections. Most drivers also support connection properties that limit how long to wait when creating a connection before generating an error.

Try to initialize the pool every five minutes if the DB is unavailable.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

WebLogic JDBC data sources include the Connection Creation Retry Frequency option (under Advanced) that you can use to specify the number of seconds between attempts to establish connections to the database. If it is set, and if the database is unavailable when the data source is created, WebLogic Server attempts to create connections in the pool again after the number of seconds you specify, and continues to attempt to create the connections until it succeeds. This option applies to connections created when the data source is created at server startup, when the data source is deployed, or if the initial capacity is increased. It does not apply to connections created for pool expansion or to replace a defunct connection in the pool. By default, Connection Creation Retry Frequency is 0 seconds. When the value is set to 0, connection creation retries is disabled and data source creation fails if the database is unavailable. For data sources used by WLS subsystems (persistent store, for example), a value of 0 can also prevent your server from starting successfully. When creating database connections in a JDBC data source, if the database is unavailable, the request may hang until the default system timeout expires. On some systems, this can be as long as several minutes. The request will hang for each connection in the JDBC data source. To minimize this hang time, you can specify a login timeout value for the connection. All WebLogic Type 4 JDBC Drivers support this connection property.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 22

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Retry Frequency and Login Timeout

Testing helps avoid scenarios in which an application is given an unusable connection.

Test before giving a connection to application. Test connections periodically.

Table name or custom SQL to test connection

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Data sources rely on the Test Reserved Connections feature to know when database connectivity is lost. Testing reserved connections must be enabled and configured for all of the data sources within the multi data source. WebLogic Server will test each connection before giving it to an application. With the failover algorithm, the multi data source uses the results from connection test to determine when to fail over to the next data source in the multi data source. After a test failure, the data source attempts to re-create the connection. If that attempt fails, the multi data source fails over to the next data source. • Test Connections on Reserve: Select this check box to test the database connection before giving it to your application when your application requests a connection from the data source. • Test Frequency: Enable periodic background connection testing by entering the number of seconds between periodic tests. • Test Table Name: Enter the name of a small table to use in a query to test database connections. The standard query is select count(*) from . Most database servers optimize this SQL to avoid a full table scan, but it is still a good idea to use the name of a table that is known to have few rows, or even no rows. If you prefer to use a different query as a connection test, enter SQL followed by a space and the SQL code that you want to use to test database connections.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 23

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Connection Testing: Review

• • •

WLS will “trust” a connection if it has recently been used by an application without problems ( Thresholds and Quotas. The available threshold attributes are the same as for destinations.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 28

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring Thresholds

Flow control settings for a connection factory

Maximum number of messages per second

Minimum number of messages that producer can be slowed to

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

To configure flow control for a JMS connection factory, use its Configuration > Flow Control tab. Edit the following fields: • Flow Maximum: The maximum number of messages per second allowed for a producer that is experiencing a threshold condition. When a producer is flow-controlled it will never be allowed to go faster than this rate. If a producer is not currently limiting its flow when a threshold condition is reached, the initial flow limit for that producer is set to Flow Maximum. If a producer is already limiting its flow when a threshold condition is reached (the flow limit is less than the maximum), the producer will continue at its current flow limit until the next time the flow is evaluated. • Flow Minimum: The minimum number of messages per second allowed for a producer that is experiencing a threshold condition. This is the lower boundary of a producer’s flow limit. That is, WebLogic will not further slow down a producer whose message flow limit is at its Flow Minimum. • Flow Interval: The adjustment period of time, in seconds, when a producer adjusts its flow from the Flow Maximum number of messages to the Flow Minimum amount (or vice versa) • Flow Steps: The Flow Interval adjustment period is divided into a number of steps. For example, 60 seconds divided by six steps results in 10 seconds per step.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 29

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Tuning Flow Control

• •

Lost messages are successfully added to a destination but are later removed without being delivered to consumers. Typical causes include: – – – – –

Expired messages JMS system failure and nonpersistent messages Insufficient retry settings Message delay settings Nondurable topic subscriptions

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

A persistent message is guaranteed to be delivered once and only once. The message cannot be lost due to a JMS provider failure, and it must not be delivered twice. It is not considered sent until it has been safely written to a file or database. WebLogic JMS writes persistent messages to a WebLogic persistent store (disk-base file or JDBC-accessible database) that is optionally targeted by each JMS server during configuration. Nonpersistent messages are not stored. They are guaranteed to be delivered “at most once” (unless there is a JMS provider failure, in which case messages may be lost) and must not be delivered twice. If a connection is closed or recovered, all nonpersistent messages that have not yet been acknowledged will be redelivered. After a nonpersistent message is acknowledged, it will not be redelivered. For durable subscriptions, WebLogic JMS stores a message in a persistent file or database until the message has been delivered to the subscribers or has expired, even if those subscribers are not active at the time that the message is delivered. Durable subscriptions are supported for publish/subscribe (topic) messaging only.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 30

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Lost Messages

•

A TTL is assigned to all new messages in one of the following ways: – Programmatically – Using connection factory defaults – Using destination overrides

• •

By default, expired messages are removed. JMS log entries include message expiration events.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When sending a message, you can optionally specify the delivery mode, priority, and time-tolive (in milliseconds) values. If not specified, these attributes are set to the connection factory configuration attributes. These same attributes can also be administratively overridden by the target destination. To set the default TTL on a connection factory, use the Configuration > Default Delivery tab. To override the current TTL for all messages on a destination, use the destination’s Configuration > Overrides tab. Messages are not necessarily removed from the system at their expiration time, but they are removed within a user-defined number of seconds. The smaller the window, the closer the message removal is to the actual expiration time. Edit a JMS server. Then, using the Scan Expiration Interval field, enter the amount of time (in seconds) that you want the JMS server to pause between its cycles of scanning its destinations for expired messages to process. To disable active scanning, enter a value of 0 seconds. Expired messages are passively removed from the system as they are discovered.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 31

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Time to Live (TTL)

Destinations can perform one of the following actions for expired and failed messages: • Add an entry to the server log file. • Move or “redirect” the message to another destination. 1 2

3

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

To configure expiration policies for a destination: 1. Edit an existing topic or queue in a JMS module. 2. Click the Configuration > Delivery Failure tab. 3. Edit the following fields. • Expiration Policy: The valid expiration policies are: - Discard (default): Removes expired messages from the messaging system. The removal is not logged and the message is not redirected to another location. - Log: Removes expired messages from the system and writes an entry to the server log file indicating that the messages have been removed from the system - Redirect: Moves expired messages from their current location to the Error Destination indicated. The message retains its body and all of its properties. The message also retains all of its header fields, but all property overrides associated with the error destination are applied to the redirected message. In addition, quotas are ignored when redirecting expired messages. • Expiration Logging Format: Specify any additional information that should be logged, such as all message headers (%header%) and all message properties (%properties%).

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 32

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Expiration Policies

• •

All new messages must indicate whether or not they are eligible for persistence. This delivery mode is set in one of the following ways: – Programmatically – Using connection factory defaults – Using destination overrides

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

After you configure a connection factory, you can define various default message delivery parameters that also override any values configured by the client. For example, if a client does not specify certain delivery parameters, the value of those parameters can be controlled with the delivery parameters on this page: 1. Select the connection factory. 2. Click Configuration > Default Delivery. 3. In Default Delivery Mode, select the delivery mode (Persistent or Non-Persistent) assigned to all messages sent by a producer using this connection factory. After you create a queue or topic, you can define message delivery override values that can override those specified by a message producer. 1. Select the queue or topic. 2. Click Configuration > Overrides. 3. In Delivery Mode Override, select the delivery mode (Persistent, Non-Persistent, or NoDelivery) assigned to all messages that arrive at the queue. A value of No-Delivery specifies that the producer’s delivery mode will not be overridden. This attribute is dynamically configurable, but only incoming messages are affected; stored messages are not affected.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 33

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Delivery Mode

•

For temporary consumer problems, destinations can retry failed messages: – After waiting some amount of time – A specific number of times

• •

These redelivery parameters are set programmatically or through connection factory and destination settings. JMS log entries include redelivery events.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

If JMS immediately redelivers the message, the error condition may not be resolved and the application may still not be able to handle the message. However, if an application is configured for a redelivery delay, then when it rolls back or recovers a message, the message is set aside until the redelivery delay has passed, at which point the messages are made available for redelivery. You can specify a limit on the number of times that WebLogic JMS will attempt to redeliver a message to an application. Once WebLogic JMS fails to redeliver a message to a destination for a specific number of times, the message can be redirected to an error destination that is associated with the message destination. If the redelivery limit is configured but no error destination is configured, persistent and nonpersistent messages are simply deleted when they reach their redelivery limit. A session inherits the redelivery delay and limit from its connection factory when the session is created (Configuration > Default Delivery tab). However, the application that creates the session can then override these connection factory settings by using WebLogic-specific extensions to the javax.jms.Session interface. Regardless of the redelivery delay and limit that are set on the session, the destination where a message is being rolled back or recovered can override the setting by using the Configuration > Delivery Failure tab.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 34

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Message Redelivery

•

•

New messages can be delayed before being eligible for consumption: – A specific amount of time – Until a given time of day A TTD is assigned to messages in one of the following ways: – Programmatically – Using connection factory defaults – Using destination overrides

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

JMS messages include a “Time to Deliver” header field that defines the earliest absolute time at which the message can be delivered. That is, the message is held by the messaging system and is not given to any consumers until that time. If the specified “Time to Live” (TTL) value is less than or equal to the specified TTD value, message delivery succeeds. However, the message is then silently expired. When a producer is created, it inherits its “Time to Deliver” attribute, expressed in milliseconds, from the connection factory used to create the connection that the producer is a part of. Regardless of the value that is set on the producer, the destination to which a message is being sent or published can override the setting. An administrator can set the Time-To-Deliver Override attribute on a destination in either a relative or a scheduled string format. Examples of scheduled TTD values: 0 0 0,30 * * * * : Exact next nearest half-hour * * * 9-16 * * * : Between 9 AM and 5 PM (09:00:00 to 16:59:59) * * * 13-16 * * 0: Between 1 PM and 5 PM on Sunday (13:00:00 to 16:59:59; day 0 = Sunday)

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 35

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Time to Deliver (TTD)

• •

By default, inactive or failed topic consumers will miss any messages produced in their absence. If consumers register as durable and provide a unique client ID, messages will be saved across connections.

Topic

Topic

Topic

ID

ID

Consumer (Durable Subscriber)

Consumer (Durable Subscriber)

Consumer (Durable Subscriber)

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Support for durable subscriptions is a feature that is unique to the publish/subscribe messaging model, so client IDs are used only with topic connections; queue connections also contain client IDs, but JMS does not use them. Nondurable subscriptions only last for the lifetime of their current session with the JMS server. That is, a client will only see the messages published on a topic while it is active. If the subscriber is not active, it is potentially missing messages that are published on its topic. By default, subscribers are nondurable. A durable subscriber, on the other hand, registers a durable subscription with a unique identity that is retained by JMS. Subsequent subscriber objects with the same identity resume the subscription in the state it was left in by the previous subscriber. If there is no active subscriber for a durable subscription, JMS retains the subscriber’s messages until they are received by the subscriber or until they expire.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 36

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Durable Subscriber Review

Delete abandoned subscriptions. Show saved message.

Manually create a subscription.

Is subscriber connected?

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Check WebLogic statistics for topics with high pending counts. This usually indicates that there are topic subscriptions that are not being serviced. There may be a slow or unresponsive consumer client that is responsible for processing the messages. Or it is possible that a durable subscription may no longer be needed and should be deleted, or the messages may be accumulating due to delayed distributed topic forwarding. You can check statistics for individual durable subscriptions on the Administration Console. A durable subscription with a large backlog may have been created by an application but never deleted. Unused durable subscriptions continue to accumulate topic messages until they are either administratively destroyed or unsubscribed by a standard JMS client. 1. Select a topic in a JMS module. 2. Click Monitoring > Durable Subscribers to display the topic’s subscription management options. 3. In addition to monitoring statistics for each subscriber, you can perform the following: - Click New to create a new durable subscriber. - Click Delete to delete specific durable subscribers. - To manage the saved messages for a subscriber, select the check box next to the subscriber’s name, and then click Show Messages to access its message management page.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 37

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Monitoring and Managing Subscriptions

•

Messages are not typically removed from a destination until consumers send acknowledgements back to WLS. – Consumers indicate an acknowledgement mode when connecting to the server.

• • •

Acknowledgement and/or transaction issues can lead to a consumer receiving duplicate messages. A consumer can check if a message has been delivered before with the JMSRedelivered field in the header. If the consumer can handle duplicate messages, but the message redelivery is happening too quickly for the consumer, set the Redelivery Delay to slow down redelivery of the duplicate.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When you create a transacted JMS session, the acknowledge mode is ignored. When an application commits a transaction, all the messages that the application received during the transaction are acknowledged by the messaging system and the messages that it sent are accepted for delivery. If an application rolls back a transaction, the messages that the application received during the transaction are not acknowledged and the messages that it sent are discarded. JMS messages include a standard header field named JMSRedelivered, whose initial value is false. If WebLogic attempts to redeliver a message due to a missing acknowledgement or some other condition, this header is set to true. WebLogic Server also adds a custom message property named JMSXDeliveryCount, which indicates the number of redelivery attempts the server has made for the current message. The first attempt is 1, the next attempt is 2, and so on. WebLogic Server makes a best effort to persist the delivery count so that it does not reset to 1 after a server reboot.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 38

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Duplicate Messages

•

A poison message is any message that the consumer is unable to process. Perhaps the message is: – Corrupt – Not the message type the consumer expected – A duplicate, but the consumer cannot handle duplicates

•

•

The message is not processed, so it is not acknowledged. That causes it to be sent again (and again). This wastes resources. Set attributes to deal with poison messages: – Set Redelivery Limit and Error Destination —

When the Redelivery Limit is reached, the message is sent to the Error Destination rather than to the consumer.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Poison messages are messages that cannot be processed, for whatever reason. Because they are not processed, they are not acknowledged. Because they are not acknowledged, they are re-sent. They can tie up resources being sent again and again. To alleviate this situation, set the Redelivery Limit and an Error Destination. When the message has been redelivered for the number of times specified in the limit, it is sent to the Error Destination rather than to the consumer. Later, messages in the Error Destination can be investigated to see why they could not be processed by the consumer in the usual way.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 39

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Poison Messages

Mode

Description

Auto

Acknowledgements are immediately sent after receiving messages.

Client

Acknowledgements must be sent manually by the application.

Dups OK

Acknowledgements are sent in batches; depending on redelivery settings, duplicate messages can result.

No (WLS only)

Server do not expect acknowledgement and assume that all messages were received successfully.

Transactional

When using transactions, mode settings are ignored and messages are acknowledged as part of a commit.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

In a nontransacted session, the application creating the session programmatically selects one of several available acknowledgement modes: • AUTO_ACKNOWLEDGE: The Session object acknowledges receipt of a message after the receiving application method has returned from processing it. • CLIENT_ACKNOWLEDGE: The Session object relies on the application to call an acknowledge() method on a received message. After the method is called, the session acknowledges all messages received since the last acknowledgement. This mode allows an application to receive, process, and acknowledge a batch of messages with one method call. • DUPS_OK_ACKNOWLEDGE: The Session object acknowledges receipt of a message after the receiving application method has returned from processing it; duplicate acknowledgements are permitted. This mode is most efficient in terms of resource usage, but you should avoid using this mode if your application cannot handle duplicate messages. Duplicate messages may be sent if an initial attempt to deliver a message fails. • NO_ACKNOWLEDGE: Messages sent to this session are immediately deleted from the server. Messages received in this mode are not recovered; as a result, messages may be lost and/or duplicate messages may be delivered if an initial attempt to deliver a message fails. This mode is supported for applications that do not require this quality of service.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 40

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Consumer Acknowledgement Modes

•

By default, consumers may not receive messages in the order in which they were produced due to factors such as: – Message priority, time-to-deliver, and sorting features – Transactional boundaries and rollbacks – Consumer acknowledgement modes

•

This issue can require complex filtering and collating logic in consumers, which can affect performance.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

While the JMS specification provides an ordered message delivery, it does so in a very strict sense. It defines the order between a single instance of a producer and a single instance of a consumer, but does not take into account the following common situations: • Many consumers on one queue • Multiple producers within a single application acting as a single producer • Message recoveries or transaction rollbacks where other messages from the same producer can be delivered to another consumer for processing • Use of filters and destination sort keys As a result, consumers that need to guarantee message ordering must employ some complex and often expensive design patterns, such as: • A dedicated consumer with a unique selector per each subordering • A new destination per subordering, with one consumer per destination

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 41

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Messages Out of Sequence

•

With WebLogic’s “Unit of Order” feature: – Producers provide a unique ID to the server – Multiple producers can use a common ID and act as a single producer – The server ensures that messages are always consumed sequentially in the order in which they were produced

•

UOO IDs can be set programmatically, configured using a connection factory, or assigned automatically. ID

Producer

1

2

Destination

Consumer

1

2

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The WebLogic Server “Unit of Order” feature enables a message producer or group of message producers acting together as one to group messages into a single unit that is processed sequentially in the order in which the messages were created. The message processing of a single message is complete when a message is acknowledged, committed, recovered, or rolled back. Until message processing for a message is complete, the remaining unprocessed messages for that UOO are blocked. UOO is ideal for applications that have strict message-ordering requirements. UOO simplifies administration and application design and in most applications improves performance. Member messages of a UOO are delivered to queue consumers sequentially in the order in which they were created. The message order within a UOO will not be affected by sort criteria, priority, or filters. However, messages that are uncommitted, have a Redelivery Delay, or have an unexpired Time To Deliver will delay messages that arrive after them. A queue that has several messages from the same UOO must finish processing all of them before they can be delivered to any queue consumer or before the next message can be delivered to the queue.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 42

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Unit of Order (UOO): Overview

•

WebLogic’s “Unit of Work” feature has the same benefits as “Unit of Order” with the following additions: – Allows producers to identify groups of messages that must be consumed together as a unit – Ensures that a message group is processed only by a single consumer

•

Unlike UOO, UOW must be explicitly enabled on individual destinations.

Producer

UOW IDs

Consumer Destination Consumer

1

2

3

4

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Many applications need an even more restricted notion of a group than provided by the message UOO feature. If this is the case for your applications, WebLogic JMS provides the “Unit of Work (UOW)” feature. The UOW enables applications to send JMS messages, identifying some of them as a group and allowing a single JMS consumer to process them as such. No message within the UOW will be available to a consumer until all of them are available on the destination. The group of messages will be delivered to the user without interruptions. In other words, all messages in the group will be delivered to the consumer before messages from any other group (or part of no group at all). For WebLogic JMS to identify a message as part of a UOW, the producer must include several message properties. First, each UOW must be assigned its own unique ID. Next, each message that is a component of the UOW must be assigned a sequence number, starting with 1. Finally, the last message in the UOW must declare itself as such. A destination will only process UOW message properties when it has been identified as a terminal destination. This is done using the Unit-of-Work Message Handling Policy field in the console. Optionally, you can also specify an “Expiration Time for Incomplete UOW Messages.” After this amount of time (in milliseconds), any incomplete message groups automatically expire, regardless of each message's time-to-live setting.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 43

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Unit of Work (UOW): Overview

Message-driven EJBs: • Are packaged and deployed as part of EJB applications • Automatically connect to and consume JMS messages • Hide JMS programming from the developer • Are pooled and share JMS client resources for performance Conn. Factory Producers

Destination MDB A Conn. Factory

Producers

Destination MDB B Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

A message-driven bean implements loosely coupled or asynchronous business logic in which the response to a request does not need to be immediate. An MDB receives messages from a JMS queue or topic, and performs business logic based on the message contents. It is an asynchronous interface between EJBs and JMS. The EJB container interacts directly with an MDB, creating bean instances and passing JMS messages to those instances as necessary. The container creates bean instances at deployment time, adding and removing instances during operations based on message traffic. Unlike other EJB types, clients do not directly access MDBs through interfaces. WebLogic Server maintains a free pool where MDB instances that are not currently servicing requests reside. The number of MDB instances in the free pool is determined by the number of available threads in the thread pool, but can also be controlled through EJB deployment descriptors. Each MDB that is deployed to a server instance creates a single JMS connection. In a queue-based JMS application (point-to-point model), each MDB instance has its own session. In a topic-based JMS application (the publish/subscribe model), all local instances of an MDB share a JMS session. A given message is distributed to multiple MDBs, with one copy to each subscribing MDB. If multiple MDBs are deployed to listen on the same topic, each MDB receives a copy of every message. A message is processed by one instance of each MDB that listens to the topic.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 44

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Message-Driven Beans (MDBs): Review

•

MDBs support most standard and WLS-specific JMS consumer features: – – – – – – –

•

Local and remote JMS connections Automatic reconnection after JMS server failure Acknowledgement modes Transactions and batching Durable subscriptions for topics Load balancing and failover for clustered destinations Custom work managers

Configure MDB settings using a combination of code annotations, deployment descriptors, and deployment plans.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

An MDB pool processes each message at least once. Potentially, a message can be processed more than once. A message is redelivered and processed again if an application fails, a transaction rolls back, or the hosting server instance fails during or after the onMessage() method completes, but before the message is acknowledged or committed. Nonpersistent messages are also redelivered in the case of failure, except for the case where the message’s host JMS server shuts down or crashes, in which case the messages are destroyed. To ensure that a message is processed exactly once, use container-managed transactions so that failures cause transactional MDB work to roll back and force the message to be redelivered. WebLogic Server provides a mechanism for grouping onMessage() calls together as a single transaction. This mechanism can help increase database performance of an EJB application by grouping all of the transactions into a single I/O request. Grouping transactions requires fewer transaction logs. WebLogic Server supports migration and recovery for clustered MDB applications. In the event of failure, you can bring a JMS destination and MDBs back online. After an MDB application migrates to another server, it reconnects to the migrated JMS destination and begins to receive messages from the JMS destination again. An MDB automatically detects the JMS server migration target during deployment and uses that as its migratable target. You must ensure that MDBs are deployed everywhere that a JMS Server is deployed.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 45

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

MDB Capabilities

MessageDrivenEJBRuntime Attribute

Description

JMSConnectionAlive Is the MDB currently connected to the destination? ProcessedMessage Count

The total number of messages consumed since the EJB was deployed

1 EJBPoolRuntime Attribute

Description

PooledBeansCurrent Current MDB pool size Count BeansInUseCurrent Count

Number of MDB instances in the pool that are currently processing messages

WaiterCurrentCount

Number of threads currently waiting for an available MDB instance in the pool

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Refer to the online documentation for a list of all available attributes and operations for these MBeans: • EJBRuntimeMBean • • •

MessageDrivenEJBRuntimeMBean (a child of EJBRuntimeMBean) EJBPoolRuntimeMBean (a child of MessageDrivenEJBRuntimeMBean) EJBTransactionRuntimeMBean (a child of MessageDrivenEJBRuntimeMBean)

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 46

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

MDB Runtime Attributes

WLDF Monitor MDB Before/After/Around Message Received

Debug Flag

Description Trigger diagnostic actions each time an MDB in this application processes a message.

Description

DebugEJBPooling

Trace EJB pool utilization.

DebugEJBMDBConnection

Trace JMS (re-)connections from MDB pool.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

In addition to MBean attributes, you can also monitor MDBs by using the WebLogic diagnostics framework or server debug flags. The instrumentation component of the diagnostics framework includes monitors that can trigger actions whenever a MDB receives a message. For example, you can automatically capture stack or thread dumps to troubleshoot MDB problems. WebLogic also includes several debug flags for the EJB subsystem. The two flags that pertain to message driven EJBs are shown in the slide.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 47

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

MDB Diagnostics and Debugging

Which of the following is not a technique for monitoring JMS? a. WLST b. Message Logs c. WLDF Images d. Flow Control e. Console

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: d

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 48

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

Which of these is not an override setting available on JMS destinations? a. Pool Size b. Time to Deliver c. Redelivery Delay d. Time to Live e. Delivery Mode

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: a

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 49

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

Name two techniques to help prevent JMS out-of-memory scenarios. a. Time to Deliver b. Unit or Order c. Quotas d. Durable Subscribers e. Paging

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: c, e

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 50

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

In this lesson, you should have learned how to: • Describe the components of JMS communication • Manage and monitor JMS resources by using the console, WLST, and WLDF • Investigate JMS issues by using logs and debug flags • Discuss some common JMS configuration issues • Configure JMS to handle large message backlogs • Configure JMS to handle failed and expired messages • Troubleshoot problems with message-driven EJBs

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 51

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Summary

This practice covers the following topics: • Analyzing JMS producer and consumer problems • Monitoring JMS communication by using WLST, logs, and debug flags • Configuring delivery failure settings • Responding to failed messages

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 10 - 52

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Practice 10-1 Investigating JMS Problems

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle University and Sentra inversiones y servicios LTDA use only

Troubleshooting Security

After completing this lesson, you should be able to: • Identify several WLS scenarios that involve SSL • Describe the fundamentals of SSL and LDAP communication • Trace SSL and security realm functionality • List some common causes of SSL errors • Work with Java keystore files • Troubleshoot the WLS embedded LDAP

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 2

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Objectives

•

SSL Diagnostics – – – – –

•

Keystores and Keytool Configuration Review Debug Flags Host Name Verification Chain Validation

Security Realm Diagnostics

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 3

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Road Map

SSL: • Is an enhancement to the TCP protocol that secures a point-to-point socket connection • Enables the transmission of sensitive information without the information being read or tampered with by a third party SSL involves the: • Encryption/decryption of data using a randomly generated pair of symmetric keys • Distribution of an asymmetric public key to clients by using a digitally signed certificate • Digital validation of the certificate by using a public key from a trusted certificate authority who issued the certificate

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Secure Sockets Layer (SSL) provides secure connections by allowing two applications connecting over a network to authenticate each other’s identity and by encrypting the data exchanged between the applications. Authentication allows a server and optionally a client to verify the identity of the application on the other end of a network connection. Encryption makes data transmitted over the network intelligible only to the intended recipient. SSL uses public key encryption technology for authentication. With public key encryption, a public key and a private key are generated for a server. Data encrypted with the public key can only be decrypted using the corresponding private key and data encrypted with the private key can only be decrypted using the corresponding public key. The private key is carefully protected so that only the owner can decrypt messages that were encrypted using the public key. The public key is embedded in a digital certificate with additional information describing the owner of the public key, such as name, street address, and email address. A private key and digital certificate provide identity for the server. The data embedded in a digital certificate is verified by a certificate authority and digitally signed with the certificate authority’s digital certificate. Well-know certificate authorities include Verisign and Entrust.net. The trusted certificate authority (CA) certificate establishes trust for a certificate.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 4

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Secure Sockets Layer (SSL): Review

Client

1

Greeting, supported SSL standards Selected SSL standards

3 Obtain public key from cert using trust cert

Server

2

Certificate request Certificate and response

4

Response encrypted with private key

Decrypt response with public key Generate symmetric key and encrypt with public key

5

Symmetric key

6

Decrypt and use new key

SECURE COMMUNICATION Trust Certificates

Server Cert and Private Key Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

1. The SSL session begins with a negotiation between the client and the server as to which cipher suite they will use. This negotiation is known as the SSL handshake. A cipher suite is a set of cryptographic algorithms and key sizes that a computer can use to encrypt data. The cipher suite includes information about the public key exchange algorithms or key agreement algorithms, and cryptographic hash functions. The client tells the server which cipher suites it has available. 2. The server chooses the best mutually acceptable cipher suite. 3. The client requests the server’s certificate, so the server can prove it is legitimate. 4. The server presents its public key certificate to the client. If this certificate is valid, the client can be sure of the identity of the server. 5. The client and server exchange information that allows them to agree on the same secret key. With RSA, the client uses the server’s public key, obtained from the public key certificate, to encrypt the secret key information. The client sends the encrypted secret key information to the server. Only the server can decrypt this message because only it has the private key. 6. Both the client and the server now have access to the same secret key. With each message, they use the cryptographic hash function, chosen in the first step of this process, and shared secret information, to compute a media access control (MAC) address that they append to the message.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 5

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

SSL Communication: Review

ONE-WAY INCOMING SSL Browser

HTTPS

Trust Certs Other Client

HTTPS or T3S

WebLogic Server Cert and Private Key Server SSL port

TWO-WAY INCOMING SSL Trust Certs

Browser

Client Cert and Private Key

Other Client

HTTPS

WebLogic Server Cert and Private Key

HTTPS or T3S

Trust Certs

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

In one-way SSL authentication, the target or server is required to present a digital certificate to the initiator or client to prove its identity. The client verifies that the certificate is trusted and valid. This implies that the certificate was issued by the client’s trusted CA and has not expired. In two-way SSL authentication, both the client and the server must present digital certificates before the SSL connection is established. Therefore, in this case, WebLogic Server not only authenticates itself to the client, which is the minimum requirement for certificate authentication, it also requires authentication from the requesting client. Two-way SSL authentication is useful when you need to restrict access of your resources to trusted clients only. To configure two-way SSL for a server, update its Two Way Client Cert Behavior attribute. The following options are available: • Client Certs Not Requested: The default (meaning one-way SSL). • Client Certs Requested But Not Enforced: Requires a client to present a certificate. If a certificate is not presented, the SSL connection continues. • Client Certs Requested And Enforced: Requires a client to present a certificate. If a certificate is not presented, the SSL connection is terminated.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 6

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

WebLogic SSL Scenarios

ONE-WAY OUTGOING SSL WebLogic

HTTPS or T3S

Trust Certs

WebLogic Other Server

Server Cert and Private Key

ADMIN CHANNEL SSL, SERVER STARTUP Managed Server Trust Certs

Admin Server

T3S

Server Cert and Private Key

Admin port

ADMIN CHANNEL SSL, SERVER ADMINISTRATION WebLogic Trust Certs WLST

T3S

WebLogic Server Cert and Private Key

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

You can define an optional administration port for your domain. When configured, the administration port is used by each managed server in the domain exclusively for communication with the domain’s Administration Server. If an administration port is enabled, WebLogic Server automatically generates an administration channel based on the port settings upon server instance startup. The administration port accepts only secure, SSL traffic, and all connections via the port require authentication. The Administration Server and all managed servers in your domain must be configured with support for the SSL protocol. Managed servers that do not support SSL cannot connect with the Administration Server during startup. You will have to disable the administration port in order to configure them After enabling the administration port, all Administration Console and WLST traffic must connect via the administration port. If you boot managed servers either at the command line or using a start script, specify the administration port in the Administration Server’s URL. The URL must also specify the HTTPS/T3S protocol rather than HTTP/T3. If you use Node Manager for starting managed servers, it is not necessary to modify startup settings or arguments for the managed servers. Node Manager obtains and uses the correct URL to start a managed server.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 7

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

WebLogic SSL Scenarios

PERIMETER SSL

Browser

HTTPS

Web or Proxy Server

HTTP

Server Cert and Private Key

Trust Certs

Two separate SSL sessions

PASS-THROUGH SSL

Browser Trust Certs

WebLogic

HTTPS

Web or Proxy Server

HTTPS

Server Cert and Private Key

WebLogic Server Cert and Private Key

Trust Certs

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

If using a WebLogic Server proxy plug-in, you can configure it to use SSL for communications between the proxy server and WebLogic. The available plug-in settings include: • WLProxySSL: Set this parameter to ON to maintain SSL communication between the plug-in and WebLogic Server when incoming client requests specify the HTTPS protocol • SecureProxy: Set this parameter to ON to enable the use of the SSL protocol for all communication between the plug-in and WebLogic Server. Remember to configure a port on the corresponding WebLogic Server for the SSL protocol before defining this parameter. • TrustedCAFile: Name of the file that contains the digital certificates for the trusted certificate authorities for the plug-in. This parameter is required if the SecureProxy parameter is set to ON. The required format of this file and of the included certificates will vary depending on the proxy vendor. • RequireSSLHostMatch: Determines whether the host name to which the plug-in is connecting must match the Subject Distinguished Name field in the digital certificate of the WebLogic Server to which the proxy plug-in is connecting. WebLogic Server proxy plug-ins do not support two-way SSL. However, the plug-ins can be set up to require the client certificate and pass it on to WebLogic Server. The steps to configure this will vary depending on the proxy vendor.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 8

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Proxy Server SSL Scenarios

• •

The secure storage of private keys and other SSL artifacts is critical to the integrity of your organization. Java applications such as WLS use keystore files (.jks), which provide multiple layers of password-based encryption.

Requires keystore password to access file

Requires alias password to access specific certs/keys

Keystore Alias1

Cert and/or Private Key

Alias2

Cert and/or Private Key

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

A keystore is logically a database of security artifacts that can only be accessed with the proper credentials. Security artifacts are used for a variety of purposes, including authentication and data integrity. There are various types of keystores available, including PKCS12 and Sun’s Java Keystore (JKS) format. WebLogic supports both of these types, but JKS is preferred and PKCS12 support is deprecated as of this release. Generally speaking, keystore information can be grouped into two different categories: key entries and trusted certificate entries. A key entry consists of an entity’s identity and its private key, and can be used for a variety of cryptographic purposes. In contrast, a trusted certificate entry only contains a public key in addition to the entity’s identity. Thus, a trusted certificate entry cannot be used where a private key is required. For the JKS type, a keystore may contain both key entries and trusted certificate entries. Demonstration keystores with digital certificates, private keys, and trusted CA certificates are found in the WL_HOME\server\lib directory. These demonstration security artifacts should be used in a development environment only. Because the digital certificates and trusted CA certificates in the demonstration keystores are signed by a WebLogic Server demonstration certificate authority, a WebLogic Server installation using the demonstration keystores will trust any WebLogic Server installation that also uses the demonstration keystores. You want to create a secure environment where only your installations trust each other.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 9

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Keystore: Review

• •

Web browsers maintain a repository of trust certificates (public keys) from popular authorities. Java applications use a separate keystore file to manage trust certificates, which: – Is bundled with the JDK and is named cacerts – Includes many of the same entries found in Web browsers – Uses the password changeit

•

By default, WLS uses the JDK trust keystore, but a custom keystore can be used instead.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

A truststore is a keystore that is used when making decisions about what to trust. If you receive some data from an entity that you already trust, and if you can verify that the entity is the one it claims to be, then you can assume that the data really came from that entity. An entry should be added to a truststore only if the user makes a decision to trust that entity. By either generating a key pair or by importing a certificate, the user has given trust to that entry, and thus any entry in the keystore is considered a trusted entry. A file named cacerts resides in the JVM runtime’s security properties directory, jre/lib/security. The cacerts file represents a system-wide JKS keystore with CA certificates. System administrators can configure and manage that file by using Keytool. The cacerts keystore file ships with several common root CA certificates, each with an alias and an X.500 owner distinguished name. The initial password of the cacerts keystore file is changeit. System administrators should change that password and the default access permission of that file when installing the SDK. Because you trust the CAs in the cacerts file as entities for signing and issuing certificates to other entities, you must manage the cacerts file carefully. The cacerts file should contain certificates of only those CAs that you trust. It is your responsibility to verify the trusted root CA certificates bundled in the cacerts file and make your own trust decisions.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 10

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Trust Keystores

JDKs include a command-line tool to create, view, and modify keystore files. Generate a new self-signed certificate and private key and add it to a store: keytool –genkeypair –alias mykey –keypass mykeypass –keyalg RSA –keysize 512 -dname "CN=payroll.mycompany.com..." -keystore mykeys.jks –storepass mypass Import a signed certificate from a CA into a store: keytool –importcert –file payroll.pem –alias mykey –keypass mykeypass -keystore mykeys.jks –storepass mypass Inspect the contents of a store: keytool –list –v -keystore mykeys.jks –storepass mypass

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Keytool is a key and certificate management utility. It allows users to use digital signatures to administer their own public-private key pairs and associated certificates for use in selfauthentication (where users authenticate themselves to other users or services) or data integrity and authentication services. It also allows users to cache the public keys (in the form of certificates) of their communicating peers. Keytool also enables users to administer secret keys used in symmetric encryption/decryption (for example, DES). Keytool stores the keys and certificates in a keystore. Available commands include: • -genkeypair: Generates a key pair (a public key and associated private key). Wraps the public key into an X.509 v3 self-signed certificate, which is stored as a singleelement certificate chain. This certificate chain and the private key are stored in a new keystore entry identified by alias. keyalg specifies the algorithm to be used to generate the key pair, and keysize specifies the size of each key to be generated. sigalg specifies the algorithm that should be used to sign the self-signed certificate; this algorithm must be compatible with keyalg. dname specifies the X.500 distinguished name to be associated with alias, and is used as the issuer and subject fields in the self-signed certificate. If no distinguished name is provided at the command line, the user will be prompted for one.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 11

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Keytool: Review

•

•

-importcert: Reads the certificate or certificate chain (in which the latter is supplied in a PKCS#7 formatted reply or a sequence of X.509 certificates) from the file cert_file and stores it in the keystore entry identified by alias. If no file is given, the certificate or certificate chain is read from standard in (“standard input” or “standard input stream”). Keytool can import X.509 v1, v2, and v3 certificates, as well as PKCS#7-formatted certificate chains consisting of certificates of that type. The data to be imported must be provided either in binary encoding format or in printable encoding format (also known as Base64 encoding) as defined by the Internet RFC 1421 standard. In the latter case, the encoding must be bounded at the beginning by a string that starts with -----BEGIN and bounded at the end by a string that starts with -----END. If the alias does not point to a key entry, then Keytool assumes you are adding a trusted certificate entry. In this case, the alias should not already exist in the keystore. If the alias does already exist, Keytool outputs an error (because there is already a trusted certificate for that alias) and does not import the certificate. If the alias points to a key entry, Keytool assumes you are importing a certificate reply. When importing a certificate reply, the certificate reply is validated using trusted certificates from the keystore, and optionally using the certificates configured in the cacerts keystore file (if the -trustcacerts option was specified). If the reply is a single X.509 certificate, Keytool attempts to establish a trust chain, starting at the certificate reply and ending at a self-signed certificate (belonging to a root CA). The certificate reply and the hierarchy of certificates used to authenticate the certificate reply form the new certificate chain of the alias. If a trust chain cannot be established, the certificate reply is not imported. In this case, Keytool does not print out the certificate and prompt the user to verify it, because it is very hard (if not impossible) for a user to determine the authenticity of the certificate reply. -list: Prints to standard output the contents of the keystore entry identified by alias. If no alias is specified, the contents of the entire keystore are printed. This command by default prints the MD5 fingerprint of a certificate. If the -v option is specified, the certificate is printed in human-readable format, with additional information such as the owner, issuer, serial number, and any extensions. If the -rfc option is specified, the certificate contents are printed using the printable encoding format, as defined by the Internet RFC 1421 standard. -delete: Deletes from the keystore the entry identified by alias. The user is prompted for the alias if no alias is provided at the command line.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 12

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

•

• •

•

WebLogic Server supports both the SSL v3.0 and Transport Layer Security (TLS) v1.0 protocols. When WebLogic Server acts as an SSL server, it agrees to use either of the two protocols that the client specifies as preferred. When WebLogic Server acts as an SSL client, it specifies TLS v1.0 as the preferred protocol.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

WebLogic Server supports both the SSL V3.0 and TLS V1.0 protocols. When WebLogic Server is acting as an SSL server, the protocol that the client specifies as preferred in its client hello message is used. Note that WebLogic Server does not support SSL V2.0. When WebLogic Server is acting as an SSL client, it specifies TLS1.0 as the preferred protocol in its SSL V2.0 client hello message. But it can use SSL V3.0 as well if that is the highest version that the SSL server on the other end supports. The peer must respond with an SSL V3.0 or TLS V1.0 message; otherwise, the SSL connection is dropped. Although the SSL V3.0 protocol is acceptable in most cases, some circumstances (compatibility, SSL performance, and environments with maximum security requirements) make the TLS V1.0 protocol more desirable. The weblogic.security.SSL.protocolVersion command-line argument lets you specify which protocol is used for SSL connections. The SSL V3.0 and TLS V1.0 protocols cannot be interchanged. Use the TLS V1.0 protocol only if you are certain that all desired SSL clients are capable of using the protocol.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 13

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

WebLogic SSL Support

Perform the following for each server: 1. Enable an SSL port (and optionally an admin port). 2. Set identity and trust keystore file locations and passwords. 3. Set the alias/password of the certificate/key to use in the identify keystore file.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Before you configure WebLogic Server for SSL, you must do the following: • Obtain private keys and digital certificates from a reputable certificate authority such as Verisign, Inc., or Entrust.net. • Create identity and trust keystore files by using tools like Keytool. • Load the private keys and trusted CAs into the keystores. Register your keystores with WLS. Click the name of the server for which you want to configure the identity and trust keystores. Select Configuration > Keystores. Next to the Keystores information, click the Change button. When the Keystores drop-down list is shown, select the method for storing and managing private keys/digital certificate pairs and trusted CA certificates. Then click Save. The options for the Keystores field are: • Demo Identity and Demo Trust: The demonstration identity and trust keystores, located in the MIDDLEWARE_HOME\server\lib directory and the JDK cacerts keystore, are configured by default. Use for development only. • Custom Identity and Java Standard Trust: A keystore that you create and the trusted CAs defined in the cacerts file in the JAVA_HOME\jre\lib\security directory • •

Custom Identity and Custom Trust: Identity and trust keystores that you create Custom Identity and Command Line Trust: An identity keystore that you create and command-line arguments that specify the location of the trust keystore Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 14

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

SSL Configuration: Review

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

To configure SSL settings for a server, use its Configuration > SSL tab. The available fields include: • Private Key Alias: The keystore attribute that defines the string alias used to store and retrieve the server’s private key • Private Key Passphrase: The keystore attribute that defines the passphrase used to retrieve the server’s private key • Export Key Lifespan: Indicates the number of times WebLogic Server can use an exportable key between a domestic server and an exportable client before generating a new key. The more secure you want WebLogic Server to be, the fewer times the key should be used before generating a new key. • Use Server Certs: Sets whether WLS should use the server certificates/key as the client identity when initiating a two-way outgoing connection over SSL • Two Way Client Cert Behavior: By default, WebLogic Server is configured to use oneway SSL (implied by the Client Certs Not Requested value). Selecting one of the other available options enables two-way SSL. Other fields are only for backward compatibility when not using keystores (for example, Private Key File Name, Server Certificate File Name, and Trusted CA File Name).

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 15

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

SSL Configuration: Review

WLS allows you to dynamically stop and restart the SSL subsystem and port after configuration changes. 1

2

Restart SSL using WLST: serverRuntime() cmo.restartSSLChannels()

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

All the server SSL attributes are dynamic; when modified via the console or WLST, they cause the corresponding SSL server or channel SSL server to restart and use the new settings for new connections. Old connections will continue to run with the old configuration. To ensure that all the SSL connections exist according to the specified configuration, you must reboot WebLogic Server. Using the console, select a specific server and then select Control > Start/Stop tab. Then click the Restart SSL button found at the bottom of the page. The corresponding MBean operation is ServerRuntime.restartSSLChannels(), which can be invoked using WLST.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 16

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Restarting SSL

Flag

Description

DebugSecuritySSL

Trace initialization of Java SSL library and SSL channels. View available SSL ciphers. Trace loading of identity and trust keystores. View metadata for each keystore entry. Trace SSL handshake and data communications.

DebugSecuritySSLEaten

View certain benign Java SSL exceptions that are otherwise ignored by WLS.

DebugSecurityKeyStore

View the values of WLS keystore configuration parameters.

Server log messages with DebugSecuritySSL enabled:

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

SSL debugging provides more detailed information about the SSL events that occurred during an SSL handshake. The SSL debug trace displays information about trusted certificate authorities, SSL server configuration information, server identity keystores, the encryption strength that is allowed, the enabled ciphers, the SSL communication during a handshake, and additional information about any SSL failures detected by WebLogic Server. WebLogic Server allows SSL sessions to be cached. Those sessions live for the life of the server. Clients that use SSL sockets directly can control the SSL session cache behavior. The SSL session cache is specific to each SSL context. All SSL sockets created by SSL socket factory instances returned by a particular SSL context can share the SSL sessions. Clients default to resuming sessions at the same IP address and port. Multiple SSL sockets that use the same host and port share SSL sessions by default, assuming the SSL sockets are using the same underlying SSL context. SSL sessions exist for the lifetime of the SSL context; they are not controlled by the lifetime of the SSL socket. Therefore, creating a new SSL socket and connecting to the same host and port used by a previous session can resume a previous session as long as you create the SSL socket by using an SSL socket factory from the SSL context that has the SSL session in its cache. By default, clients that use HTTPS URLs get a new SSL session for each URL because each URL uses a different SSL context and, therefore, SSL sessions cannot be shared or reused.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 17

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

SSL Debug Flags

Normal SSL handshake log messages with DebugSecuritySSL enabled: Messages include an internal SSL session ID. Greeting ... Certificate exchange ... Protocol negotiation ... Secure session established

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

1. The client sends the server information, including the highest version of SSL that it supports and a list of the cipher suites that it supports. (TLS 1.0 is indicated as SSL 3.1.) The cipher suite information includes cryptographic algorithms and key sizes. 2. The server chooses the highest version of SSL and the best cipher suite that both the client and server support, and sends this information to the client. 3. The server sends the client a certificate or a certificate chain. A certificate chain typically begins with the server’s public key certificate and ends with the certificate authority’s root certificate. If the server needs to authenticate the client (two-way SSL), it sends the client a certificate request. 4. The server tells the client that it is finished with its initial negotiation messages. 5. The client generates information used to create a key to use for symmetric encryption. For RSA, the client then encrypts this key information with the server’s public key and sends it to the server. 6. The client sends a message telling the server to change to encrypted mode. 7. The client tells the server that it is ready for secure data communication to begin. 8. The server tells the client that it is ready for secure data communication to begin. This is the end of the SSL handshake.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 18

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

SSL Handshake Trace

Normal SSL data transfer log messages with DebugSecuritySSL enabled: Embedded LDAP tab. 3. Edit embedded LDAP options, including: - Backup Hour, Backup Minute: These two values are used in conjunction to determine the time of day at which the embedded LDAP server data files are backed up. At the specified time, WLS suspends writes to the embedded LDAP server, creates a zip file of the latest LDAP data found in the server’s data/ldap/ldapfiles directory, and records it to the server’s data/ldap/backup directory. -

Backup Copies: The maximum number of backup copies that should be made for the embedded LDAP server. This limits the number of zip files that are created. A value of 0 disables the backup feature.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 33

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Embedded LDAP Backups

If a managed server’s security behavior does not seem to correspond to the latest changes, try the following: • Force an LDAP refresh at startup. • Delete the LDAP replica from the file system.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The most common LDAP synchronization problem occurs when you change the domain’s administrative account while a managed server is not running. Consequently, when the managed server later boots and cannot locate the correct credentials in its LDAP replica, it fails to start. You can configure the replication behavior of the embedded LDAP server. Select the name of the domain in the Administration Console. Then click the Security > Embedded LDAP Server tab and use these attributes: • Refresh Replica At Startup: Specifies whether the embedded LDAP server in a Managed Server should refresh all replicated data at boot time. This setting is useful if you have made many changes when the Managed Server was not active, and you want to download the entire replica instead of having the Administration Server push each change to the Managed Server. • Master First: Specifies whether a Managed Server should always connect to the embedded LDAP server on the Administration Server, instead of connecting to the local replicated LDAP server

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 34

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Embedded LDAP Synchronization Issues

1. Update the default generated LDAP password. 2. Obtain an LDAP browsing/editing tool. 3. Connect to the server’s default port (or SSL port). a. Set the base distinguished name (DN) to your domain name. b. Set the user’s DN to “Admin.”

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When you delete a WebLogic Security provider, the security data in the embedded LDAP server is not automatically deleted. The security data remains in the embedded LDAP server in case you want to use the provider again. Use an external LDAP browser to delete the security data from the embedded LDAP server. To view the contents of the embedded LDAP server through an LDAP browser: 1. In the WLS console, change the Credential for the embedded LDAP server. Select your domain name and then click the Security > Embedded LDAP tab. Reboot WebLogic Server. Note that changing the credential can affect the operation of the domain. Do not perform this step on a production server. 2. Download and install an external LDAP browser. You can find one LDAP browser at the following location: http://www.openldap.org/. 3. Launch the LDAP browser and supply the host name and port of your WebLogic Server. Set the base DN to dc=, where is the name of the WebLogic Server domain that you are using. Set the user DN to cn=Admin. Set the password to the credential you specified earlier. 4. Use the LDAP browser to navigate the hierarchy of the embedded LDAP server, including users, groups, roles, and policies.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 35

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Viewing Embedded LDAP Contents

•

LDAP: – Is a TCP/IP protocol – Provides a hierarchical lookup and search service – Models information as a tree of entries, whose attributes are defined by a schema or “object class” – Defines default schemas for common entities such as people and groups – Supports SSL

•

Entries: – Identify their locations in the tree by using a distinguished name (DN) – Can be referrals that link to other LDAP servers

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The Lightweight Directory Access Protocol (LDAP) is based on the X.500 standard but is significantly simpler and more readily adapted to meet custom needs. Unlike X.500, LDAP supports TCP/IP, which is necessary for Internet access. The core LDAP specifications are all defined in Request for Comments (RFCs). LDAP is a protocol that provides access to a compliant directory via TCP/IP. The strengths of LDAP-compliant directories include speed, simplicity, and the ability to be replicated and distributed across several servers. An LDAP directory can be used to store a great deal of information, from user login credentials to company telephone directories. Unlike databases that are designed for processing hundreds or thousands of changes per minute, LDAP directories are heavily optimized for read performance. LDAP is intentionally designed for environments where search operations are much more common than modify operations. LDAP Version 3 implements a referral mechanism that allows servers to return references to other servers as a result of a directory query. This makes it possible to distribute directories globally by partitioning a directory information tree (DIT) across multiple LDAP servers.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 36

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

LDAP Concepts

Domain or organization

myldap.com Organizational unit

Employees

Mike

Mark

Contractors

Mary

Mimi

Mike

Person DN: uid=mike, ou=Contractors, o=myldap.com

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Like a computer’s file system, directories are viewed as a tree. Each entry in a directory is called an object. These objects are of two types: containers and leaves. A container is like a folder; it contains other containers or leaves. A leaf is simply an object at the end of a tree. A tree cannot contain an arbitrary set of containers and leaves. It must match the schema defined for the directory. The top level of the LDAP directory tree is the base, referred to as the base DN. A base DN can be one of several forms. Here are some examples: • A domain name, broken into components (dc=Acme,dc=com) • An organization name (o=Acme Corp) • An organization name along with a country (o=Acme Corp,c=India) Organizational units are standard LDAP object classes that act as containers for other entries. The identifying attribute for an organizational unit is ou. The standard LDAP schema also defines a person class and a group class, which is a collection of people. The person type also includes such other attributes as Common Name (cn), Unique Identifier (uid), Last Name (sn), and Password (userpassword).

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 37

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

LDAP Structure

Searching for LDAP entries involves: 1. The base DN from which to start searching 2. A search filter that specifies the: – Search criteria in terms of attribute values – The type or “object class” of the desired entries

3. An indication whether or not the search should include any child entities An LDAP search filter that finds all people whose user ID begins with “m,” while ignoring those whose name is “Mike”: (&(uid=m*)(!cn=Mike*)((objectclass=person))

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The “&” represents a logical “and” when combining multiple expressions that have been grouped together in parentheses. Similarly, the “|” represents a logical “or,” and a “!” represents a logical “not.” Search filters can specify one or more object classes. Here is an example: (&(&(objectClass=person)(objectClass=organizationalPerson))(objectCl ass=user))

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 38

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

LDAP Search Operations

1. 2. 3. 4.

Shut down the domain. Back up the master LDAP data. Run the AdminAccount command-line tool. Remove the admin server’s data/ldap/DefaultAuthenticatorInit.initia lized file.

5. Restart the Administration Server and enter the new password (remove any boot.properties file temporarily). 6. Set the original password by using the console or WLST. java weblogic.security.utils.AdminAccount weblogic temppass1 /home/oracle/domains/mydomain/security

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

WebLogic Server provides a method of resetting the administrative password if the password becomes lost. This tool re-creates part of the domain’s LDAP structure. However, it is only effective on the default administrative user assigned to the Admin role. To preserve additional LDAP customizations, perform an export of the realm’s providers prior to these steps by using the console or WLST. After these steps are complete, use the console or WLST to import the data back into the reset realm.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 39

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Resetting Admin Password in Embedded LDAP

• •

Each server in a domain caches security data retrieved from the realm’s database store. If configured, the admin server can use JMS to broadcast notifications to managed servers: – When security data changes – To force them to refresh their caches

•

If notifications are not configured, managed servers may temporarily use old security data.

Cache Admin Server

Cache

JMS

Managed Server

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

JMS notifications enable the security data (which is contained in the RDBMS security store and managed by security providers in the realm) to be synchronized among all server instances in the domain. If you do not configure a JMS topic that can be used by the RDBMS security store when configured in a multi-server or clustered domain, care should be taken when making security policy or security configuration updates. If no JMS topic is configured, it may be necessary to reboot the domain to ensure that all server instances function consistently with those security updates. To configure JMS notifications: 1. Create a new JMS server, JMS module, and JMS topic. 2. Edit your security realm in the console. 3. Click the Configuration > RDBMS Security Store tab. 4. Edit the fields in the section labeled Server Synchronization Configuration. These fields include JNDI User Name, JNDI Password, JMS Topic, JMS Topic Connection Factory, and Notification Properties.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 40

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Database Store Cache Synchronization Issues

The WebLogic auditing provider: • Creates a detailed record of all security changes and decisions within a domain (DefaultAuditRecorder.log) • Can also create a record of all domain configuration changes • Is not enabled by default

Security Events Configuration Events

Auditing Provider Audit Log

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Auditing is the process whereby information about operating requests and the outcome of those requests are collected, stored, and distributed for the purposes of nonrepudiation. In WebLogic Server, an auditing provider implements this electronic trail of computer activity. WebLogic Server includes a sample auditing provider, but, by default, it is not activated for a new security realm. The WebLogic auditing provider records information from a number of security requests, which are determined internally by the WebLogic Security Framework. The WebLogic auditing provider also records the event data associated with these security requests and the outcome of the requests. You can also configure the Administration Server to emit audit messages that enable tracking of configuration changes in a domain. This provides a record of changes made to the configuration of any resource within a domain, as well as invocations of management operations on any resource within a domain. Configuration audit records can be saved to a log file, sent to an auditing provider in the security realm, or both. All auditing information recorded by the WebLogic auditing provider is saved in /servers//logs/DefaultAuditRecorder.log by default. Although an auditing provider is configured for a security realm, each server writes auditing data to its own log file in the server directory.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 41

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Auditing Provider

•

Typical security events include: – An authentication or identity assertion attempt – A new role or policy – A locked/unlocked user account

•

Security events have the following characteristics: – Name – Severity (Warning, Error, Success, and so on) – Zero or more context attributes: — — — —

Protocol, port, address HTTP headers EJB method parameters SAML tokens

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

In addition to the events listed in the slide, the default WebLogic auditing provider records the following types of security events: • The lock on a user account expires. • A security policy is used and an authorization decision is made. • A role definition is used. • A role or policy is removed or “undeployed.” The WebLogic auditing provider audits security events of the specified severity, as well as all events with a higher numeric severity rank. For example, if you set the severity level to ERROR, the WebLogic auditing provider audits security events of severity level ERROR, SUCCESS, and FAILURE. You can also set the severity level to CUSTOM, and then enable the specific severity levels that you want to audit, such as ERROR and FAILURE events only. An audit event includes a context object that can hold a variety of different attributes, depending on the type of event. When you configure an auditing provider, you specify which context attributes are recorded for each event. By default, no context attributes are audited.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 42

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Security Audit Events

1 3 2

Context attributes to record

Minimum severity to record

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

1. In the left pane, select Security Realms and click the name of the realm you are configuring (for example, myrealm). Next, click the Providers tab in the right pane. Then click the tab for auditing providers and click New. 2. Select the new provider and click the Configuration > Provider-Specific tab. 3. Update the following fields if desired: - Active Context Handler Entries: Specifies which context attributes are recorded by the auditing provider, if present within an event. Use the arrow buttons to move the available entries to the Chosen list. - Rotation Minutes: Specifies how many minutes to wait before creating a new audit log file. After this time has elapsed, the audit file is closed and a new one is created. - Severity: The minimum severity level of an event to record

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 43

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring the Auditing Provider

Flag

Description

DebugSecurityRealm

Trace the initialization of the realm’s providers and the loading of initial data from the default store.

DebugSecurityAtn

Trace the authentication and management of users and groups.

DebugSecurityRoleMap

Trace role policy evaluations and results.

DebugSecurityAtz

Trace authorization policy evaluations and access decisions.

DebugSecurityAdjudicator

Trace final authorization decisions.

DebugSecurityUserLockout

Trace the locking/unlocking of user accounts based on the number of invalid login attempts.

DebugSecuritySAML*

Trace the processing and/or generation of SAML tokens.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 44

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Realm Debug Flags

Normal authentication log messages with DebugSecurityAtn enabled: ... Provider and control flag being used ... ... Search for ... user in LDAP. Search for ... groups in LDAP. ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

In WebLogic Server, authentication providers are used to prove the identity of users or system processes. Authentication providers also remember, transport, and make identity information available to various components of a system (via subjects) when needed. During the authentication process, a principal validation provider provides additional security protections for the principals (users and groups) contained within the subject by signing and verifying the authenticity of those principals. Whether the client is an application, applet, Enterprise JavaBean (EJB), or servlet that requires authentication, WebLogic Server uses the Java Authentication and Authorization Service (JAAS) classes to reliably and securely authenticate to the client. JAAS implements a Java version of the Pluggable Authentication Module (PAM) framework, which permits applications to remain independent from underlying authentication technologies. Therefore, the PAM framework allows the use of new or updated authentication technologies without requiring modifications to your application. A LoginModule does much of the work of authentication. All LoginModules are responsible for authenticating users within the security realm and for populating a subject with the necessary principals (users/groups). LoginModules that are not used for perimeter authentication also verify the proof material submitted (for example, a user’s password).

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 45

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Typical Authentication Trace

Continued:

Create internal user identity data structure (“subject”).

... ... ... ...

Digitally sign identities to validate during subsequent requests.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Authentication providers rely on principal validation providers to sign and verify the authenticity of principals (users and groups) contained within a subject. Such verification provides an additional level of trust and may reduce the likelihood of malicious principal tampering. Verification of the subject’s principals takes place when accepting incoming requests from a previously authenticated RMI client. The authenticity of the subject’s principals is also verified when making authorization decisions. As part of a successful authentication, principals are signed and stored in a subject for future use. A principal validation provider signs principals, and an authentication provider’s LoginModule actually stores the principals in the subject. Later, when a caller attempts to access a principal stored within a subject, a principal validation provider verifies that the principal has not been altered since it was signed, and the principal is returned to the caller (assuming all other security conditions are met). A principal validation provider is a special type of security provider that primarily acts as a “helper” to an authentication provider. The main function of a principal validation provider is to prevent malicious individuals from tampering with the principals stored in a subject.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 46

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Typical Authentication Trace

Normal log messages with DebugSecurityRoleMap: Resource being ... true> ... authorization ...

Evaluate conditions for all roles in scope.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Role mapping is the process whereby principals (users or groups) are dynamically mapped to security roles at run time. In WebLogic Server, a Role Mapping provider determines what security roles apply to the principals stored in a subject when the subject is attempting to perform an operation on a WebLogic resource. Because this operation usually involves gaining access to the WebLogic resource, Role Mapping providers are typically used with authorization providers. The WebLogic Security Framework calls each Role Mapping provider that is configured for a security realm as part of an authorization decision. The result of the dynamic security role computation (performed by the Role Mapping providers) is a set of security roles that apply to the principals stored in a subject at a given moment. These security roles can then be used to make authorization decisions for protected WebLogic resources, as well as for resource container and application code. The role security policies are represented as a set of expressions or rules that are evaluated to determine whether a given security role is to be granted. These rules may require the Role Mapping provider to substitute the value of context information obtained as parameters into the expression. In addition, the rules may also require the identity of a user or group principal as the value of an expression parameter.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 47

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Typical Role Mapping Trace

Normal log messages with DebugSecurityAtz:

Type of authorization policy (role-based)

... Resource being true> Evaluate policy ... conditions and

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Authorization is the process whereby the interactions between users and WebLogic resources are controlled, based on user identity or other information. In other words, authorization answers the question, “What can you access?” In WebLogic Server, an authorization provider is used to limit the interactions between users and WebLogic resources to ensure integrity, confidentiality, and availability. Like LoginModules for authentication providers, an access decision is the component of an authorization provider that actually answers the question, “Is access allowed?” Specifically, an access decision is asked whether a subject has permission to perform a given operation on a WebLogic resource with specific parameters in an application. Given this information, the access decision responds with a result of PERMIT, DENY, or ABSTAIN. The WebLogic Security Framework delegates the job of reconciling any discrepancies among the results rendered by the configured authorization providers’ access decisions to the adjudication provider. The adjudication provider determines the ultimate outcome of the authorization decision. The adjudication provider returns either a TRUE or FALSE verdict, which is forwarded to the resource container through the WebLogic Security Framework.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 48

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Typical Authorization Trace

•

When authentication debugging is enabled, any LDAP providers also generate an ldap_trace.logATN file.

•

This log includes details about all communication between the server and external LDAP systems.

Connected to ldaps://ldap.mycompany.com:389 BindRequest {version=3, name=cn=LDAPAdmin, authentication=**} BindResponse {resultCode=0} ... SearchRequest {baseObject=dc=mycompany,dc=com, scope=2, derefAliases=0,sizeLimit=0, timeLimit=0, attrsOnly=false, filter=(&(uniquemember=uid=weblogic,dc=mycompany,dc=com) (objectclass=groupofuniquenames)), attributes=cn} SearchResult {resultCode=0} SearchResponse {entry='cn=Operators,dc=mycompany,dc=com', attributes='LDAPAttribute {type='cn', values='Operators'}'}

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

This internal log file is undocumented and therefore its availability, location, and contents are subject to change in all future WebLogic Server revisions. Currently this log file is generated at your domain’s root directory.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 49

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

LDAP Trace Log

•

Multiple providers of the same type are executed in the order in which they are defined. Control flags also determine the processing logic as each provider is executed.

•

Flag

Success Action

Failure Action

REQUIRED

Execute next provider.

Execute next provider, but outcome is still failure.

REQUISITE

Execute next provider.

Do not execute next provider.

SUFFICIENT

Do not execute next provider.

Execute next provider.

OPTIONAL

Execute next provider.

Execute next provider.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Each security realm must have at least one authentication provider configured. The WebLogic Security Framework supports multiple authentication providers for multipart authentication. Therefore, you can use multiple authentication providers as well as multiple types of authentication providers in a security realm. The order in which WebLogic Server calls multiple authentication providers can affect the overall outcome of the authentication process. The authentication providers table lists the authentication providers in the order in which they will be called. By default, authentication providers are called in the order in which they were configured. Use the Reorder button on the Security Realms > Providers > Authentication page in the Administration Console to change the order in which authentication providers are called by WebLogic Server and listed in the console. When you configure multiple authentication providers, you also use the control flag for each provider to control how the authentication providers are used in the login sequence. When additional authentication providers are added to an existing security realm, by default the control flag is set to Optional. If necessary, change the setting of the control flag and the order of authentication providers so that each authentication provider works properly in the authentication sequence.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 50

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Authentication Provider Control Flags

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

The overall authentication succeeds only if all Required and Requisite providers succeed. If a Sufficient provider is configured and succeeds, then only the Required and Requisite providers prior to that Sufficient provider need to have succeeded for the overall authentication to succeed. If no Required or Requisite providers are configured for an application, then at least one Sufficient or Optional provider must succeed.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 51

WebLogic Server includes: • A base LDAP Authenticator that can be configured to support any compliant vendor • Vendor-specific LDAP authenticators, whose attributes are set to vendor-specific defaults for convenience

LDAP Authenticator

OpenLDAP Authenticator

Oracle Internet Directory Authenticator

Active Directory Authenticator

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Each LDAP authentication provider stores user and group information in an external LDAP server. The providers differ primarily in how they are configured by default to match typical directory schemas for their corresponding LDAP server. For example, the generic authenticator is configured to use a person’s common name (cn) as a user ID, while by default Oracle Internet Directory uses the uid attribute for this purpose. Similarly, the names of object classes used to represent people or groups may vary from vendor to vendor. For example, the generic authenticator is configured to use the object class groupofuniquenames, while by default Oracle Internet Directory uses the object class groupofnames. WebLogic Server does not support or certify any particular LDAP servers. Any LDAP v2- or v3-compliant LDAP server should work with WebLogic Server. If an LDAP authentication provider is the only configured authentication provider for a security realm, you must have the Admin role to boot WebLogic Server and use a user or group in the LDAP directory. If the LDAP user who boots WebLogic Server is not properly added to a group that is assigned to the Admin role, and if the LDAP authentication provider is the only authentication provider with which the security realm is configured, WebLogic Server cannot be booted.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 52

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

External LDAP Authentication Providers

• • • •

Specify the LDAP host and connection credentials. Indicate whether or not SSL should be used. Optionally, specify connection retry and failover behaviors. Supply the following for users and for groups: – – – – –

•

Base DN to start searching from Object class name Attribute name Search filter expression for all entries Search filter expression for a specific entry

For groups, additionally specify: – Membership attribute – Whether membership is static or dynamic

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

• • • • •

•

•

Host: A single host name or IP address, or a comma-separated list of host names and ports to try Port: The port number of the LDAP server Principal: The distinguished name (DN) of the LDAP user that WebLogic Server should use to connect to the LDAP server Credential: The credential (usually a password) used to connect to the LDAP server SSL Enabled: Specifies whether the SSL protocol should be used when connecting to the LDAP server. For a more secure deployment, Oracle recommends using the SSL protocol to protect communications between the LDAP server and WebLogic Server. Connect Timeout: Specifies the maximum number of seconds to wait for the connection to the LDAP server to be established. If it is set to 0, there is no maximum time limit, and WebLogic Server waits until the TCP/IP layer times out to return a connection failure. Dynamic Member URL Attribute: The attribute of the dynamic LDAP group object that specifies the URLs of the members of the dynamic group

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 53

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

LDAP Provider Configuration: Overview

User Dynamic Group DN Attribute: A user attribute indicating its dynamic group membership. If such an attribute does not exist, the provider determines if a user is a member of a group by evaluating the URLs on the dynamic group. If a group contains other groups, WebLogic Server evaluates the URLs on any of the descendants (subgroups) as well.

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

•

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 54

Location in tree to start searching from

How to retrieve all groups? How to retrieve a group given its name? Entity attribute that contains group name Schema used to model a group Entity attribute that contains members

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Many LDAP servers have a concept of dynamic (or virtual) groups. These are groups that, rather than consisting of a list of users and groups, contain some policy statements, queries, or code that define the set of users that belong to the group. Even if a group is marked “dynamic,” users must log out and log back in before any changes in their group memberships take effect. The term dynamic describes the means of defining the group and not any runtime semantics of the group within WebLogic Server. • Group Base DN: The base distinguished name (DN) of the tree in the LDAP directory that contains group definitions • All Groups Filter: An LDAP filter expression for finding all groups beneath the base group distinguished name (DN). If a filter is not specified, a simple default search filter is created based on the group object class. • Group From Name Filter: An LDAP filter expression for finding a group given the name of the group. If a filter is not specified, a simple default search filter is created based on the group schema. • Static Group Name Attribute: The attribute of a group object that specifies the name of the group • Static Member DN Attribute: The attribute of a group object that specifies the distinguished names (DNs) of the members of the group . Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 55

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

LDAP Provider Configuration: Overview

Typical causes include: • Connection credentials do not have all the necessary permissions • Wrong base DN, object class, or attribute for users or groups • Configured search filter is valid but fails to retrieve any users or groups • Insufficient maximum level for nested group memberships (only some members are retrieved) • WLS does not trust the LDAP server’s SSL certificate: – CA not in the trust keystore – Host name verification fails.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

After the user is authenticated, a search on the groups occurs to get the list of groups to which this user belongs, and this used to be able to do the role mapping between groups and roles. If the user does not belong to any group or if the search criteria are not valid, you will see debug messages similar to the following: The Max Group Membership Search Level provider field specifies how many levels of group membership can be searched. This setting is valid only if Group Membership Searching is set to “limited.” A value of 0 indicates that only direct groups will be found. That is, when searching for membership in Group A, only direct members of Group A will be found. If Group B is a member of Group A, the members will not be found by this search. Any nonzero value indicates the number of levels to search. For example, if this attribute is set to 1, a search for membership in Group A will return direct members of Group A. If Group B is a member of Group A, the members of Group B will also be found by this search. However, if Group C is a member of Group B, the members of Group C will not be found by this search.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 56

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Common LDAP Issues

In this section, you should have learned how to: • Enable the security realm auditing provider • Define basic LDAP terminology • Identify and troubleshoot common embedded and external LDAP issues • Work with authentication providers • Trace security realm actions and decisions

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 57

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Section Summary

Which of the following is not a typical step in establishing an SSL session? a. Exchange certificate chains. b. View a keystore by using Keytool. c. Check that certificates are trusted. d. Negotiate and select a protocol. e. Generate and share a unique secret key.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: b

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 58

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

Name three ways in which WLS establishes trust for a supplied certificate. a. Use public keys of trusted authorities. b. Perform host name verification. c. Perform role mapping. d. Synchronize security realm data. e. Check whether it has expired.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: a, b, e

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 59

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

What domain attribute affects how embedded LDAP servers are synchronized? a. Private Key Alias b. Static Group Object Class c. Refresh Replica at Startup d. Hostname Verification e. User Base DN

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: c

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 60

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

Which of the following is not an available authentication provider control flag? a. SUFFICIENT b. REQUISITE c. OPTIONAL d. REQUIRED e. ALWAYS

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: e

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 61

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

In this lesson, you should have learned how to: • Identify several WLS scenarios that involve SSL • Describe the fundamentals of SSL and LDAP communication • Trace SSL and security realm functionality • List some common causes of SSL errors • Work with Java keystore files • Troubleshoot the WLS embedded LDAP

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 62

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Summary

This practice covers the following topics: • Diagnosing a server that fails to start • Working with authentication control flags • Debugging the WLS security realm • Analyzing LDAP user and group data • Tuning LDAP authentication settings

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 11 - 63

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Practice 11-2 Investigating Security Realm Problems

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle University and Sentra inversiones y servicios LTDA use only

Troubleshooting Node Manager

After completing this lesson, you should be able to: • Compare the architectures of the Java-based Node Manager and the script-based Node Manager • Configure a Node Manager to use server start and stop scripts • Describe some typical Node Manager problems • Initialize a Node Manager’s security files by using WLST • Tune the SSL settings of the Java Node Manager

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 2

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Objectives

Node Manager: • Is a process that accepts remote commands to start, stop, or suspend servers on the same machine • Monitors server availability and restarts failed processes • Periodically monitors server health and kills/restarts unhealthy processes • Can manage servers for multiple domains • Can use SSL WLST Admin Server

Node Manager

Start/stop Restart Monitor

Server Server Server

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Server instances in a WebLogic Server production environment are often distributed across multiple domains, machines, and geographic locations. Node Manager is a WebLogic Server utility that enables you to start, shut down, and restart Administration Server and Managed Server instances from a remote location. Although Node Manager is optional, it is recommended if your WebLogic Server environment hosts applications with high-availability requirements. A Node Manager process is associated not with a specific WebLogic domain but with a machine. You can use the same Node Manager process to control server instances in any WebLogic Server domain, as long as the server instances reside on the same machine as the Node Manager process. If a server instance that was started using Node Manager fails, Node Manager automatically restarts it. If Node Manager fails or is explicitly shut down, on restart it determines the server instances that were under its control when it exited. Node Manager can restart any failed server instances as needed. The Administration Console can be used to issue commands to Node Managers running on remote machines. WLST (in offline mode) also serves as a Node Manager command-line interface that can run in the absence of a running Administration Server. You can use WLST commands to start, stop, and monitor a server instance without connecting to an Administration Server.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 3

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Node Manager (NM): Review

Implementation

Java NM A Java application that accepts requests over a configured port

Script NM A shell script that relies on the local SSH server process to accept remote communication

Platform Support Windows and UNIX

UNIX

Configuration

nodemanager.properties

Script command-line arguments

Security

Supports user/password and/or one-way SSL via Java keystores

Relies on SSH security, which is based on simple symmetric key encryption

Logging

Supports log file rotation

Prints to standard output

weblogic. NodeManager

wlscontrol.sh

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The Java-based Node Manager runs within a Java Virtual Machine (JVM) process. Configuring SSL for the Java Node Manager involves obtaining identity and trust for the Node Manager and each Administration and Managed Server with which the Node Manager will be communicating and then configuring the Node Manager, the Administration Server, and Managed Servers with the proper identity and trust. In addition, the use of host name verification must be taken into consideration. For UNIX and Linux systems, WebLogic Server provides a script-based version of Node Manager. This script is based on UNIX shell scripts but uses SSH for increased security. If you are installing WebLogic Server on a Windows system, you must use the Java version of Node Manager. The scripted version of Node Manager is not supported on Windows. The script (SSH) version does not provide as much security as the Java-based version. However, the advantage of the script-based Node Manager is that it can remotely manage servers over a network that has been configured to use SSH. SSH uses user ID–based security along with secret key (symmetric) encryption. The Java version of Node Manager can be used in conjunction with inetd on supported UNIX systems. inetd allows Node Manager to be automatically restarted on receiving a request on the configured port. Refer to the documentation for an example configuration.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 4

Oracle University and Sentra inversiones y servicios LTDA use only

Criteria

SSH

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Node Manager Types: Review

1. Configure nodemanager.properties (Java type only). 2. Create Machine definitions in the console so that the admin server knows the following about each Node Manager: – – –

Type Location (host/port) The security level (SSL, SSH, none)

3. Define server startup arguments and/or startup scripts. 4. Configure security, such as: – – –

Node Manager username and password Trusted domain list SSL between Node Managers and clients (Administration Server and/or WLST) Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

A WebLogic Server machine resource associates a particular machine with the server instances that it hosts, and it specifies the connection attributes for the Node Manager process on that system. Configure a machine definition for each machine that runs a Node Manager process and then edit the settings on the machine’s Node Manager tab in the Administration Console. In the Listen Address field, enter the DNS name or IP address on which Node Manager listens. If you set the Type field to SSH or RSH, you should specify values in the Node Manager Home and Shell Command fields.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 5

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Node Manager Configuration: Review

Property

Description

NodeManagerHome

Location of NM property and log files

ListenAddress, ListenPort

Address and port from which NM will accept commands

Authentication Enabled

Require a username/password to use this NM.

StartScriptEnabled

Start servers by using local scripts.

StopScriptEnabled

Stop servers by using local scripts.

StartScriptName

Name of the script used to start local servers (/bin/startWebLogic.sh by default)

StopScriptName

Name of the script used to stop local servers (/bin/stopWebLogic.sh by default)

CrashRecovery Enabled

Automatically restart servers after machine restart. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Node Manager properties define a variety of configuration settings for a Java-based Node Manager process. In many environments, the SSL-related properties in nodemanager.properties may be the only Node Manager properties that you must explicitly define. However, nodemanager.properties also contains non-SSL properties that you might need to specify, depending on your environment and preferences. Any values supplied on the command line override the values in nodemanager.properties. By default, the StartScriptEnabled, StopScriptEnabled, and CrashRecoveryEnabled properties are false. The default value for AuthenticationEnabled is true. You can also use nodemanager.properties to specify the default restart parameters for servers launched from this Node Manager instance: RestartInterval: The amount of time Node Manager will spend attempting to restart a failed server. By default, Node Manager will attempt to restart a server indefinitely until the FAILED_NOT_RESTARTABLE state is reached. RestartMax: The number of times Node Manager will attempt to restart a failed server within the interval defined by RestartInterval. RestartMax is recognized only if RestartInterval is defined.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 6

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Basic Java Node Manager Properties

Property

Description

LogFile

Location of NM log (default is nodemanager.log)

LogLimit

Maximum size of NM log before rotation

LogCount

Maximum number of NM log files

LogAppend

Continue to use current log file if NM restarted.

LogLevel

Log severity threshold (FINEST, FINE, INFO, WARNING, ERROR, ...)

Node Manager log excerpts: ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Node Manager creates a log file located in NodeManagerHome/nodemanager.log. This log file stores data about all of the domains administered by Node Manager. Log output is appended to the current nodemanager.log. Log rotation is disabled by default but can be enabled by setting LogCount in nodemanager.properties. For each server instance that it controls, Node Manager can also maintain a log file that contains stdout and stderr messages generated by the server instance. Use the LogToStdout property in nodemanager.properties. Node Manager creates the server output log for a server instance in the server instance’s logs directory, with the name .out. If the debug property is enabled as a remote start property for the server instance, or if the Node Manager debug property is enabled, Node Manager will include additional debut information in the server output log information. You cannot limit the size of these log files.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 7

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Java Node Manager Logging

• • •

Ideally, the Node Manager Java process should always be running. On Windows, use installNodeMgrSvr.cmd to help install the Node Manager as an automatic service. On UNIX, develop a custom init script that: – Responds to the start argument sent by the init process – Navigates to your NM home directory – Calls /common/bin/commEnv.sh – Launches a JVM process for the application weblogic.NodeManager

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

WebLogic Server does not provide command scripts for uninstalling and reinstalling the Node Manager as a UNIX daemon process. Refer to your operating system documentation for specific instructions. For example, here is one approach that is applicable to most Linux flavors: 1. Add a new script file (or create a symbolic link to a file at another location) named S99wlsnmd to /etc/rc.d/rc5.d. 2. Within the script, check for an argument named start. If it is set, launch the Java Node Manager by using a local JVM. On Windows, the Node Manager service management tools can be found at WL_HOME/server/bin. Perform the following steps: 1. Edit uninstallNodeMgrSvc.cmd and installNodeMgrSvc.cmd to specify your Node Manager’s listen address and listen port. 2. Run uninstallNodeMgrSvc.cmd to delete any existing Node Manager service. 3. Run installNodeMgrSvc.cmd to reinstall Node Manager as a service, listening on the updated address and port.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 8

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Java Node Manager Availability

Argument

Description

-d

Domain name

-s

Server name

-c

Start/stop servers using local scripts.

-f

Name of the script used to start local servers (/bin/startWebLogic.sh by default)

-p

Name of the script used to stop local servers (/bin/stopWebLogic.sh by default)

-x

Print debug messages to standard output.

A typical invocation of the script Node Manager from the admin server: ssh –l myuser –o PasswordAuthentication=no –p 22 192.168.1.1 /u01/mw/wlserver/common/bin/wlscontrol.sh –d MyDomain –s ServerA –c START

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When you use the script Node Manager and you issue commands from your domain’s Administration Server or WLST, these clients internally create a SSH or RSH session with the Node Manager’s host machine. As part of creating this remote session, the Node Manager client also indicates the script to run (wlscontrol.sh) along with the necessary arguments. For troubleshooting purposes, it can be helpful to know the details of this interface. For example, if you experience issues with the script Node Manager, try to establish the remote session from the command line without the Administration Server or WLST. The results help determine whether the issue is related to the configuration of your domain, Node Manager, or SSH/RSH service. Node Manager clients use a template to send commands using SSH/RSH. This template includes variables for which the client (such as the Administration Server) provides values, including: • %H: Host name of the machine to connect to • %N: Remote Node Manager’s home directory • %P: Port number of the machine to connect to • %S: Name of the WebLogic Server to start or stop

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 9

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Basic Script Node Manager Interface

•

%D: The domain of which this Administration Server and the remote managed server are members %C: The name of the command to execute: START, KILL, STAT, GETLOG, or VERSION

The default template resembles the following: ssh -o PasswordAuthentication=no %H %P wlscontrol.sh -d %D -r %R -s %S -x -c -f sample_custom_startscript.sh %C To override this template for a domain, either include the JVM argument -Dweblogic. nodemanager.ShellCommand when starting the Administration Server, or update the Shell Command field in the machine’s configuration by using the Administration Console.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 10

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

•

•

NM starts servers by using parameters provided by clients: – Health Monitoring server attributes in the console – Server Start attributes in the console (if not using scripts) – WLST command arguments

•

•

Prior to starting a server, NM records these start settings at servers//data/nodemanager/ startup.properties. If not using start scripts, JAVA_HOME, CLASSPATH, and other variables are either: – Set using Server Start attributes – Inherited from the NM process environment

•

To minimize potential issues, Oracle recommends that NM use standard scripts to start servers. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Node Manager uses the startup.properties file to record the initial configuration when starting a server as well as when automatically restarting the server later. In general, the values of these properties are sent by the Administration Server to Node Manager based on the domain’s configuration settings. These include: • RestartMax: The number of times Node Manager can attempt to restart the server • RestartDelaySeconds: The number of seconds Node Manager should wait before attempting to restart the server • AutoRestart: Specifies whether Node Manager can automatically restart this server if it fails • AutoKillIfFailed: Specifies whether Node Manager should automatically kill the server if its health status is failed • AdminURL: The URL of the Administration Server The following additional properties may also be set if you are not using a server start script: • JavaHome: Defines the Java home directory used when starting the server

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 11

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Node Manager Server Start Parameters

Arguments: Other JVM arguments used when starting the server ClassPath: The classpath to use when starting a server SecurityPolicyFile: Specifies the security policy file to use when starting this server

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

• • •

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 12

Admin credentials this server will boot with Restart server automatically?

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

In the Administration Console, on the Configuration > Server Start tab for a managed server, specify the startup arguments that Node Manager will use to start this server. If you do not specify startup arguments for the server and do not indicate that Node Manager should use a start script, Node Manager uses its own default settings to start the server. Although these defaults are sufficient to boot a server, ensure a consistent and reliable boot process by explicitly specifying the scripts or arguments to use. The console arguments are used for starting managed servers only. They are not used if Node Manager starts an Administration Server. A server can monitor key aspects of its subsystems and report when a subsystem is not functioning properly. If the server is running under a Node Manager, the Node Manager can automatically restart a server with an unhealthy subsystem. Select a server in the console and edit its Configuration > Health Monitoring tab. You can also use the nmGenBootStartupProps WLST online command to test the generation of the startup.properties file for a given server, without actually requesting Node Manager to start that server. This command requires that you are connected to an Administration Server and takes a single argument—the name of the managed server.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 13

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring Server Start Parameters

Use the console or WLST to remotely monitor the status and logs of all registered Node Managers.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The Administration Console gives you the ability to download and view the log files for all Node Managers in use by the current domain. The Node Manager must be running in order to accept this command. Simply select a machine and click the Monitoring > Node Manager Log tab. If this tab does not display any log messages, use the Monitoring > Node Manager Status tab to determine whether this machine’s Node Manager is currently available or unreachable. The same functionality is also available from WLST. You can also download the output messages for a server being managed by a Node Manager, if the Node Manager has been configured to direct server output to a file. Select a specific server in the console and click the Control > Remote Start Output tab.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 14

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Monitoring Node Managers

Connect to a Node Manager and manage servers on a machine: nmConnect('myuser','mypassword','myhost',5556,'MyDomain') nmStart('ServerA') ... nmKill('ServerC')

Connect to a Node Manager and download its log: nmConnect('myuser','mypassword','myhost',5556,'MyDomain') nmLog('ServerB')

Connect to a Node Manager and download a server log: nmConnect('myuser','mypassword','myhost',5556,'MyDomain') nmServerLog('ServerB')

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

WLST offline serves as a Node Manager command-line interface that can run in the absence of a running Administration Server. You can use WLST commands to start, stop, and monitor a server instance without connecting to an Administration Server. However, if an Administration Server is available, the preferred approach is to connect to this server and use it to issue commands to Node Manager. • nm(): Determines whether WLST is currently connected to Node Manager • nmConnect(): Connects WLST to Node Manager to establish a session. After connecting to Node Manager, you can invoke any Node Manager commands via WLST. Once connected, the WLST prompt displays as follows, where indicates the name of the WebLogic domain that is being managed: wls:/nm/. A command named nmDisconnect is also available. • nmStart(): Starts the specified server in the current WebLogic domain by using Node Manager • nmKill(): Kills the specified server instance that was started with Node Manager • nmServerStatus(): Returns the status of the specified server that was started with Node Manager

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 15

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Node Manager: WLST Examples

Problem Description Invalid nodemanager.properties

Java NM

Script NM

X

Invalid SSH script template

X

Invalid Node Manager authentication setup

X

Missing server startup credentials

X

X

Target domain is not trusted

X

X

Invalid SSL identity or trust configuration

X

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

If you misspell the name of a property in nodemanager.properties, the Java Node Manager may not print a warning message and may simply use the default value for that property. This mistake can significantly affect Node Manager behavior. To override this template for a domain, either include the JVM argument -Dweblogic. nodemanager.ShellCommand when starting the Administration Server, or update the Shell Command field in the machine’s configuration using the Administration Console. If you experience issues with the script Node Manager, try to establish the remote session from the command line without the Administration Server or WLST. The results will help determine whether the issue is related to the configuration of your domain or Node Manager, or whether the issue is related to the actual SSH/RSH command being used. When communication between the Administration Server and a Node Manager uses SSL, the administrator server becomes, in effect, an SSL client and performs all the same handshake and validation tasks that any SSL client performs, such as a Web browser.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 16

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Common Configuration Errors

• •

Start NM without using a nodemanager.properties file, and NM will create a template file with default values. Compare the generated template to your version to troubleshoot configuration issues.

nodemanager.properties: DomainsFile=nodemanager.domains LogLimit=0 PropertiesVersion=10.3 AuthenticationEnabled=true LogLevel=Info ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The first time Node Manager starts, it checks the path specified by NodeManagerHome for the existence of the nodemanager.properties file. If one is not found, a new file is created. When Node Manager is starting and the home path is unknown, the current directory is used instead. However, as a best practice, you should always specify a home path.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 17

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Generating Template Properties for Java NM

•

•

Any credentials required by NM to manage a specific domain are stored in the encrypted file / config/nodemanager/nm_password.properties. Similar to boot.properties, any changes made to this file are automatically encrypted again when NM is restarted.

Client

user/password

Node Manager

Domain nm_password.properties

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The nm_password.properties file contains the Node Manager username and password. These are used to authenticate the connection between a client (for example, the Administration Server) and Node Manager. These credentials are independent from the WLS administrative account that is used to boot servers. The nm_password.properties file must exist on each physical machine that runs Node Manager. However, the Node Manager username and password do not have to be identical for every domain running on the same machine. If you edit nm_password.properties manually, you must restart Node Manager for the changes to take effect.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 18

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Node Manager Authentication

You can use the console to update the credentials used by the admin server to access Node Managers.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When you create a new domain, it automatically generates a random username and password that the Administration Server will use to access Node Managers. Select the name of your domain in the Administration Console, click the Security tab, and then click Advanced. If these credentials do not match those indicated in the nm_password.properties file on a Node Manager’s host machine, access is denied by the Node Manager. As a best practice, do not edit these generated credentials. Instead, use the nmEnroll WLST command to synchronize the nm_password.properties file with these credentials.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 19

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring Node Manager Credentials

The nodemanager.domains file specifies: • A list of domains that can be managed from this NM • The local file system location of each trusted domain

nodemanager.domains: MyDomain1=/u01/domains/MyDomain1 MyDomain2=/u01/domains/MyDomain2 ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The nodemanager.domains file specifies the domains that a Node Manager instance controls, as well as their locations on the file system. Stand-alone clients thus do not need to specify the domain directory explicitly. This file provides additional security by restricting Node Manager client access to the domains listed in this file. After the system is restarted, Node Manager checks each managed domain specified in the nodemanager.domains file to determine whether there are any server instances that were not cleanly shut down. This is determined by the presence of any lock files which are created by Node Manager when a WebLogic Server process is created. This lock file contains the process identifier for the WebLogic Server startup script. If the lock file exists but the process ID is not running, Node Manager will attempt to automatically restart the server if configured to do so. If you are using the Java Node Manager, you can disable the use of this file for security purposes by using the DomainsFileEnabled property in nodemanager.properties. If you simply want to change the default name of this file, use the DomainsFile property.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 20

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Node Manager Trusted Domains

• •

For convenience, you can also use WLST to initialize NM so that it accepts commands from a specific domain. The nmEnroll() command: – Generates nm_password.properties and initializes it with the NM credentials configured for the domain to which you are currently connected – Adds the domain to the nodemanager.domains file

Establish trust between a Node Manager and a running admin server: connect('mydomainuser','mypassword','myadminhost:7001') nmEnroll('/u01/domains/MyDomain1','/u01/nodemanager') Not required if running WLST from this location Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Use the nmEnroll() command to create and update the necessary Node Manager configurations files after creating a domain. You should run nmEnroll() on each machine that is running a managed server. Additionally, you should run nmEnroll() for each domain directory on each machine. WLST must be connected to an Administration Server to run this command. WLST does not need to be connected to Node Manager. The following files are created or updated: • nm_password.properties • SerializedSystemIni.dat (used to encrypt/decrypt credentials) • nodemanager.domains Specify the path of the root domain directory to which you want to save nm_password.properties and SerializedSystemIni.dat. This argument defaults to the directory in which WLST was started. Also specify the path of your Node Manager home and the nodemanager.domains file. This argument defaults to /common/nodemanager, where refers to the top-level installation directory for WebLogic Server.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 21

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Machine Enrollment

•

Due to the nature of NM, server startup credentials must be supplied in boot.properties files.

•

If the client sends startup credentials to NM, it automatically generates the /data/nodemanager/boot.properties file for the server to boot against.

Node Manager output: ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Any server instance that is started by Node Manager encrypts and saves the credentials with which it started in a server-specific boot.properties file, for use in automatic restarts. You can also use the WLST online command nmGenBootStartupProps to test the generation of the boot.properties and startup.properties files for a given server without actually requesting the Node Manager to start that server. This command requires you to be connected to an Administration Server. When you invoke Node Manager to start an Administration Server, the boot identity can be provided on the command line or obtained from the Administration Server’s boot.properties file. For convenience, the WebLogic Server Configuration Wizard initializes the boot.properties file and the startup.properties file for an Administration Server when you create the domain.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 22

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Server Boot Identity

Property

Description

SecureListener

Require SSL.

Keystores

Use the demo keystore or a custom one.

CustomIdentityKeyStoreFileName

Node Manager’s identity keystore

CustomIdentityKeyStorePassPhrase CustomIdentityAlias CustomIdentityPrivateKeyPass Phrase WLST or Admin Server Trust Certs

T3S

Node Manager Server Cert and Private Key

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Node Manager supports one-way SSL and keystore-based SSL artifacts. For backward compatibility, additional non-keystore and two-way SSL properties are supported by Node Manager but should not be used on new WebLogic Server installations. By default, the Keystores property is set to the value DemoIdentityAndDemoTrust, meaning that it will use the sample certificate and associated private key found in your WebLogic installation at server/lib/DemoIdentity.jks. For production environments, change this value to CustomIdentityAndJavaStandardTrust and use the remaining properties to specify the location and credentials for your keystore file and its contents. As with all SSL communication, the use of host name verification must be taken into consideration. By default, the host name present in the Node Manager’s identity certificate must match the host name used by the Administration Server or WLST to access this Node Manager. An additional property, named CipherSuite, is available to indicate which protocol and encryption algorithms are preferred. The available options depend on your Java SSL implementation. Enable SSL debugging on a server to view a list of the available ciphers in your WebLogic Server installation. Currently, the default value used by Node Manager is TLS_RSA_EXPORT_WITH_RC4_40_MD5.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 23

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring Node Manager SSL

Which is not an available configuration parameter for the Java Node Manager? a. StopScriptEnabled b. LogCount c. AuthenticationEnabled d. StartScriptName e. SSHEnabled

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: e

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 24

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

Name three components of the Java Node Manager’s security infrastructure. a. nodemanager.domains b. c. d. e.

Keystore Role Mapping Username/Password nodemanager.keys

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Answer: a, b, d

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 25

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Quiz

In this lesson, you should have learned how to: • Compare the architectures of the Java-based Node Manager and the script-based Node Manager • Configure a Node Manager to use server start and stop scripts • Describe some typical Node Manager problems • Initialize a Node Manager’s security files by using WLST • Tune the SSL settings of the Java Node Manager

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 26

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Summary

This practice covers the following topics: • Debugging Node Manager client communication • Enrolling Node Manager with a domain • Debugging remote server startup issues • Configuring Node Manager to use standard and custom scripts

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 12 - 27

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Practice 12-1 Investigating Node Manager Problems

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle University and Sentra inversiones y servicios LTDA use only

Troubleshooting Clusters

After completing this lesson, you should be able to: • Describe how a proxy plug-in creates connections and performs failover • Explain the details of heartbeat and replication communication within a cluster • Monitor and debug proxy plug-ins • Perform some basic tasks with Oracle HTTP Server • Monitor cluster members and session utilization • Identify several common proxy and cluster issues

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 2

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Objectives

•

Proxy Diagnostics – – – – –

•

OHS Log Plug-in Parameters Connection Architecture Proxy Debug Page Proxy Debug Log

Replication Diagnostics

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 3

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Road Map

A WLS cluster supports additional features: • To provide high availability for applications and services • To perform load balancing and failover • That are transparent to both applications and clients Hardware or software

Cluster Server 1

Web client

Proxy Server 2

EJB client JMS client

Stub Server n

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Clustering is configuring a group of Oracle WebLogic Servers to work together to provide client access to the services offered by the servers in the cluster. The cluster appears to a client as one instance, whether the client is a Web client or a Java application. By replicating the services provided by one instance, an enterprise system achieves a fail-safe and scalable environment. Scalability is achieved by balancing the load of incoming requests across the servers in the cluster. High availability is achieved through the replication of services, so that when one service fails, another service can resume where the first service left off. A cluster uses the redundancy of multiple servers to insulate clients from failures. Oracle WebLogic Server provides clustering support for Web applications by replicating the HTTP session state of clients. You can balance the Web application load across a cluster by using an Oracle WebLogic Server proxy plug-in or the external load-balancing hardware. Failover for Enterprise JavaBeans (EJBs) objects is accomplished using the object’s replicaaware stub. When a client makes a call through a replica-aware stub to a service that fails, the stub detects the failure and retries the call on another replica.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 4

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Cluster Review

Oracle provides plug-ins to popular Web servers that: • Dynamically load-balance requests across a cluster of WebLogic Servers using round robin • Transparently failover requests if a server is unavailable • Support “sticky” sessions using cookies Cluster Client Client

Web Server Plug-In

Server Server

Client Server

Client

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Plug-ins enable WebLogic Server to communicate with applications deployed on Oracle HTTP Server, Apache HTTP Server, Sun Java System Web Server, or Microsoft’s Internet Information Server. Typically, WebLogic Server handles the application requests that require dynamic functionality, the requests that can best be served with dynamic HTML pages or JSPs (Java Server Pages). The Web server then hosts static content and also in the case of a cluster, performs load balancing and failover. As always, refer to the documentation to make sure your that your specific Web server version and host OS are supported by Oracle. The plug-in does a simple round-robin between all available servers in a cluster. The server list specified in this property is a starting point for the dynamic server list that the server and plug-in maintain. WebLogic Server and the plug-in work together to update the server list automatically with new, failed, and recovered cluster members. The plug-in also supports “sticky” session behavior, similar to hardware load balancers. This means that the plug-in directs HTTP requests containing a session cookie (or URL-encoded session ID or a session ID stored in the POST data) to the same server in the cluster that originally created the cookie. In other words, the client is pinned to a specific server in the cluster. This behavior ensures that users do not lose their session data between requests.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 5

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Proxy Plug-in Review

•

Plug-ins are: – Bundled with WLS and Oracle HTTP Server – Also available online to download separately – Released on a separate cycle than that for the server itself, so always confirm that a newer version is not available

•

Although all plug-ins share the same major capabilities, the installation and configuration steps vary slightly by vendor.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The WebLogic Sun Java System Web Server plug-in module is distributed as a shared object (.so) on UNIX platforms and as a dynamic-link library (.dll) on Windows. These files are located in the WL_HOME/server/plugin/OperatingSystem/Architecture directory of your WebLogic Server distribution. WL_HOME represents the top level installation directory for your WebLogic platform. The server directory contains installation files for WebLogic Server. OperatingSystem refers to the operating system, such as UNIX or Windows. Add the following lines to the beginning of the magnus.conf file. These lines instruct Sun Java System Web Server to load the native library (the .so or .dll file) as a module: Init fn="load-modules" funcs="wl_proxy,wl_init"\ shlib=/usr/local/netscape/plugins/(.so or .dll file) Init fn="wl_init" To install the Apache HTTP Server Plug-In as a dynamic shared object, locate the shared object directory for your specific platform, such as linux/i686, solaris/sparc, or win/32. Then copy the mod_wl_20.so file to the APACHE_HOME/modules directory and add the following line to your APACHE_HOME/conf/httpd.conf file: LoadModule weblogic_module modules/mod_wl_20.so

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 6

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Obtaining and Using Plug-Ins

•

OHS is: – A component of the Oracle Web Tier suite – Based on Apache HTTP Server (httpd.conf) – Installed with a WLS plug-in module (mod_wl_ohs) – Managed and monitored using command-line tools – Configurable through the Fusion Middleware Control application, if installed on WLS

•

The OHS plug-in must be configured manually using the mod_wl_ohs.conf file.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle HTTP Server 11g, Release 1 (11.1.1.2.0) is based on Apache 2.2.13 infrastructure, and includes modules developed specifically by Oracle. The features of single sign-on, clustered deployment, and high availability enhance the operation of the Oracle HTTP Server. Oracle HTTP Server can also be a proxy server, both forward and reverse. A reverse proxy enables content served by different servers to appear as if coming from one server. Configuration for Oracle HTTP Server are specified through directives in configuration files in the exact same manner as Apache HTTP Server configuration files. The mod_wl_ohs module provides the same functionality as the Oracle WebLogic Server Plug-in for Apache HTTP Server (mod_weblogic) except for some minor differences, as follows: • Uses Oracle’s security layer (NZ) to provide SSL support for the module. A new directive, WlSSlWallet, has been added to Oracle HTTP Server through the mod_wl_ohs module that allows the use of Oracle Wallets. • •

Supports two-way SSL between Oracle HTTP Server and Oracle WebLogic Server. Supports IPv6 for communication with WebLogic Server.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 7

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Oracle HTTP Server (OHS) Review

Similar to WLS domains: • A single OHS product installation can support multiple processes or components. • Processes that are managed together are called an instance. • Instances have a root directory under which process configuration and log files are stored. Client

Component1

Component2

Config, Logs

Config, Logs

Client

Instance OHS Installation

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle HTTP Server directories are divided between the Oracle home and the Oracle instance. The Oracle home directories are read-only, and contain the Oracle Fusion Middleware binaries. The Oracle instance directories contain the modules, applications, and logs for Oracle HTTP Server. Each OHS component has a root configuration directory found at /config/OHS/, which includes the WLS plug-in configuration file, mod_wl_ohs.conf. Similarly, each component’s log files are found at /diagnostics/logs/OHS/. When Oracle HTTP Server starts up, it writes the process ID (PID) of the parent httpd process to the httpd.pid file located, by default, in the following directory: /diagnostics/logs/OHS/. The process ID can be used by the administrator when restarting and terminating the daemon. If a process stops abnormally, it is necessary to stop the httpd child processes by using the kill command. The PidFile directive in httpd.conf specifies the location of the PID file.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 8

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Oracle HTTP Server (OHS) Review

OPMN is: • A process that is used to start, manage, and monitor other Oracle Web Tier and Identity Management processes • Similar conceptually to the WLS node manager • Accessed using the opmnctl command-line tool or Oracle Web-based tools Start Stop OPMNCTL

OPMN

Status Restart

Virtual Directory OHS 1 OHS 2 Web Cache

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle Fusion Middleware provides a high availability infrastructure integration with Oracle Process Manager and Notification Server (OPMN), for process management, failure detection, and failover for Oracle HTTP Server processes. Oracle Fusion Middleware components that are managed by OPMN should never be started or stopped manually. Do not use command-line scripts or utilities from previous versions of Oracle Fusion Middleware for starting and stopping system components. OPMN must be the last service turned off whenever you restart or turn off your computer. Oracle Process Manager (PM) is the centralized process management mechanism used to manage system processes. The PM is responsible for starting, restarting, stopping, and monitoring every process it manages. The PM handles all requests sent to OPMN associated with controlling a process or obtaining status about a process. The PM is also responsible for performing death-detection and automatic restart of the processes it manages. The system processes that PM is configured to manage are specified in the opmn.xml file. Oracle Notification Server (ONS) is the transport mechanism for failure, recovery, startup, and other related notifications between components in Oracle Fusion Middleware. It operates according to a publish-subscribe model: a system component receives a notification of a certain type for each subscription to ONS. When such a notification is published, ONS sends it to the appropriate subscribers.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 9

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Oracle Process Manager and Notification Server (OPMN) Review

Start OPMN and all managed processes, if not already started: opmnctl startall Get the name, status, memory usage, and port number of processes: opmnctl status -l Restart a specific OHS process: opmnctl restartproc ias-component=PayrollWeb1 Stop all OHS processes: opmnctl stopproc process-type=OHS

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The opmnctl program provides a centralized way to control and monitor system components from the command line. The available commands include: • start: Start the OPMN server for a local Oracle instance without starting system processes. • startall: Start OPMN as well as the system processes for a local Oracle instance. startall is equivalent to start followed by startproc without arguments. • stopall: Shut down the OPMN server as well as the system processes for the local Oracle instance. This request operates synchronously; it waits for the operation to complete before returning. • startproc, restartproc, stopproc: Use these commands to start, restart, or stop system processes. The OPMN server must be up and running. The following attributes are supported. Any of these attributes may be omitted, and treated as a wild card: • ias-component: The name of a specific managed process, as defined in opmn.xml. • process-type: The type of managed process to command, as defined in opmn.xml. • process-set: The name of a custom process group defined in opmn.xml.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 10

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

OPMNCTL Examples

•

Error and access logs are found at /diagnostics/logs/OHS/ by

•

default. Error logs can use Apache or Oracle (ODL) formats.

Use httpd.conf to configure ODL format logging: OraLogMode odl-text OraLogSeverity NOTIFICATION OraLogRotationParams S 10:70

Also ERROR, WARNING, or TRACE Rotate log when size (S) is 10 MB.

Typical error log output when plug-in enabled: [NOTIFICATION:16] [mod_weblogic.c] Oracle WebLogic plugin build date/time: ... [NOTIFICATION:32] [mod_weblogic.c] mod_weblogic: Testing Debug=ON LogFile=...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

There are two types of logs for Oracle HTTP Server. Error logs record server problems by default, but can also be configured to record other server events. Access logs record which components and applications are being accessed and by whom. You can view Oracle Fusion Middleware log files using either Fusion Middleware Control or a text editor. The default name of the error log file is .log. Oracle HTTP Server enables you to choose the format in which you want to generate log messages. You can choose to generate log messages in the legacy Apache message format, or use Oracle Diagnostic Logging (ODL) to generate log messages in text or XML-formatted logs, which complies with Oracle standards for generating error log messages. By default, Oracle HTTP Server error logs use ODL for generating diagnostic messages. OHS supports two types of log rotation policies: size-based and time-based. You can configure both error log and access log to use either one of these two rotation polices. When rotation type is set to S (sized-based), set the policy to : (in MB). The specified message severity is interpreted as the lowest desired severity, and all messages of that severity level and higher are logged. Additional, you can also supply an integer in the range of 1–32, where 1 is the most severe, and 32 is the least severe. Using level 1 will result in fewer messages than using level 32.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 11

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

OHS Logs

• •

Plug-ins route to servers or clusters based on the incoming URL path, requested file type, or virtual host definition. The plug-in configuration file name and format vary based on proxy type.

Use mod_wl_ohs.conf to proxy to a cluster:

Default parameters for all locations

WebLogicCluster nodea.xyz.com:7001,nodeb.xyz.com:7001, nodec.xyz.com:7001 Proxy based on URL path SetHandler weblogic-handler Debug OFF Parameters for this specific location Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

For Apache HTTP Server, edit the httpd.conf file, or alternatively create a separate configuration file and use the Include directive in httpd.conf. This technique is already done for you if using Oracle HTTP Server. The separate configuration file is named mod_wls_ohs.conf. Oracle recommends that you use an Apache IfModule block to define parameters only when the plug-in module is found. If you choose to not use the IfModule, you can instead directly place the WebLogic properties inside Location or VirtualHost blocks. If you use an Apache HTTP Server VirtualHost block, you must include all configuration parameters (MatchExpression, for example) for the virtual host within the VirtualHost block. If you are proxying requests to a cluster of WebLogic Servers, use the WebLogicCluster parameter instead of the WebLogicHost and WebLogicPort parameters. To proxy requests by path, use the Location block and the SetHandler statement. SetHandler specifies the handler for the Apache HTTP Server Plug-In module. To proxy requests by MIME type, add a MatchExpression line to the IfModule block. Note that if both MIME type and proxying by path are enabled, proxying by path takes precedence over proxying by MIME type. You can also use multiple MatchExpressions.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 12

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Plug-in Configuration Review

Parameter

Description

WebLogicHost, WebLogicPort

Proxy to a single server

WebLogicCluster

Proxy to this initial list of clustered servers

MatchExpression

Proxy requests for files of this MIME type

PathTrim

Remove this text from the incoming URL path before forwarding a request.

PathPrepend

Add this text to the incoming URL path before forwarding a request.

ErrorPage

URL to direct users to if all servers are unavailable

WLExcludePathOrM imeType

Do not proxy for this specific URL path or MIME type.

WLProxySSL

Set to ON to establish an SSL connection to WebLogic if the incoming request also uses HTTPS.

TrustedCAFile

Location of PEM file that contains trusted CA certificates

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

•

•

• •

WebLogicCluster: List of WebLogic Servers that can be used for load balancing. The server or cluster list is a list of host:port entries. The plug-in does a simple round-robin between all available servers. The server list specified in this property is a starting point for the dynamic server list that the server and plug-in maintain. WebLogic Server and the plug-in work together to update the server list automatically with new, failed, and recovered cluster members. WLDNSRefreshInterval: If defined in the proxy configuration, specifies the number of seconds interval at which WebLogic Server refreshes DNS name to IP mapping for a server. This can be used in the event that a WebLogic Server instance is migrated to a different IP address, but the DNS name for that server’s IP remains the same. In this case, at the specified refresh interval the DNSIP mapping will be updated. ErrorPage: Create your own local error page that is displayed when your Web server is unable to forward requests to WebLogic Server. MaxPostSize: Maximum allowable size of POST data, in bytes. If the content-length exceeds this value, the plug-in returns an error message. If set to -1, the size of POST data is not checked. This is useful for preventing denial-of-service attacks that attempt to overload the server.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 13

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Basic Plug-in Parameters

By default, plug-ins: • Pool and reuse connections to servers using the “keep alive” feature of the HTTP protocol • Fail over to the next server if the initial connection times out • Continue to retry connections for a certain amount of time • Fail over to the next server after waiting too long for a response to an individual request • Eventually retry “failed” servers after a certain amount of time

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When the Apache HTTP Server Plug-In attempts to connect to WebLogic Server, the plug-in uses several configuration parameters to determine how long to wait for connections to the WebLogic Server host and, after a connection is established, how long the plug-in waits for a response. If the plug-in cannot connect or does not receive a response, the plug-in attempts to connect and send the request to other WebLogic Server instances in the cluster. If the connection fails or there is no response from any WebLogic Server in the cluster, an error message is sent.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 14

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Proxy Connection Architecture

2

1 Client request

Connection refused or timeout

Create connection if none available in pool.

FAILOVER— Select next server.

Check whether time left to retry.

3

Send request to WLS and wait for response.

4 Client response

Request timeout

Client error

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

1. The plug-in receives a request from the Web server. If a WebLogic session ID is found in the request, the plug-in connects to the primary server defined in the cookie. Otherwise, the plug-in connects to the next server in the round-robin cycle. 2. If a previously created connection to the selected server is not available, the proxy attempts to establish a connection. If the connection is unsuccessful because the server refused it or it timed out, the plug-in sleeps for the time defined by the ConnectionRetrySecs parameter and then tries the next server in the list. If at any time ConnectTimeoutSecs is exceeded, an HTTP error is sent to the client or the page identified by the ErrorPage parameter is shown. 3. After a connection is successful, the request is sent and the server waits for the response for the time specified by the WLIOTimeoutSecs parameter. If the plug-in does not receive a response from the server, the plug-in checks the value of the Idempotent parameter. If the parameter is enabled (default), the server is marked as “bad.” Then the process starts again from step 2 above, until the request succeeds. If the Idempotent parameter is disabled or all servers have been exhausted, an HTTP error is sent to the client or the page identified by the ErrorPage parameter is shown. 4. If the server responds successfully to the request, the response is forwarded back to the original client that sent the request.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 15

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Proxy Connection Architecture

• •

•

The WebLogicCluster parameter simply specifies an initial list of servers for the proxy to try. Cluster members use heartbeat messages to determine the availability of current members and the presence of new ones. Responses from a cluster member to a proxy include an updated list of servers to try. Server A

Proxy A,B,C

Server B A,B Server C

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When you use the WebLogicCluster parameter in your configuration file to specify a list of WebLogic Servers, the plug-in uses that list as a starting point for load balancing among the members of the cluster. After the first request is routed to one of these servers, a dynamic server list is returned containing an updated list of servers in the cluster. The updated list adds any new servers in the cluster and deletes any that are no longer part of the cluster or that have failed to respond to requests. This list is updated automatically with the HTTP response when a change in the cluster occurs.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 16

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Dynamic Server List

Parameter

Description

KeepAliveEnabled

Enable/disable connection pooling and reuse.

KeepAliveSecs

Maximum time that a connection can be idle and remain open, before a new connection must be created

ConnectTimeoutSecs

Maximum total wait time to establish a connection

WLSocketTimeoutSecs

Wait time for a connection attempt

ConnectRetrySecs

Pause time between retrying failed connections

WLIOTimeoutSecs

Maximum wait time to receive a response for a request sent over a connection

Idempotent

Enable/disable failover after WLIOTimeoutSecs.

MaxSkipTime

Wait time before retrying a server marked as failed

DynamicServerList

Enable/disable dynamic server list feature.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

•

• •

•

•

KeepAliveSecs: The length of time after which an inactive connection between the plug-in and WebLogic Server is closed. You must set KeepAliveEnabled to true (ON when using the Apache plug-in) for this parameter to be effective. The default is 20 seconds. WLSocketTimeoutSecs: Set the timeout for the socket while connecting, in seconds. The value must be greater than 0 and the default is 2 seconds. WLIOTimeoutSecs: Defines the amount of time the plug-in waits for a response to a request from WebLogic Server. The plug-in waits for the server to respond and then declares that server dead, and fails over to the next server. The minimum value is 10 seconds and the default is 300 seconds. ConnectRetrySecs: Interval in seconds that the plug-in should sleep between attempts to connect to the WebLogic Server host (or all of the servers in a cluster). The number of times the plug-in tries to connect before returning an HTTP 503/Service Unavailable response to the client is calculated by dividing ConnectTimeoutSecs by ConnectRetrySecs. To specify no retries, set the parameters to equal values. However, the plug-in always attempts to connect at least twice. MaxSkipTime: The amount of time after which the plug-in will retry the server marked as “bad.” The plug-in attempts to connect to a new server in the list each time a unique request is received (that is, a request without a session cookie). The default is 10 seconds.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 17

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Connection Parameters

•

Connectivity problems can cause: – Unnecessary failover conditions – Unexpected HTTP errors sent to the client

•

Causes of unexpected connection failures include: – WebLogicCluster has typos. – ConnectTimeoutSecs is set too low. – ConnectRetrySecs is set higher than ConnectTimeoutSecs.

•

Causes of unexpected request failures include: – WLIOTimeoutSecs is set too low. – MaxPostSize is set too low on either the proxy or WLS.

•

If the proxy is slow to use a restarted cluster member, MaxSkipTime may be set too high.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The value of the KeepAliveSecs parameter must be less than or equal to the value of the Duration field set using a WebLogic Server’s HTTP tab in the administration console (the KeepAliveSecs attribute of the MBean). Similarly, if the MaxPostSize parameter is greater than or equal to the same WLS setting, it will have no effect. The WLIOTimeoutSecs parameter should typically be set to a large value (default is 5 minutes). If the value is less than the time your application takes to process a request, then you may see unexpected results.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 18

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Common Connectivity Issues

• •

Refer to your vendor’s documentation on configuring and troubleshooting SSL between the client and the proxy. When SSL is enabled on the plug-in, it behaves exactly as if it were a standard WLS client: – Validate WLS certificate using trusted CAs. – Verify WLS certificate host name. – Check for required constraint in WLS certificate. Parameter

Description

SecureProxy

If enabled, always use SSL between proxy and WLS.

RequireSSLHostMatch

Enable/disable host name verification.

SSLHostMatchOID

Attribute in certificate’s distinguished name that contains the host name (CN by default)

EnforceBasicConstraint

The level of validation for certificate constraints

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

You can use the SSL protocol to protect the connection between the Apache HTTP Server Plug-In and WebLogic Server. The SSL protocol provides confidentiality and integrity to the data passed between the Apache HTTP Server Plug-In and WebLogic Server. RequireSSLHostMatch: Determines whether the host name to which the plug-in is connecting must match the Subject Distinguished Name field in the digital certificate of the WebLogic Server to which the proxy plug-in is connecting. Depending on your network configuration, the host name in the certificate should be the same as the server’s Listen Address or External DNS Name attributes. SSLHostMatchOID: The field in the Subject Distinguished Name of the peer digital certificate is to be used to perform the host match comparison. Supported values are 23 (Surname), 22 (Common Name), 13 (Email), 30 (Organizational Unit), 29 (Organization) and 26 (Locality). EnforceBasicConstraint: This parameter closes a security hole that existed with SSL certificate validation where certificate chains with invalid V3 CA certificates would not be properly rejected. This allowed certificate chains with invalid intermediate CA certificates, rooted with a valid CA certificate to be trusted. X509 V3 CA certificates are required to contain the BasicConstraints extension, marked as being a CA, and marked as a critical extension. This checking protects against non-CA certificates masquerading as intermediate CA certificates. Supported values are STRONG (default), STRICT, and OFF.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 19

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Proxy SSL Issues

When enabled, plug-ins: • Check for a special query string parameter • Display the current status of each server in the initial and dynamic lists • Display various configuration settings and communication statistics ... DebugConfigInfo ON

http://proxyhost:port/proxypath/?__WebLogicBridgeConfig

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

DebugConfigInfo: Enables the special query parameter __WebLogicBridgeConfig. Use it to get details about configuration parameters from the plug-in. The plug-in gathers the configuration information and runtime statistics and returns the information to the browser. The plug-in does not connect to WebLogic Server in this case. This parameter is strictly for debugging and the format of the output message can change with releases. For security purposes, keep this parameter turned OFF in production systems.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 20

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Proxy Debug Page

General Server List: 1. Host: 'nodea' Port: 7001 SecurePort: 7001 Status: OK 2. Host: 'nodeb' Port: 7001 SecurePort: 7001 Status: OK 3. Host: 'nodec' Port: 7001 SecurePort: 7001 Status: OK ConnectRetrySecs: '2' ConnectTimeoutSecs: '10' ...

Configuration parameters

Status of cluster members

Runtime statistics: • requests: 524 Runtime • successful requests: 518 statistics • Exception objects created: 6 ... • CONNECTION_REFUSED exceptions: 0 • CONNECTION_TIMEOUT exceptions: 0 • READ_ERROR_FROM_SERVER exceptions: 6

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Additional configuration parameters are displayed. For example: WLCookieName: JSESSIONID Debug: 'ALL' DebugConfigInfo: 'ON' DefaultFileName: '' DynamicServerList: 'ON' ErrorPage: '' FileCaching: ON Idempotent: ON KeepAliveEnabled: ON KeepAliveSecs: 20 MaxPostSize: '0' MaxSkipTime: '10' PathPrepend: '

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 21

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Proxy Debug Page

• •

Plug-ins can generate a log file for troubleshooting purposes. Configure the data that is logged for each request.

... Debug ERR WLLogFile /tmp/wlproxy.log

Debug Value

What Is Logged

ON

Only error and informational messages

ERR

Only error messages

ALL

All messages and HTTP headers

HFC

ON + HTTP headers sent from client

HFW

ON + HTTP headers sent back from WLS Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The debugging information is written to the /tmp/wlproxy.log file on UNIX systems and c:\TEMP\wlproxy.log on Windows systems. Override this location and file name by setting the WLLogFile parameter to a different directory and file. The WLTempDir parameter provides an additional way to change this location. Ensure that the directory has write permission assigned to the user who owns the proxy process. Use the Debug parameter to set any of the listed options. The default is OFF. HFC, HTW, HFW, and HTC options may be set in combination by entering them separated by commas, for example “HFC,HTW”. Additional options include: HTW: The plug-in logs headers sent to WebLogic Server, and informational and error messages. HTC: The plug-in logs headers sent to the client, informational messages, and error messages.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 22

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Proxy Debug Log

Using Uri /hrWeb/updateProfile.action ... Found cookie from cookie header ... ... attempt #0 out of a max of 5 ... getPooledConn: No more connections in the pool for Host[nodea.xyz.com] Port[7001] SecurePort[7001] ... general list: created a new connection ... ... Hdrs from clnt:[Cookie2]=[USER="tom123"] ... Method is POST URL::sendHeaders(): meth='POST' file='/hrWeb/updateProfile.action' protocol='HTTP/1.1' ... Reader::fill(): first=0 last=0 toRead=4096

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Each incoming client request is assigned a unique ID, which can be used to correlate log messages. The above example traces the following proxy activities: 1. Accept an incoming request from a client. 2. Check for session ID cookie in the request and if present map it to its primary server. 3. Start timer to process the request. If the configured maximum time is exceeded, an HTTP error is sent to the client. The number of attempts is calculated by dividing the configured try time from the maximum time. 4. Check whether a connection for the selected server is available in the connection pool. 5. Create a new connection to the server if needed. 6. Print incoming request headers, including cookies. 7. Forward the request to the server. 8. Wait for the server response and forward it back to the client.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 23

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Typical Proxy Trace

In this section, you should have learned how to: • Perform basic administrative tasks on OHS • List several plug-in parameters • Describe the plug-in connection and failover processes • Configure and interpret the plug-in debug page • Analyze the contents of a plug-in log file

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 24

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Section Summary

This practice covers the following topics: • Monitoring cluster utilization under load • Monitoring runtime statistics for the proxy plug-in • Analyzing proxy plug-in logs • Tuning proxy connection settings

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 25

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Practice 13-1 Investigating Proxy Problems

• •

Proxy Diagnostics Replication Diagnostics – – – –

Unicast Communication Runtime Monitoring Debug Flags Serialization

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 26

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Road Map

•

Clusters use a “unicast” communication model to distribute messages intended for all members: – Heartbeats so that all members know each other’s availability in real time – JNDI tree synchronization, such as during JMS migration

•

Clusters use standard IP communication for simple peerto-peer interactions: – Session replication and failover for Web or EJB applications – Distributing JMS messages to member destinations

•

Oracle does not recommend the use of IP multicast communication and supports it only for backwards compatibility.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

WebLogic Server provides an alternative to using multicast to handle cluster messaging and communications. WebLogic’s “unicast” (leader distribution) model tends to be simpler to configure and manage because it does not require the cross network configuration that multicast requires. Additionally, it reduces potential network errors that can occur from multicast address conflicts. Each WebLogic Server instance in a cluster uses this unicast model to distribute regular heartbeat messages that advertise its availability. By monitoring heartbeat messages, server instances in a cluster determine when a server instance has failed. Clustered server instances also monitor IP sockets as a more immediate method of determining when a server instance has failed. Each instance in a cluster uses unicast to announce the availability of clustered objects that are deployed or removed locally. Each server instance monitors these announcements and updates its local JNDI tree to reflect current deployments of clustered objects. When a JMS message is produced to a distributed destination that has been targeted to a cluster, the initial server (member destination) that receives the message may choose to forward it to another server. For example, distributed queue members with active consumers are preferred over those with zero consumers.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 27

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Cluster Communication Review

• •

WLS unicast is a “hub and spoke” communication model. As servers join or leave a cluster, members communicate with each other to dynamically: – Nominate a subset of the members as group leaders – Assign members to groups

•

Leaders are responsible for relaying broadcast messages to their group members. Server Server

Server Server

Server

Server

Server Server

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Cluster members communicate to the group leader when they need to send a broadcast message, which is usually the heartbeat message. When the cluster members detect the failure of a group leader, the next oldest member becomes the group leader. The frequency of communication in unicast mode is similar to the frequency of sending messages on multicast port.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 28

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Unicast Architecture

•

• •

Web and EJB applications use sessions to track information in server memory for each client (a shopping cart, for example). Sessions that are not accessed for a period of time are invalidated. WLS periodically scans all sessions and deletes any invalidated ones.

weblogic.xml: ... 3600 60

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Session tracking enables you to track a user’s progress over multiple servlets or HTML pages, which, by nature, are stateless. A session is defined as a series of related browser requests that come from the same client during a certain time period. Session tracking ties together a series of browser requests that may have some meaning as a whole, such as a shopping cart application. The cookies that WebLogic Server uses to track sessions are set as transient by default and do not outlive the session. When a user quits the browser, the cookies are lost and the session ends. This behavior is in the spirit of session usage and it is recommended that you use sessions in this way. You can specify a period of time after which HTTP sessions expire if inactive. When a session expires, all data stored in the session is discarded. You can set the timeout period in either web.xml or weblogic.xml. You also use weblogic.xml to tune the time, in seconds, that WebLogic Server waits between doing house-cleaning checks for timed-out and invalid sessions, and deleting the old sessions and freeing up memory.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 29

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Session Management Review

• •

By default, when a client fails over to another server in the cluster, its session information is lost. WebLogic Server supports several session persistence strategies to recover sessions when clients are redirected from failed servers, including: – – – –

•

In-memory replication JDBC persistence File persistence Cookie persistence

Session persistence – Can be done synchronously or asynchronously – Is configured in the weblogic.xml descriptor

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Web application components, such as servlets and JavaServer Pages (JSPs), maintain data on behalf of clients using an HttpSession instance that is available on a per-client basis. To provide high availability of Web applications, shared access to one HttpSession object must be provided. HttpSession objects can be replicated within WebLogic Server by storing their data using in-memory replication or file system persistence, or by storing it in a database. For clusters that use a supported hardware load-balancing solution, the load-balancing hardware redirects client requests to any available server in the WebLogic Server cluster. The cluster itself obtains the replica of the client’s HTTP session state from a secondary server in the cluster. In clusters that use Web servers with WebLogic proxy plug-ins, the proxy plug-in handles failover transparently to the client. If a server fails, the plug-in locates the replicated HTTP session state on a secondary server and redirects the client’s request accordingly. With synchronous replication, the secondary is updated at the end of the current request and before the response is sent back to the client. This helps to guarantee reliability and data integrity. Alternatively, in order to achieve greater performance, replication can instead be performed asynchronously. With this approach, session updates are placed on an internal queue, and when the queue is full, all changes are “flushed” to the secondary.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 30

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Session Persistence Review

Cookie

3

Client

Session stickiness

7

Primary = A

Client Proxy may fail over to any server Proxy

Proxy

1

Cluster ServerA

ServerB

2 Primary

Secondary

Primary = C

ServerC

6

Cluster ServerA

4

ServerB

ServerC

5 Secondary

Primary

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The graphic in the slide depicts a client accessing a Web application hosted in a cluster. All client requests are forwarded to the WebLogic Server cluster via a Web server plug-in: 1. The proxy load balances a client’s initial request to Server A. The Web application creates a new session for this user. 2. To provide failover services for the Web application, the primary server replicates the client’s session state to a secondary WebLogic Server in the cluster. This ensures that a replica of the session state exists even if the primary server fails. 3. The HTTP response back from Server A includes a cookie indicating that it is the primary server and Server B is the secondary. 4. Suppose at some point in the future the primary server for the user fails. 5. Server B detects the failure and becomes the primary server hosting this user’s session state. The server then selects another server and a new secondary is created. 6. On the next client request, the proxy transparently fails over to some other server in the cluster. 7. Server C promotes itself as the primary for this user’s session and it updates the client’s cookie to reflect the new primary and secondary servers.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 31

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

In-Memory Replication Review

Monitor a cluster from a specific member: serverRuntime() cluster = getMBean('/ClusterRuntime/MyCluster') print 'Alive Count: ', cluster.getAliveServerCount() print 'Alive Names: ', cluster.getServerNames() print 'Unicast Sent: ', cluster.getFragmentsSent() print 'Unicast Received: ', cluster.getFragmentsReceived() print 'Primary Sessions: ', cluster.getPrimaryCount() print 'Secondary Sessions: ', cluster.getSecondaryCount() Monitor cluster unicast communication from a specific member: serverRuntime() unicast = getMBean('/ClusterRuntime/MyCluster/ UnicastMessaging/UnicastMessagingRuntime') print 'Total Groups: ', unicast.getTotalGroupCount() print 'Group Leaders: ', unicast.getDiscoveredGroupLeaders() print 'My Leader: ', cluster.getLocalGroupLeaderName()

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Refer to the documentation for the ClusterRuntimeMBean (which implements the interface ReplicationRuntimeMBean) and UnicastMessagingRuntimeMBean for a list of all available attributes and operations.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 32

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Cluster Monitoring WLST Examples

Monitor sessions for a Web module within an enterprise application: serverRuntime() web = getMBean('/ApplicationRuntimes/HRApp/ComponentRuntimes/ ServerC_/payrollWeb') print 'Current: ', web.getOpenSessionsCurrentCount() print 'High: ', web.getOpenSessionsHighCount()

Destroy all sessions for a Web module: web = ... sessionIDs = web.getServletSessionsMonitoringIds() for sid in sessionIDs: web.invalidateServletSession(sid)

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Refer to the documentation for the WebAppComponentRuntimeMBean for a list of all available attributes and operations.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 33

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Session Monitoring WLST Examples

• •

For security purposes, WLS does not expose the real session IDs sent back to clients via JMX. To aid monitoring and troubleshooting, WLS can instead track a custom session attribute such as a username.

weblogic.xml: ... user_profile

Use WLST to view sessions with custom monitoring IDs: ... sessionIDs = web.getServletSessionsMonitoringIds() print 'Session User 1' + sessionIDs[0]

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

HTTP Sessions are identified with a monitoring ID. By default, the monitoring ID for a given HTTP session is a random string (not the same as a session ID for security reasons). This monitoring ID can be configured by setting the monitoring-attribute-name element in the weblogic.xml deployment descriptor. The toString() of the session attribute value will then be used as a monitoring ID. The monitoring-attribute-name element is useful for tagging session runtime information for different sessions. For example, you can set it to username, if you have a username attribute that is unique. The WebAppComponentRuntimeBean.getServletSessionMonitoringIds() method returns an array of session attribute values with this name. If it is not set, it returns an array of randomly generated unique strings.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 34

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Session Monitoring Attribute

•

WLDF includes application-scoped monitors that support standard diagnostic actions: – Servlet_Before_Session – Servlet_After_Session – Servlet_Around_Session

•

The additional monitor HttpSessionDebug has a built-in action that records a session’s current size any time it is updated.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

•

•

Servlet_Before_Session/Servlet_After_Session: Runs the specified diagnostic actions before or after calls to HttpSession methods getAttribute(), setAttribute(), removeAttribute(), and invalidate(). HttpSessionDebug: Inspects the target HTTP session before and after calls to HttpSession methods getAttribute(), setAttribute(), and removeAttribute(). At inspection points, the approximate session size is computed and stored as the payload of a generated event. The size is computed by flattening the session to a byte array. If an error is encountered while flattening the session, a negative size is reported.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 35

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Session Instrumentation

• •

Servers generate info messages to trace the addition and removal of members to and from the cluster. Use debug flags for more detailed log messages. Flag

Description

DebugCluster

View very low level details of cluster communications.

DebugClusterHeartbeats

View how often heartbeats are sent from this server.

DebugCluster Announcements

Trace all heartbeats sent from and received by this server.

DebugLeaderElection

Trace unicast group initialization and updates.

DebugHttpSessions

Trace the creation, updating, invalidation, persistence, replication, and size of all HTTP sessions.

DebugReplication

More detailed trace of session replication activities

DebugReplicationDetails

View some additional details on session replication decisions.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The following is an example of a cluster informational log message: The flags DebugCluster, DebugClusterHeartbeats, DebugClusterAnnouncements, DebugLeaderElection, DebugReplication, and DebugReplicationDetails are found in the weblogic.core.cluster scope. The weblogic.cluster scope contains flags for the cluster-leasing feature, which is used to support WebLogic Server’s service migration and whole server migration capabilities. The DebugHttpSessions flag is found in the weblogic.servlet.internal.session scope. Some of these cluster debug flags, particularly those involved with heartbeat communications, generate a huge amount of log information in a short period of time. Consequently, you may also need to increase the log file size as well as the maximum number of log files.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 36

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Cluster Debug Flags

Normal log messages with DebugClusterHeartbeats and DebugClusterAnnouncements enabled: ... ... ... ... ... ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

When a cluster member becomes unavailable and other members stop receiving heartbeat messages, an informational log message is generated: The debug trace above depicts the following activities: 1. As part of starting up, the current server prepares to broadcast its local attributes, including its internal JVM ID, to other cluster members. In this example, the current server is MyServer2. 2. The initial announcement with this server’s attributes is sent. 3. The current server begins broadcasting heartbeat messages to other cluster members. 4. The current server begins receiving announcements and heartbeat messages from other cluster members. In this example, other running servers in the cluster include MyServer1 and MyServer3. 5. Heartbeats continue to be sent and received.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 37

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Typical Cluster Heartbeat Trace

• •

Each server in a cluster is assigned a unique internal JVMID. Each replication task is assigned a unique internal ID.

DebugHttpSessions and DebugReplication enabled on primary server: ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The debug trace above depicts the following activities: 1. The application requests a new session during the current request. The server creates a new session and assigns it a unique session ID, which includes the JVM ID of the current server. 2. Because in-memory replication is enabled for this application, the current server assigns itself as the primary for the new session. An initial replication ID is also generated. 3. Another member of the cluster is selected as a secondary server for this session. The current server sends the initial session data to the secondary. This request completes and the response is sent back to the client along with the session ID. 4. A subsequent request updates the session. A new replication ID is generated. 5. The secondary server assigned to this session is sent the latest session changes.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 38

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Typical Replication Trace: Primary

DebugHttpSessions and DebugReplication enabled on secondary server: ...

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The debug trace above depicts the following activities: 1. The current server receives a request from another cluster member to initialize a secondary copy of a session. The request includes the replication ID that was generated on the primary server. 2. The server confirms that it is now the secondary for this session. 3. The server receives the initial session data for the secondary copy. 4. Some time later the server receives a message from the primary server with session modifications. Once again, the message includes a replication ID for tracking purposes. 5. The server updates the secondary copy of the session.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 39

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Typical Replication Trace: Secondary

• •

Replication problems often result in the loss of session data (requiring the user to log in again, for example). Typical culprits include: – – – – – – – –

Cluster configuration/deployment is not uniform. Cluster members are not using the same release of WLS. Network or firewall issues that result in missed heartbeats Invalid session persistence settings Session and/or cookie timeout settings are too low. Proxy and WLS session cookie names do not match. Application is not using HttpSession API appropriately. Application is storing non-serializable objects in the session.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

All servers in the cluster must have the same major version number, but can have different minor version numbers and service packs. The cluster’s administration server is typically not configured as a cluster member, but it should run the same major version of WebLogic Server used on the managed servers. Also make sure the value of CLASSPATH is the same on all managed servers in the cluster. CLASSPATH is set by the setDomainEnv (and therefore commEnv as well) script, which is used by other server startup scripts. Network issues can keep cluster group leaders from properly distributing heartbeat messages throughout the cluster. This scenario can lead to unnecessary work concerning WebLogic’s high availability services. For example, a proxy plug-in’s dynamic server list may be missing candidate servers. Or servers may replicate HTTP data excessively. To help uncover the culprit in your network topology, use the console or WLST to monitor the general and unicast runtime data of every cluster member.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 40

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Common Replication Issues

•

Web applications or their underlying application frameworks (JSF, for example) must use the correct APIs to support session persistence. HttpSession Method

WebLogic Behavior

setAttribute(name,value)

Persist or replicate new or updated attribute.

removeAttribute(name)

Delete attribute from secondary or persisted copy.

invalidate()

Mark session for deletion along with its secondary or persisted copy.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

An HttpSession object is created if one does not already exist for that client when the request.getSession(true) method is called with the argument true. The session object lives on WebLogic Server for the lifetime of the session, during which the session object accumulates information related to that client. Your servlet adds or removes information from the session object as necessary. A session is associated with a particular client. Each time the client visits your servlet, the same associated HttpSession object is retrieved when the getSession() method is called. You can store data in an HttpSession object using name=value pairs. Data stored in a session is available through the session. To add or overwrite a named attribute, use the setAttribute() method. To remove a named attribute altogether, use the removeAttribute() method. If your application deals with sensitive information, consider offering the ability to log out of the session. This is a common feature when using shopping carts and Internet email accounts. When the same browser returns to the service, the user must log back in to the system. Use the invalidate() method to mark the current session for deletion. Do not use session persistence for storing long-term data between sessions. In other words, do not rely on a session still being active when a client returns to a site at some later date. Instead, your application should record long-term or important information in a database.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 41

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

HttpSession API Overview

•

•

•

WebLogic’s session persistence capabilities require Java objects to be sent over I/O channels (network, file, database, and so on). JVMs allow a Java object to be sent over I/O only if its type is declared as Serializable; otherwise, the JVM throws an error. Most standard Java types can be serialized; exceptions are the I/O objects themselves (connections, files, sockets). JVM JVM

Socket File

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

To serialize an object means to convert its state to a byte stream so that the byte stream can be reverted back into a copy of the object. A Java object is serializable if its class or any of its superclasses implements either the java.io.Serializable interface or its subinterface, java.io.Externalizable. Deserialization is the process of converting the serialized form of an object back into a copy of the object. You can add any Java descendant of Object as a session attribute and associate it with a name. However, if you are using session persistence, your attribute value objects must implement java.io.Serializable.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 13 - 42

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Serialization Overview

• •

WebLogic logs error and debug messages when it detects that a session attribute cannot be persisted or replicated. The debug messages include the attribute name along with the original Java exception and stack trace.

Log messages on the primary server during replication: Counter Monitors. Then click New. 3. Enter the following information: - Name: The name of the monitor - Monitored MBean Type: The runtime MBean type that defines the attribute you want to monitor - Monitored MBean Name: The name of the MBean instance that you want to monitor. If you leave the name undefined, WebLogic Server monitors all instances of the MBean type that you specify in Monitored MBean Type. Alternatively, use the User Entered MBean name field to enter a name directly. - Monitored Attribute Name: The name of an MBean attribute to monitor. This attribute must be in the WebLogic Server MIB. - Polling Interval: The frequency (in seconds) that the server checks the attribute value

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting A - 14

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Creating Trap Monitors

1

2 3

Location of management console

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Trap destinations specify the SNMP management station to which a WebLogic Server SNMP agent sends notifications, both automatically generated ones and those generated by custom monitors (counters, gauges, and so on). 1. Click the name of an existing SNMP agent. 2. Click Configuration > Trap Destinations. Then click New. 3. Enter the following information: - Name: The name of this trap destination. This value is for your identification purposes only. - Community: The “password” that a WebLogic Server SNMP agent sends to the SNMP manager when the agent generates SNMPv1 or SNMPv2 notifications. The community name that you enter in this trap destination must match the name that the SNMP manager defines. For SNMPv3, use the Security Name and Security Level fields instead. - Host: The DNS name or IP address of the computer on which the SNMP manager is running - Port: The UDP port on which the SNMP manager is listening

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting A - 15

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Creating Trap Destinations

• •

SNMP V1 and V2 simply support clear text passwords called “communities.” SNMP V3 supports: – Authentication based on a username/password – MD5 or SHA hashing of authentication data to prevent message tampering – DES or AES encryption based on a privacy password

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

The security features that are available for SNMP depend on which SNMP protocol an agent uses to communicate with managers. To ensure that an SNMP manager requesting data from the WebLogic SNMP agent has permission to obtain the data, and to verify that the agent has permission to send notifications to a target manager, SNMPv1 and SNMPv2 use clear-text passwords called community names. In the SNMPv3 protocol, both SNMP agent and manager must encode identical credentials in their PDUs for the communication to succeed. The credentials include several tokens: a username, an SNMP engine ID, an authorization protocol, and an optional privacy password, all of which are encrypted before being transported over the network.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting A - 16

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

SNMP Security

Use V1,V2 community strings?

Use V3 authentication?

Use V3 encryption?

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Because SNMPv1 and SNMPv2 use clear-text passwords, the level of security is weak. If you can use SNMPv3 to communicate with managers, consider disabling SNMPv1 and SNMPv2 by disabling Community Based Access Enabled for each SNMP agent. • Community Prefix: The password that you want this SNMP agent to use to secure SNMPv1 or v2 communication with SNMP managers. SNMPv3 does not use community names, so this field is ignored. • Authentication Protocol: The protocol that this SNMP agent uses to ensure that only authorized users can request or receive information about your WebLogic Server domain. Applicable only with SNMPv3. The protocol also ensures message integrity and prevents masquerading and reordered, delayed, or replayed messages. If you do not choose an authentication protocol, the SNMP agent does not authenticate incoming SNMPv3 requests; anyone can use SNMPv3 to retrieve information about your WebLogic Server domain. • Privacy Protocol: The protocol that this SNMP agent uses to encrypt and unencrypt messages. If you do not choose a privacy protocol, communication between this agent and managers can be viewed (but not altered) by unauthorized users.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting A - 17

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring Agent Security

•

Register username/passwords that will be: – Granted access to the SNMP agent – Included when sending notifications – Used as privacy passwords for encryption

•

Usernames must map to existing WLS users that have the appropriate monitoring privileges.

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

In WebLogic Server, SNMPv3 agents work with the domain’s security realm to secure communication. The SNMP agent decodes SNMP credentials in requests and passes the SNMP username to the security realm. The security realm maps the SNMP username to a WebLogic Server user, authenticates the user, and authorizes access to monitoring data in the domain. To map the SNMP credentials to a user in a WebLogic Server security realm, you create a credential map. Under Domain Structure, click SNMP. Then click the Security tab. After clicking the New button, select Authentication from the Credential Mapping Type list. In User Name, enter the name of the WebLogic Server user. In SNMP Password, enter the authentication password that SNMP managers will send in their requests. Finally, repeat these steps to create a “Privacy” credential mapping and enter the privacy password that SNMP managers will send in their requests. To optimize performance, an SNMPv3 agent caches the credential maps that correlate WebLogic Server users with SNMP credentials. To make sure that the cache contains the latest set of SNMP credentials, an agent periodically invalidates its cache. After the cache is invalidated, the next time the agent requests credentials, it regenerates its cache.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting A - 18

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring SNMP V3 Credentials

Use V1,V2 community strings? Username for V3 authentication

Password for user stored in credential map

Use V3 security?

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

If the SNMP agent sends SNMPv1 or v2 notifications and the management console requires a specific community to accept traps, use the Community field in the corresponding trap destination in WebLogic Server. To use a specific SNMPv3 authentication and or privacy protocol when sending responses or notifications, you must also configure the security level of your trap destinations. In Security Name, enter the username on whose behalf the WebLogic SNMP agent sends notifications. The username must be the name of an existing WebLogic Server user for whom you have created an SNMP credential map. When the WebLogic SNMP agent prepares a notification, it uses the credential map to look up and encode SNMP credentials. In the Security Level list, select a security level that is equal to or higher than the security level that is configured for receiving requests from SNMP managers. For example, if the WebLogic SNMP agent requires incoming SNMPv3 requests to use the authentication protocol, the security level for this trap destination must either require authentication or both authentication and privacy.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting A - 19

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Configuring Trap Destination Security

WLS includes a command-line SNMP utility that supports: • V1, V2, or V3 messages and all message types • Polling and capturing traps (UDP or TCP) • MIB modules (XML format) • V3 security features Print Help for a specific command: java weblogic.diagnostics.snmp.cmdline.Manager SnmpWalk -?

Poll a management object by using the WLS MIB: java weblogic.diagnostics.snmp.cmdline.Manager SnmpWalk -M /weblogic/diagnostics/snmp/mib -m BEA-WEBLOGIC-MIB -h localhost -p 7090 –O -u myuser -A mypassword -e myEngineID safAgentRuntimeMessagesPendingCount

Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

WebLogic Server provides a command line utility that offers many of the same features as an SNMP manager. You can use this utility to test and troubleshoot the configuration of your SNMP agents in a WebLogic Server domain. Like most WLS tools, first open a command prompt (shell) and invoke the following script: \server\bin\setWLSEnv.sh (or setWLSEnv.cmd on Windows). The script adds a supported JDK to the shell’s PATH environment variable and adds WebLogic Server classes to the CLASSPATH variable. Available commands include: • SnmpGet: Retrieves the value of one or more MIB variables. This command does not accept OIDs for managed objects. You can specify an optional interval at which this command repeatedly retrieves the value of the specified variable. • SnmpInform: Constructs a test INFORM notification and distributes it to an SNMP manager or trap monitor. • SnmpGetBulk: Returns a collection of MIB variables by repeatedly invoking SnmpGetNext in a pattern that you specify. Use the -Bn and -Bm arguments and one or more OIDs to specify the pattern. • SnmpTrapMonitor: Starts a process that listens for notifications. Prints each notification that it receives to standard out.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting A - 20

Oracle University and Sentra inversiones y servicios LTDA use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

WLS SNMP Utility