THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Activity Guide
D61523GC20
Edition 2.0
March 2011
D72555
Oracle University and Sentra inversiones y servicios LTDA use only
Oracle WebLogic Server 11g: Diagnostics and Troubleshooting
Disclaimer This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle. The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not warranted to be error-free. Restricted Rights Notice If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract. Trademark Notice Oracle and Java are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Author Bill Bell Technical Contributors and Reviewers Will Lyons, TJ Palazzolo, Serge Moiseev This book was published using:
Oracle Tutor
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 1 .....................................................................................................................................1-1 Practices for Lesson 1....................................................................................................................................1-2 Practices for Lesson 2 .....................................................................................................................................2-1 Practices for Lesson 2....................................................................................................................................2-2 Practice 2-1: Connecting to the Classroom Grid ............................................................................................2-3 Practice 2-2: Developing a Custom Monitoring Script ....................................................................................2-5 Solution Instructions .......................................................................................................................................2-9 Practices for Lesson 3 .....................................................................................................................................3-1 Practices for Lesson 3....................................................................................................................................3-2 Practice 3-1: Using Guardian to Evaluate a Domain ......................................................................................3-3 Solution Instructions .......................................................................................................................................3-7 Practices for Lesson 4 .....................................................................................................................................4-1 Practices for Lesson 4....................................................................................................................................4-2 Practice 4-1: Harvesting Diagnostic Metrics ...................................................................................................4-3 Solution Instructions .......................................................................................................................................4-8 Practice 4-2: Monitoring Diagnostic Metrics ...................................................................................................4-9 Practices for Lesson 5 .....................................................................................................................................5-1 Practices for Lesson 5....................................................................................................................................5-2 Practice 5-1: Configuring and Monitoring Diagnostic Events .........................................................................5-3 Solution Instructions .......................................................................................................................................5-6 Practice 5-2: Tracing a Client Request ...........................................................................................................5-7 Solution Instructions .......................................................................................................................................5-11 Practices for Lesson 6 .....................................................................................................................................6-1 Practices for Lesson 6....................................................................................................................................6-2 Practice 6-1: Troubleshooting a Running JVM ...............................................................................................6-3 Practice 6-2: Troubleshooting Applications on JRockit ..................................................................................6-7 Practices for Lesson 7 .....................................................................................................................................7-1 Practices for Lesson 7....................................................................................................................................7-2 Practice 7-1: Investigating Classpath Problems .............................................................................................7-3 Practices for Lesson 8 .....................................................................................................................................8-1 Practices for Lesson 8....................................................................................................................................8-2 Practice 8-1: Investigating Server Problems ..................................................................................................8-3 Solution Instructions .......................................................................................................................................8-7 Practices for Lesson 9 .....................................................................................................................................9-1 Practices for Lesson 9....................................................................................................................................9-2 Practice 9-1: Investigating JDBC Problems ...................................................................................................9-3 Solution Instructions .......................................................................................................................................9-8 Practices for Lesson 10 ...................................................................................................................................10-1 Practices for Lesson 10..................................................................................................................................10-2 Practice 10-1: Investigating JMS Problems ....................................................................................................10-3 Solution Instructions .......................................................................................................................................10-10
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle WebLogic Server 11g: Diagnostics and Troubleshooting Table of Contents i
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Table of Contents
Practices for Lesson 12 ...................................................................................................................................12-1 Practices for Lesson 12..................................................................................................................................12-2 Practice 12-1: Investigating Node Manager Problems ...................................................................................12-3 Solution Instructions .......................................................................................................................................12-8 Practices for Lesson 13 ...................................................................................................................................13-1 Practices for Lesson 13..................................................................................................................................13-2 Practice 13-1: Investigating Proxy Problems ..................................................................................................13-3 Solution Instructions .......................................................................................................................................13-7 Practice 13-2: Investigating Cluster Replication Problems .............................................................................13-8
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Oracle WebLogic Server 11g: Diagnostics and Troubleshooting Table of Contents ii
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 11 ...................................................................................................................................11-1 Practices for Lesson 11..................................................................................................................................11-2 Practice 11-1: Investigating SSL Problems ....................................................................................................11-3 Solution Instructions .......................................................................................................................................11-9 Practice 11-2: Investigating Security Realm Problems ...................................................................................11-10 Solution Instructions .......................................................................................................................................11-15
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 1
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 1
Chapter 1 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 1
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 1
Practices Overview
There are no practices for this lesson.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 1
Chapter 1 - Page 2
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 2
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2
Chapter 2 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 2
Practices Overview In the practices for this lesson, you connect to your practice machine for the first time. You then write a WLST script to monitor a running server.
Naming Conventions The instructions for the course practices use variable names to refer to frequently used locations on your file system. Some of these are also defined as Linux environment variables. Other variable names will be introduced in later practices as needed. Variable Path /u01/app/oracle/Middleware/11.1.1
/wlserver_10.3
/home/oracle/wls11g_trouble
/labs/LabXX_YY, where XX_YY is the current practice number
/work
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2 Chapter 2 - Page 2
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 2
Duration: 10 minutes Skills Learned At the end of this practice, you should be able to: • Identity your classroom virtual machine (VM) instance • Connect to a classroom virtual machine by using an NX client • Establish an NX session Overview Oracle University employs a Grid VM infrastructure to allow students to work with complex lab environments from anywhere in the world. You will be assigned a specific Linux VM instance on this Grid for you to perform the subsequent course practices. In this practice, you use NoMachine, an NX client, to interact with this remote VM as if it were running on your local machine. Instructions 1. Obtain your VM hostname, username, and password from your instructor. Grid hosts are typically of the form egNNNN.us.oracle.com. 2. From the Windows Start menu, select All Programs > NX Client for Windows > NX Connection Wizard. 3. Click Next. 4. Enter the following values (NNNN is your assigned VM instance number):
5.
6.
Field
Value
Session
egNNNN
Host
egNNNN.us.oracle.com
Click Next. Select the GNOME option and click Next.
Click Finish. The NX Login window appears. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2 Chapter 2 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 2-1: Connecting to the Classroom Grid
Enter your assigned Username and Password. Confirm that your new session, egNNNN, is selected. Click Login.
8.
Confirm that your VM Linux desktop is displayed. Additional Notes: − When you leave your remove session, click the “X” in the top-right corner of the window. You do not need to Terminate your remote session, instead you may choose Disconnect, which leaves programs running on the remote system. You may leave programs running on the remote system for the duration of the course. − After a period of inactivity, your remote session may lock. If this occurs, you must simply enter your Password to resume the session.
Solution Instructions No solution exists for this practice.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2 Chapter 2 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
7.
Duration: 35 minutes Skills Learned At the end of this practice, you should be able to: • Navigate the WLS runtime MBean hierarchy • Retrieve and display MBean attributes • Use loop and condition Jython constructs Overview The Oracle Medical Records Java EE application, abbreviated MedRec, provides patients with the ability to view current and previous medical examinations and prescriptions over the Web. MedRec, like all Java EE applications, is dependent on various server resources. These resources include JDBC data sources, JMS queues and topics, and shared libraries. In this practice, you develop a custom WLST script that periodically monitors the health of the MedRec application, its host server, and its supporting resources. The MedRec domain infrastructure used in this practice is depicted in the following diagram:
Instructions 1. Initialize the MedRec server infrastructure. a. Launch the Lab Framework command shell by executing the /bin/prompt.sh file. Tip: Remember that you can refer to the Naming Conventions section earlier in this guide for the definition of variables such as . b. c.
Change the current directory to . Execute the following: ant setup_exercise Note: When the Ant script completes its work you will see the message “BUILD SUCCESSFUL” in the Lab Framework command shell. Important: You must log in to each WebLogic Server as it comes up (see below). The Lab Framework creates files and folders in your directory. The framework: − Starts the Oracle database and loads the MedRec schema − Creates a skeleton MedRec domain from a template Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2 Chapter 2 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 2-2: Developing a Custom Monitoring Script
− − − − − 2.
Creates a JDBC data source Creates a JMS server and module Creates a work manager Deploys the shared libraries Deploys the MedRec application
− Copies a test client to , which you will use in a later practice. Connect to the server runtime MBean hierarchy. a. Create a new folder: /scripts. b. Launch an editor and create a new file: /scripts/monitorMedRec.py. c.
3.
Use the connect() WLST command to connect to the MedRecSvr1 managed server (port 7021). Tip: Remember, the three arguments you will be using with connect() are an adminlevel username, password, and server URL. Each argument must be delimited (for example in single quotes). The protocol of the URL, if omitted, defaults to t3, which is what you want. d. Use the serverRuntime() command to switch to the server’s runtime hierarchy. Print server, application, and data source MBean attributes. a. Create a new while loop whose contents iterate indefinitely, as shown below. while 1: # Loop contents here
b.
Tip: Remember that in Jython, you use whitespace (a tab, for example) to denote the contents of a loop or condition. Within the loop, locate this server’s ThreadPoolRuntime MBean and retrieve its throughput attribute. For example: threadPool = getMBean('/ThreadPoolRuntime/ThreadPoolRuntime') throughput = threadPool.getThroughput()
c.
Similarly, retrieve the status attribute of the following application MBean: /ApplicationRuntimes/medrec/ComponentRuntimes/MedRecSvr1_/medrec Tip: This MBean represents the Web module with the context path /medrec, found in the enterprise application medrec.ear.
d. e.
f.
Get the value of the openSessionsCurrentCount attribute from the same MBean. Retrieve the following MBean: /JDBCServiceRuntime/MedRecSvr1/JDBCDataSourceRuntimeMBeans/ MedRecGlobalDataSourceXA
Get the following attributes from this MBean: state numAvailable activeConnectionsCurrentCount Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2 Chapter 2 - Page 6
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
− Starts the administration server and a managed server (be sure to log in as weblogic/Welcome1 as each server boots)
Print all the collected attributes. Tip: You can convert numeric values to text by using either of the following techniques: print 'Text ', num print 'Text ' + str(num) Tip: Optionally, you can format floating point values such as throughput by using code like the following: print '%.1f'%(throughput)
h.
Add the following import statement to the top of the file: import time as jythontime
i.
Add a 1 second pause within the while loop: jythontime.sleep(1)
4.
Tip: Alternatively, you can call the equivalent operation in the Java language: Thread.sleep(1000). j. Save your script. Monitor the server without any load. a. Locate your Lab Framework prompt or start a new one. Navigate to /scripts. b. Use WLST to run your new script: java weblogic.WLST monitorMedRec.py c.
5.
Confirm that the current status of the application is DEPLOYED and the state of the data source is Running. Correct any errors. Tip: If you need it, a completed script can be found at /solution. Tip: If you must restart your servers at any point in the course, use the startWebLogic.sh and startServer1.sh scripts found in your domain. d. Leave the script running. Monitor the server under load. a. Launch a second Lab Framework prompt by using /bin/prompt.sh.
b. Navigate to /client. c. d.
e.
Execute the runclient.sh script. Note: You may ignore any warnings in the MedRecSvr1 window about logging. Return to your monitoring script output. Confirm that the number of active sessions has increased by 1. The data source active connections may have increased by 1 as well. After the client completes its execution, if the number of active connections went to 1, it should return to 0. Note: You may not see the active database connections go up. It is very possible that a database connection is retrieved, used, and returned within the 1 second sleep time. Execute the runclients.sh script. This script simulates 10 concurrent users.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2 Chapter 2 - Page 7
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
g.
g.
Return to your monitoring script output. Confirm that the number of active sessions has increased by 10. Also, depending on the responsiveness of the server and database, you may witness the number of data source connections increase as well. Kill the monitoring script.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
f.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2 Chapter 2 - Page 8
1.
Launch the Lab Framework command shell by executing the /bin/prompt.sh file.
2. 3.
Change the current directory to . Execute the following: ant setup_solution The Lab Framework performs the same tasks as described in the “Initialize the MedRec server infrastructure” section. The solution monitoring script can be found at /solution/monitorMedRec.py.
4.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2 Chapter 2 - Page 9
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 2
Chapter 2 - Page 10
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 3
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 3
Chapter 3 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 3
Practices Overview In this practice you will use Guardian to evaluate an existing domain. Naming Conventions In addition to the variable names listed earlier, these practices use the following variable name to refer to a frequently used location on your file system. This value is also defined as a Linux environment variable. Variable
Path
/u01/app/oracle/Guardian/10.3.2
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 3 Chapter 3 - Page 2
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 3
Duration: 35 minutes Skills Learned At the end of this practice, you should be able to: • Update Guardian with newer problem signatures • Deploy the Guardian agent to a domain • Activate and inventory a domain • Use Guardian to evaluate a domain for problems • Correct a problem identified by Guardian Overview Regularly running Oracle Guardian allows you to quickly detect and fix common problems with your domain’s configuration. Guardian consists of two parts: a graphical interface and an agent that runs on every server in your domain. In this practice, you activate the Guardian agent within your domain, and then use the Guardian tool to connect to and evaluate your domain. You also use this opportunity to correct a problem detected by Guardian. Normally, you would use your Oracle Support account to download the latest Guardian problem signatures from the Web, but due to the constraints of the lab environment, a signature pack has already been downloaded for you. Tip: Unless otherwise indicated, all labs require the MedRecDomain domain. If you do not have this domain, follow the Solution Instructions for the “Developing a Custom Monitoring Script” practice. Instructions 1. Initialize the lab environment. a. Start your administration server, if not already started. b. Launch a Lab Framework command shell, if not already open. c. Change the current directory to . d. Execute the following: ant setup_exercise The Lab Framework updates the MedRecDomain domain with a configuration problem that Guardian will later detect. 2. Update and launch Guardian. a. Locate the /repository/archives directory. Rename the file weblogic-signatures.jar to weblogic-signatures.jar.old.
b. Locate the /resources/weblogic-signatures.jar file. Copy this file. c. Paste the newer signature file into the /repository/archives directory.
d. e.
Why? You are replacing the signature file that came with the Guardian installation with a later signature file. Launch a Terminal and navigate to . Execute the following: export PATH=$MIDDLEWARE_HOME/jdk160_18/bin:$PATH ./guardian Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 3 Chapter 3 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 3-1: Using Guardian to Evaluate a Domain
3.
When prompted, enter the following new Workspace: /guardianWorkspace. Then click Finish.
Tip: Replace with the actual path. Deploy the Guardian agent. a. Launch a Web browser and access the WLS administration console: http://localhost:7020/console b. Log in as weblogic/Welcome1 and Lock the console. c.
In the Domain Structure panel, click the domain name, MedRecDomain.
d. e. f. g.
Select the Enable Oracle Guardian Agent check box and click Save. Activate your changes. Kill (Ctrl + c) your administration and managed servers. Launch two Terminals and restart the servers: cd /home/oracle/wls11g_trouble/work/domains/MedRecDomain ./startWebLogic.sh cd /home/oracle/wls11g_trouble/work/domains/MedRecDomain ./startServer1.sh
4.
Tip: Alternatively, open two tabs within a single Terminal window. Tip: To avoid supplying credentials each time a server is started, recall that you can optionally create a boot.properties file for each of your servers. Activate and inventory a domain. a. Return to the Guardian application. b. On the main toolbar, click Activate.
c. d.
5.
Enter the location and credentials of your administration server and click Finish. An initial domain inventory is generated and displayed. Browse the contents of the Servers and JDBC sections. Evaluate a domain. a. Click the Evaluate button on the toolbar. The Evaluation Wizard appears. b. Confirm that MedRecDomain is selected and enter the Domain Credentials. c. In the Bundle column, select the All Signatures option. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 3 Chapter 3 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
f.
e.
Select the Create Shortcut check box. Click Finish. When the evaluation finishes, an evaluation summary is generated and displayed. Confirm that you have at least one Detected Signature. For example:
f.
6.
Locate and select the signature concerning HTTP POST. Tip: To avoid scrolling, maximize the evaluation summary panel. g. Read the details of this problem signature in the Description section. If time permits at the end of the practice, you can read about any remaining detected signatures as well. Resolve a detected problem. a. Return to the console and Lock it. Tip: You will have to log in again since the admin server has been restarted. b. Locate and select the MedRecSvr1 server. c. Click the Protocols > HTTP tab. d. Set the value of Max Post Size to 1048576. Click Save.
e. f. g. h. i. j.
Why? The value of -1 means an unlimited post size. You are setting it to a more reasonable 1 MB. Repeat the previous steps to update the Max Post Size of the MedRecAdmSvr server. Activate your changes. Note: You may ignore any errors in the server windows. Kill and restart your servers. Return to Guardian. Select File > Close All. Click the Shortcut Explorer tab.
k. l.
Double-click the shortcut for MedRecDomain. Perform another evaluation. Verify that the HTTP POST (MaxPostSize) problem is no longer detected. m. When you are finished with the practice, do the following: Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 3 Chapter 3 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
d.
Exit Guardian. Use the console to disable the Guardian agent for the domain. Restart both servers.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
1) 2) 3)
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 3 Chapter 3 - Page 6
There is no solution for this practice. If you only partially completed the practice instructions, however, then you need to undo what you have done by following these steps: 1. Launch the Lab Framework command shell by executing the /bin/prompt.sh file. 2. 3.
Change the current directory to . Execute the following: ant setup_solution The Lab Framework: −
Makes a backup copy of your current work
− −
Starts database and WebLogic servers, if not already started Disables the Guardian agent
−
Updates server protocol settings
−
(Re)deploys the MedRec application
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 3 Chapter 3 - Page 7
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 3
Chapter 3 - Page 8
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 4
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4
Chapter 4 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 4
Practices Overview In these practices you will set up WebLogic Diagnostic Framework harvesters, watches, and notifications. You will also use the Monitoring Dashboard to view diagnostic information as charts and graphs.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 4
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4 Chapter 4 - Page 2
Duration: 40 minutes Skills Learned At the end of this practice, you should be able to: • Setup an existing JMS queue for diagnostic notifications • Create a system diagnostic module • Configure metric collectors for several MBean types • Trigger notifications under various MBean watch conditions • Access diagnostic archives and inspect recorded metrics Overview Patients have reported intermittent problems with the MedRec application. Before contacting the development team and starting to troubleshoot, however, the IT group would like to get more insight into the server and application. The WebLogic Diagnostics Framework (WLDF) allows you to collect and record server and application metrics over time for later analysis. WLDF also supports watch conditions that can trigger notifications, such as posting a JMS message or sending an SNMP trap. In this practice, you configure a new system-wide WLDF module along with several metric collectors (harvesters), watches, and notifications. The diagnostic configuration is shown in the following diagram:
Instructions 1. Create a JMS queue for diagnostic notifications. a. Copy the following file to /client: /resources/runclients.sh Overwrite any previous version of the file. b. Launch a Lab Framework command shell, if not already open. c. Navigate to /resources and execute the createWLDFQueue.py WLST script. d. Access the admin console. e. From the Domain Structure panel, select Services > Messaging > JMS Modules. f. Select the MedRec-jms module. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4 Chapter 4 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 4-1: Harvesting Diagnostic Metrics
2.
3.
Confirm that a new queue is available named WLDFNotificationQueue. Note its JNDI Name:
Create a system diagnostic module. a. Lock the console. b. From the Domain Structure panel, select Diagnostics > Diagnostic Modules. c. Click New. d. Name the module MedRecDiagnostics and click OK. e. Edit the new module. f. Click the Targets tab. g. Select MedRecSvr1 and click Save. Define a metric collector (harvester). a. Click the Configuration > Collected Metrics tab. b. Set the Sampling Period to 30000 (30 seconds) and click Save. c. Click the New button. d. Click Next. Note: Since this diagnostic module is targeted to a managed server, only the Server Runtime MBeans apply. e. From the select box, select the weblogic.management.runtime. JDBCConnectionPoolRuntimeMBean option. Click Next. f. Move the following attributes from the Available column to the Chosen column: ActiveConnectionsCurrentCount CurrCapacity LeakedConnectionCount Click Next. g. Note the available values for Collected Instances, but make no selection. Click Finish. Note: If you do not identify specific MBean instances, WLDF collects the selected attributes from all available instances. h. Click New once again. i. Repeat this process to define the following additional metric: Field
Value
MBean Server Location
ServerRuntime
MBean Type
weblogic.management.runtime. WebAppComponentRuntimeMBean Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4 Chapter 4 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
g.
Collected Instances
Select the option that contains the text: Name=MedRecSvr1/medrec, ServerRuntime=MedRecSvr1
Create a diagnostic JMS notification. a. Click the Configuration > Watches and Notifications tab. b. Click the Notifications tab at the bottom of the screen.
c. d. e. f.
5.
OpenSessionsCurrentCount
Click New. For the Type field, select the JMS Message option. Click Next. Enter JMSNotification as the Notification Name, and click Next. Enter the following values: Field
Value
JMS Destination JNDI Name
com.medrec.jms.WLDFNotificationQueu e
Connection Factory JNDI Name
weblogic.jms.XAConnectionFactory
Click Finish. Tip: Be sure you type these values exactly as shown. Create diagnostic watches. a. Click the Watches tab at the bottom of the screen. Then click New. b. Enter JDBCWatch for the Watch Name, and confirm that the Watch Type is Collected Metrics. Click Next. c. Click the Add Expressions button. d. Click Next. e. Select the same JDBCConnectionPoolRuntimeMBean that is being collected and click Next. f. Click Next to watch all instances of this MBean. g. Enter the following values: Field
Value
Message Attribute
ActiveConnectionsCurrentCoun t
Operator
>
Value
1
Click Finish. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4 Chapter 4 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
4.
Collected Attributes
k.
6.
Click Next. Select the Use a Manual Reset Alarm option and click Next. Move the JMSNotification option from the Available column to the Chosen column. Click Finish. Repeat this process to configure a second watch for the Web application metric being collected: Field
Value
Watch Name
SessionWatch
Watch Type
Collected Metrics
MBean Server Location
ServerRuntime
MBean Type
weblogic.management.runtime. WebAppComponentRuntimeMBean
Instance
Select the option that contains the text: Name=MedRecSvr1/medrec, ServerRuntime=MedRecSvr1
Message Attribute
OpenSessionsCurrentCount
Operator
>
Value
10
Config Watch Alarm
Use a manual reset alarm
Chosen
JMSNotification
l. Activate your changes. Trigger WLDF notifications. a. From a Lab Framework prompt, execute /client/runclients.sh. Note: The test clients require a few minutes to finish. Recall that the metric collector is running every 30 seconds. b. From the administration console, return to the MedRec-jms JMS module. Tip: Remember that you can return to a prior screen by using the breadcrumb trail at the top of the console. c. Click the WLDFNotificationQueue. d. Click the Monitoring tab. e. Confirm that at least one message has been added to the queue.
f.
Tip: If there are no current messages try running the test client again. Locate the shell running MedRecSvr1. Confirm that the watch notification was also written to the server log. For example: Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4 Chapter 4 - Page 6
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
h. i. j.
7.
Within the message, find the collected value of either the JDBC or session MBean metric. g. Return to the console and Lock it. h. Navigate to your MedRecDiagnostics module. i. Update the module’s configuration and disable the metric collector. Tip: From the Configuration > Collected Metrics tab, uncheck Enabled. j. Similarly, disable the watch and notification component of the module. k. Activate your changes. Browse collected metrics by using the admin console. a. From the Domain Structure panel, select Diagnostics > Log Files. b. Locate and select the HarvestedDataArchive log file for the MedRecSvr1 server. Click View. c. Click the Customize this table link. d. For Time Interval, select Last 15 Minute(s). e. Under Column Display, update the Chosen column to contain only these fields: Date Attribute Value f. For Number of rows displayed per page, select 100. Click Apply. g. Browse the harvester entries. Verify that the four MBean attributes are being collected approximately every 30 seconds. h. Click Customize this table again. i. For WLDF Query Expression, enter the following: ATTRNAME = 'CurrCapacity' Click Apply. Tip: Be sure to use single quotes. j. Confirm that now only this attribute is displayed. k. Update the WLDF Query Expression again and replace it with the following: TYPE LIKE '%WebAppComponentRuntimeMBean' Tip: Be sure to use single quotes. l. Confirm that now only the OpenSessionsCurrentCount attribute is displayed.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4 Chapter 4 - Page 7
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
... Log Files, select HarvestedDataArchive for MedRecSvr1 and click View. Click Customize this table. For WLDF Query Expression, enter: ATTRNAME = 'CurrCapacity' and click Apply. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4 Chapter 4 - Page 10
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
2.
h. Click the Stop All button to stop collecting this view’s MBean metrics. Reactivate the metric collector. a. Return to the administration console and Lock it. b. Locate and edit your existing diagnostics module, MedRecDiagnostics. c. Click the Configuration > Collected Metrics tab. d. Select the Enabled check box and click Save. e. Activate your changes. f. Kill and restart MedRecSvr1 to reset its runtime MBean attributes. g. To create more interesting data, from a Lab Framework prompt execute /client/runclients.sh. Define a custom view for harvested metrics. a. Return to the Monitoring Dashboard.
5) At the top-right of the chart, click the down arrow for the menu and select Properties. 6) Under Time Range select Custom and enter the Start Time. Use the correct format. For example: in the log file if the Date field was "02/11/11 18:50:41 820" you would enter it here as: "02/11/2011 6:50:41 PM" Tip: Do not include the double quotes. 7) Enter a Duration of 5 minutes: 0 00:05 8) Click OK.
4.
9) You may wish to zoom out by clicking the Zoom Out button. h. Also drag and drop the ActiveConnectionsCurrentCount (int) attribute onto the same chart to create a second graph in a different color. Work with charts and graphs in real time. a. In the Monitoring Dashboard, click the View List tab and create another custom view. Name it MedRec Sessions. Select the new, empty view. b. Click the Metric Browser tab. In the Servers drop-down list, select MedRecSvr1, make sure Collected Metrics Only is NOT checked, and click the Go button. c. In the Types list of available MBeans on this server, locate and select WebAppComponent. d. Under Instances, select MedRecSvr1_/medrec, medrec. e. Under Metrics, drag and drop the OpenSessionsCurrentCount (int) attribute onto the right panel. f. Click the menu down-arrow in the chart title bar and select Properties. For Chart Title enter Sessions Chart and click OK. g.
Retest the MedRec application by using runclients.sh. While the script is still running return to the Monitoring Dashboard.
h.
5.
Click the green Start button and collect data for a few minutes. Re-run the clients script if you wish. Tip: Mouse over a specific data point to view its value and timestamp. Deactivate metric collection. a. Stop All views, if any are still running. b. Close the Monitoring Dashboard. c. Return to the administration console and disable your diagnostic module’s metric collector once again.
Solution Instructions No solution exists for this practice.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4 Chapter 4 - Page 11
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
3) Make note of the information in the Date field of the first CurrCapacity Attribute. 4) Return to the Monitoring Dashboard.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 4
Chapter 4 - Page 12
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 5
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5
Chapter 5 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 5
Practices Overview In these practices you will set up application-level instrumentation and dye masks to filter diagnostic events.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 5
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 2
Duration: 35 minutes Skills Learned At the end of this practice, you should be able to: • Use the console to generate and edit a deployment plan • Use the console to configure application instrumentation • Configure built-in diagnostic monitors • Define pointcuts for custom monitors • Inspect events found in diagnostic archives Overview A few patients have reported performance problems logging into the MedRec application. Before contacting the development team, however, you would like to gain more knowledge about how the MedRec login process works behind the scenes and also how it performs. The WLDF Instrumentation component allows you record diagnostic data when certain events occur within WLS or your application code. For example, you can automatically record the method arguments or stack trace whenever a specific Java class is executed. While some built-in instrumentation monitors can be configured at the server level, most built-in monitors and all custom monitors must be defined by using application deployment descriptors. In this practice, you create a basic diagnostic module and include it within the MedRec application. You then utilize a deployment plan to dynamically add diagnostic monitors to this module and to enable/disable them as needed. This practice’s diagnostic configuration is shown in the following diagram:
Instructions 1. Enable instrumentation on the server. a. If your domain does not include a diagnostic module, complete the Solution Instructions for the “Harvesting Diagnostic Metrics” practice. b. Launch the console and Lock it. c. Navigate to your diagnostics module, MedRecDiagnostics. d. Click the Configuration > Instrumentation tab. e. Select the Enabled check box. f. Save and Activate your changes. 2. Add a WLDF descriptor file to an application. a. Inspect the contents of the file /resources/METAINF/weblogic-diagnostics.xml. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 5-1: Configuring and Monitoring Diagnostic Events
Launch a Lab Framework prompt and navigate to /resources. Execute the following: jar uf /applications/medrec.ear META-INF
d.
3.
Using jar or the Linux Archive Manager, inspect medrec.ear and confirm that the file was added. e. Return to the console and Lock it. f. From the Domain Structure panel, click Deployments. g. Update the medrec application and Activate your changes. Generate a WLDF deployment plan by using the console. a. From the same Deployments table, select medrec. b. Click the Configuration > Instrumentation tab. c. Lock the console and click the Add Monitor From Library button. d. Move the EJB_Before_SessionEjbBusinessMethods option from the Available column to the Chosen column. Click OK. Because a deployment plan does not yet exist for this application, the Save Deployment Plan Assistant is launched.
e. Set the Path to /applications/medrec-plan.xml. Click OK. 4.
5.
Configure a standard EJB monitor. a. Return to the Configuration > Instrumentation tab and select your new monitor. b. Under Actions, move the StackDumpAction option from the Available column to the Chosen column. c. Click Save. Activate your changes. d. Return to the medrec application’s main configuration page and inspect the Overview tab. Confirm that a Deployment Plan is now assigned to the application. e. Inspect the generated plan file. The plan should dynamically add a new wldfinstrumentation-monitor element to the weblogic-diagnostics.xml descriptor, whose child elements resemble the following: Element
Value
name
EJB_Before_SessionEjbBusinessMethods
action
StackDumpAction
Tip: The console also generates a folder /applications/plan, which you can delete. Test the application and inspect the generated EJB events. a. From a Lab Framework prompt, execute /client/runclient.sh. Why? This script logs a user into the patient application and then views his/her history. b. Return to the admin console. From the Domain Structure panel, select Diagnostics > Log Files. c. Locate and select the EventsDataArchive log file for the MedRecSvr1 server. Click View. d. Click the Customize this table link. Under Column Display, remove from Chosen the columns Context ID, User ID, and Payload. Tip: The Payload column includes the stack trace at the time of the event. Later practices focus on analyzing stack traces. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
b. c.
6.
7.
Click Apply. Browse the generated events. Locate events for the com.bea.medrec.repository.impl.PatientServiceImpl class. Which method do you think corresponds to a user logging in? Now we create a custom monitor that just focuses on this code. Define a custom monitor. a. Lock the admin console and once again edit the medrec application. Go to the Configuration > Instrumentation tab. b. Edit the EJB_Before_SessionEjbBusinessMethods monitor. Clear the Enabled check box and click Save. c. Use the browser’s back button or the console’s breadcrumb links to return back to the main Configuration > Instrumentation tab. d. Click the Add Custom Monitor button. e. Enter the following values: Field
Value
Monitor Name
DebugPatientService
Location Type
Around
Pointcut
execution(* *impl.PatientServiceImpl auth*(...))
Click OK. Tip: This pointcut expression includes all calls to methods that start with the name auth in classes that end with the name impl.PatientServiceImpl. f. Select your new monitor. g. Under Actions, move the TraceElapsedTimeAction option from the Available column to the Chosen column. Click Save. h. Update the medrec application again and Activate your changes. i. Inspect the plan file again and confirm that it now defines two monitors. Analyze the application. a. Execute runclient.sh again. b. Repeat the previous steps to view the latest data in EventsDataArchive. c. Add the Payload column back to the list of displayed attributes. d. Locate the events from the custom DebugPatientService monitor. Each TraceElapsedTime action generates two events, one before the method execution and one after. The latter event includes the total time (in nanoseconds) to perform the authentication.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
e. f. g.
1. 2. 3. 4.
If the /domains/MedRecDomain location does not yet exist, follow the Solution Instructions for the “Developing a Custom Monitoring Script” practice. Launch the Lab Framework command shell /bin/prompt.sh file. Change the current directory to . Execute the following: ant setup_solution The Lab Framework: − −
Makes a backup copy of your current work Runs the solution for the “Harvesting Diagnostic Metrics” practice.
− Enables instrumentation on the WLDF module − Deploys an updated MedRec application along with a deployment plan Note that all application-scoped WLDF components will be disabled by default.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 6
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
Duration: 40 minutes Skills Learned At the end of this practice, you should be able to: • Enable server debug messages • Use context IDs to associate log entries with a request • Configure a Dye Injection monitor • Filter recorded events by using dye masks Overview When instrumentation is enabled on a diagnostic module, all WLDF events and log entries associated with a client request will be tagged with a unique context ID. These IDs allow administrators to trace all the diagnostic data for a specific request. In some troubleshooting cases, you may wish to go a step further and limit the conditions under which WLDF events are recorded. A diagnostic context ID supports various flags or “dyes” that allow you to filter the types of requests that trigger instrumentation data. In this practice, you use the console to locate context IDs and try to map them to specific MedRec application functionality. You also use this opportunity to experiment with the dye injection monitor and dye masks. The lab infrastructure is shown in the following diagram:
Instructions 1. Generate debug messages in the server log. a. If your domain does not include a diagnostic module, complete the Solution Instructions for the “Configuring and Monitoring Diagnostic Events” practice. b. Launch the console and Lock it. c. Locate and edit MedRecSvr1. d. Click the Debug tab. e. Locate the weblogic > servlet > internal debug scope.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 7
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 5-2: Tracing a Client Request
3.
Tip: To avoid scrolling, under Customize this table you may want to reorder the columns so that Context ID and Class are displayed adjacently. h. Copy the last set of non-zero digits. In the example above, that would be “1e58c”. Correlate WLDF and server log messages. a. Use the console to view the ServerLog for MedRecSvr1. b. Once again, add the Context ID column to the chosen list of columns. Also increase the Number of Rows Displayed Per Page to 1000. c. Browse the recent log entries. Notice that some entries have been assigned context IDs while others have not. The latter are internal messages not associated with a client request. d. Click Customize this table again. e. For WLDF Query Expression, enter the following: CONTEXTID LIKE '%nnnnn' For nnnnn, paste the context ID fragment that you copied from the WLDF event. Tip: If you do not see any entries, click Customize this table again and change Time Interval to Last 15 minute(s) (or longer). Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 8
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
2.
f. Select the check box for the internal scope and click the Enable button. g. Similarly, enable the weblogic > ejb > invoke debug scope. h. Activate your changes. View context IDs for WLDF records. a. Launch a second browser window and access the MedRec application: http://localhost:7021/medrec/index.action b. Under the Patient section, click the Login link. c. Log in as the user
[email protected]/weblogic. Leave this browser session open. d. Return to the console and view EventsDataArchive for MedRecSvr1. e. Click Customize this table. f. Add the Context ID column to the chosen list, if not already chosen. Click Apply. g. Locate the Context ID for the latest event generated by your custom DebugPatientService monitor. For example:
Click Apply. Browse all the server debug messages associated with this login request. The Subsystem column should indicate “Http,” “HttpSessions,” and “EjbInvoke.”
h.
Using these log messages, try to answer the following question: Upon logging into the MedRec application, what is the name of the JSP file that executes? Return to the MedRec application. Click the Successfully logged in! Click here to continue link. Then click one of the listed patient Records.
i.
j. k.
4.
Note the URL in the browser and the current time. Access the server log again and view the latest entries. You must remove your custom log filter criteria. l. Search the Web page contents for the text “viewRecordSummary.” Try to determine the context ID for the latest MedRec client request. m. Once again, use the WLDF Query Expression field to show only those log messages for this context ID. Browse the contents of these messages. n. Edit your MedRecSvr1 configuration again and Disable all debug attributes. Tip: You can use the check box in the table header to select all other check boxes at once. Configure the dye injection monitor to flag an IP address. a. Obtain the IP address of your Linux VM instance. From a Linux terminal, execute: /sbin/ifconfig. Tip: If your VM is bound to multiple IP addresses, you can use the previously generated debug messages to determine the actual IP being used. It is probably the first one listed. If you use the wrong one you will have to repeat these instructions and try the other ones. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 9
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
f. g.
5.
6.
Return to the console and Lock it. Locate and edit your system diagnostic module, MedRecDiagnostics. Click the Configuration > Instrumentation tab. Click the Add/Remove button. Move the DyeInjection monitor from the Available column to the Chosen column. Click OK. g. Edit the new monitor. h. Under Properties, enter the following: ADDR1=your_IP_address i. Click Save. Activate your changes. Configure dye filtering for a monitor. a. Lock the console again. b. Edit the medrec application. c. Click the Configuration > Instrumentation tab. d. Click Add Monitor from Library. e. Add the monitor Servlet_Before_Session and click OK. f. Edit the new monitor. Enter or select the following values: Field
Value
Enabled
Actions (Chosen)
TraceAction
EnableDyeFiltering
Dye Mask (Chosen)
ADDR1
Click Save. g. Update the medrec application. Activate your changes. Test event filtering. a. Retest the MedRec application as done previously. b. Return to the console and view EventsDataArchive for MedRecSvr1. c. Using the Monitor column, confirm that events were recorded from the Servlet_Before_Session monitor. Tip: If the events are not shown, try setting the dye flag to another available IP address. d. Edit the MedRecDiagnostics module and select the DyeInjection monitor. e. Edit this monitor’s Properties. Change the ADDR1 flag to use some other arbitrary IP address. Tip: Alternatively, use another student’s IP address. f. Update the medrec application. Activate your changes. g. Test the application a final time. h. In the admin console, view the latest event data. Using the records’ timestamps, verify that there are no new events from the Servlet_Before_Session monitor. i. When you are finished with the practice, edit MedRecDiagnostics and disable the instrumentation component. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 10
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
b. c. d. e. f.
1. 2. 3. 4.
If the /domains/MedRecDomain location does not yet exist, follow the Solution Instructions for the “Developing a Custom Monitoring Script” practice. Launch the Lab Framework command shell by executing the /bin/prompt.sh file. Change the current directory to . Execute the following: ant setup_solution The Lab Framework: − Makes a backup copy of your current work −
Runs the solution for the “Configuring and Monitoring Diagnostic Events” practice
− −
Disables debug flags, if enabled Adds a new dye injection monitor to the WLDF module
− Deploys the MedRec application by using an updated deployment plan Note that all WLDF components will be disabled by default. Note that the dye injection monitor will be configured with a dummy IP address.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5 Chapter 5 - Page 11
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 5
Chapter 5 - Page 12
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 6
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6
Chapter 6 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 6
Practices Overview In these practices you will troubleshoot the Sun JVM by using a heap profile. You will also troubleshoot the JRockit JVM by using JRockit Mission Control.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 6
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 2
Duration: 45 minutes Skills Learned At the end of this practice, you should be able to: • Start a server running in the Sun JVM • Perform and analyze a server thread dump • Configure JVM heap settings • Generate a heap profile by using the Sun JVM • Analyze a heap profile Overview Two of the most common troubleshooting scenarios in server software are server hangs (or very high response times) and memory leaks. Servers can hang for several reasons, including deadlocks and full CPU utilization. Memory leaks are less common in Java than other languages, but can occur when applications deliberately or accidentally create too much garbage (objects), or make objects ineligible for garbage collection by the JVM. For example, developers may place too many objects in the HTTP session scope, which by default are not eligible for garbage collection for at least 1 hour. WebLogic Server and the underlying JVM provide several tools to help troubleshoot these problems. In this practice, you investigate an application that seems to be hung, and use JVM thread dumps to help determine potential causes. Similarly, you investigate a possible memory leak with the aid of a JVM-generated heap profile. The lab infrastructure is depicted in the following diagram:
Instructions 1. Set up the practice. a. Locate a Lab Framework prompt or start a new one. Change directories to . b. Execute the following: ant setup_exercise The Lab Framework deploys a new version of the MedRec application.
c.
Kill MedRecSvr1.
d.
Restart the server by using the following commands: export JAVA_VENDOR=Sun ./startServer1.sh
e.
Use the initial server output to confirm that the Sun JVM is being used: Java(TM) SE Runtime Environment (build ...) Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 6-1: Troubleshooting a Running JVM
2.
f.
From another Linux terminal enter the following: ps –ef | grep MedRecSvr1
g.
Make a note of the process ID for MedRecSvr1. For example: oracle 31578 31539 ... java -server -Xms256m -Xmx512m XX:MaxPermSize=128m -Dweblogic.Name=MedRecSvr1 ...
Perform a thread dump under load. a. From the Lab Framework prompt, navigate to /client. b.
c.
Execute the runclients.sh script. You will notice that the test client and, therefore, the application as well, seem to be taking an unusually long time to respond. While the test client continues to run, signal the MedRecSvr1 process to perform a thread dump: kill -3
d.
Locate the output for MedRecSvr1. Confirm the presence of a thread dump. For example: Full thread dump Java HotSpot(TM) Server VM ... Heap ...
e.
Locate one or more threads that are currently running the MedRec application. For example: "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" ... ... at com.bea.medrec...
f.
3.
From these thread details, can you determine which MedRec Java class or method may be responsible for the slow application? Tip: For those with some Java knowledge, the culprit code can also be found at /resources. g. Allow the test client to finish, if still running. Tip: If you get tired of waiting, press Ctrl + C in the window where it is running. Trigger an "Out of Memory" error.
a. Execute the /client/runadminclients.sh script.
b.
c.
This client tests a different portion of the MedRec application. Specifically, the part of the application that allows administrative users to manage patient accounts. Locate the output for MedRecSvr1 and verify that an OutOfMemoryError has been generated similar to the following: ... java.lang.OutOfMemoryError: Java heap space ... Use the stack trace to identify some possible culprits for the high memory usage in the MedRec application (com.bea.medrec.*). For example: Caused By: ... Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Java HotSpot(TM) Server VM (build ...)
4.
Increase the server heap.
a. Kill MedRecSvr1. b.
c.
Restart the server by using the following additional environment variable: export USER_MEM_ARGS="-Xms512m –Xmx768m –XX:MaxPermSize=128m" Tip: If you use a new terminal, you must set the JAVA_VENDOR variable again. Inspect the initial server output to confirm the new server heap settings. For example: JAVA Memory arguments: -Xms512m –Xms768m ... Tip: You can also view server JVM settings by using the administration console.
d. Execute runadminclients.sh again. 5.
6.
e. Verify that this time no OutOfMemoryError messages were generated. Enable the Sun JVM memory profiler. a. Kill MedRecSvr1 again. b. Restart the server by using the following additional environment variable: export JAVA_OPTIONS=-agentlib:hprof=heap=sites,interval=100 Important: Do not miss the dash (“-”) before agentlib. Tip: The server requires a few minutes longer to boot as the agent initializes and begins collecting data. c. After the server is up, once again determine its current process ID, for later use. Generate a heap profile.
a. Execute runadminclients.sh again. b. c.
After the test finishes, issue the kill -3 command to the MedRecSvr1 process. In addition to a thread dump, the JVM begins to create a heap profile: ... Dumping allocation sites ...
d. 7.
The profile requires some time to capture, perhaps as long as 5 minutes or more. When it is finished, the JVM outputs “Done.” Analyze profiler output.
a. View the MedRecDomain/java.hprof.txt file. b.
At the end of the file, locate the SITES BEGIN section. Find the top ranked class names in terms of heap consumption. For example: SITES BEGIN... percent ... stack class rank self accum trace name 1 92.32 92.32% 408543 java.lang.Byte[] 2 ...
c.
Locate the stack trace number for this class name. Search the file for this number to locate the complete trace entry. For example: TRACE 408543: Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
at com.bea.medrec.web.controller. ViewingNewlyRegisteredPatientsController ...
d.
Use the stack trace entry to determine which part of the application is responsible for creating these large objects. Tip: Once again, the actual culprit code can also be found at /resources, for reference.
Bonus Instructions Launch Sun’s JConsole or JVisualVM tools (found at /jdk.../bin). Restart the server and run the test again while monitoring various JVM metrics, such as heap size and garbage collection times. To connect to a JVM, you must know its process ID. Solution Instructions No solution exists for this practice. To do this practice the MedRecDomain must exist. If the /domains/MedRecDomain location does not yet exist, follow the Solution Instructions for the “Developing a Custom Monitoring Script” practice.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 6
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
...
Duration: 35 minutes Skills Learned At the end of this practice, you should be able to: • Start a server running in the JRockit JVM • Navigate the features of JRockit Mission Control • Monitor CPU utilization by using the Management Console • Profile execution time by using the JRockit Flight Recorder • Profile heap usage with the Memory Leak Detector Overview In addition to providing a durable, high performance JVM, Oracle JRockit includes many troubleshooting and diagnostic tools, including JRockit Mission Control. Using Mission Control, you can monitor the health and performance of a JVM in real time by using customizable graphs and gauges. You can also record performance and memory profiles for later analysis. Finally, Mission Control includes software to help identify and troubleshoot possible memory leaks on the server. In this practice, you utilize all of these capabilities to investigate several application problems. The general Mission Control architecture for this practice is shown in the following diagram:
Instructions 1. Set up the practice. a. Locate a Lab Framework prompt or start a new one. Change directories to . b. Execute the following: ant setup_exercise The Lab Framework deploys a new version of the MedRec application. 2. Start JRockit Mission Control. a. Kill MedRecSvr1 by using the kill -9 command (to avoid the generation of any additional heap profiles). b. Restart the server by using the following commands: export USER_MEM_ARGS="-Xms512m –Xmx768m" export JAVA_VENDOR=Oracle export JAVA_OPTIONS=-Xmanagement ./startServer1.sh c.
In the server’s initial output, confirm that the following message is written: [INFO][mgmnt ] Local JMX connector started Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 7
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 6-2: Troubleshooting Applications on JRockit
From a Linux terminal enter the following: ps –ef | grep MedRecSvr1 Make a note of the process ID for MedRecSvr1.
e. Execute the file /jrockit.../bin/jrmc.
f.
3.
Tip: The exact name of the JRockit root folder depends on the version and release number that is installed in your environment. In the JVM Browser panel or “view,” locate the Java process for MedRecSvr1. If necessary, cross reference the process ID located in parentheses.
Monitor CPU utilization by using the Management Console. a. Right-click the server in the JVM Browser view and select Start Console. A console view appears to the right. Resize the views as necessary so that the CPU gauge is displayed:
b.
From the Lab Framework prompt, navigate to /client.
c. d.
Execute the runbasicclients.sh script. While the test client is running, return to Mission Control and monitor the CPU and Heap utilization. This simple login test should not yield any extraordinary results, and should finish within a minute. Now execute the runclients.sh script. Using Mission Control, confirm that during this new test the CPU gauge approaches or becomes 100%. You can also view the historical CPU utilization by using the Processor section found below the gauges:
e. f. g.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 8
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
d.
4.
The test requires a few minutes to finish. When finished, close the console view for MedRecSvr1 in JRockit Mission Control (JRMC), but do not close JRMC. Profile execution time by using the JRockit Flight Recorder (JFR). a. Right-click the server in the JVM Browser view and select Start Flight Recording. b. Notice the location of the flight recording file shown in the Filename field. Ensure Time fixed recording is selected. For Recording Time, enter 3 min. Click OK. c. While the Flight Recorder executes, start the runclients.sh script a second time. d. When the Flight Recorder finishes after 3 minutes, use the File menu to open the file. File > Open File. Select the .jfr file (under /home/oracle) and click OK. e. General > Overview is displayed by default. Inspect the Live Set, Heap Fragmentation, and GC Pause Time gauges. f. Click the Code button on the left side of the view. g. Inspect the Hot Packages and Hot Classes sections under the Overview tab. The data should indicate that the MedRec application is likely the reason for the high CPU results. For example:
h. i.
Click the Hot Methods tab at the bottom of the view. The supplied table lists the code that was executed the most frequently. For example:
j.
5.
Select the top ranked method. Within the Successors section, determine which code this method called next. k. Close the Flight Recording file view, but do not close JRMC. Tip: For those with some Java knowledge, you can also view the culprit code at /resources. Profile heap usage with the Memory Leak Detector. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 9
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
h.
c. d.
Execute the runadminclients.sh script. While the test client executes, return to Mission Control. Launch the Console view once again for MedRecSvr1. Monitor server heap usage during the test:
Tip: For additional heap-related data, you can also use the Runtime > Memory tab. After the test client finishes, close the console view. Right-click the server in the JVM Browser view and select Start Memleak. The memory leak view is displayed, which includes a table showing possible culprit Java types or classes. For example:
e.
By default the table is sorted by Growth Rate. Instead click the % of Heap column to change the sort criteria. f. Select the top ranked type. Right-click and select List Largest Arrays. The instances are displayed on the left. g. Browse through the array instances in memory and their sizes. Tip: For additional details about an object instance in memory, you can right-click it and select Inspect Instance. The Inspector view appears. Tip: Once again, you can view the culprit code at /resources. h. When finished with the practice, do the following: 1) Stop Mission Control. 2) Kill MedRecSvr1 and close its Linux terminal window. 3) Restart the server by using a new Linux terminal (to use the default environment variables). Solution Instructions No solution exists for this practice. To do this practice the MedRecDomain must exist. If the /domains/MedRecDomain location does not yet exist, follow the Solution Instructions for the “Developing a Custom Monitoring Script” practice.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 6 Chapter 6 - Page 10
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
a. b.
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 7
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 7
Chapter 7 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 7
Practices Overview In this practice you look into the Java classpath for possible errors.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 7
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 7 Chapter 7 - Page 2
Duration: 30 minutes Skills Learned At the end of this practice, you should be able to: • Locate and analyze a classpath error • Browse application and server classes • Debug server start scripts Overview Java code is made available to a JVM process, such as WebLogic Server, through class loaders. Class loaders are typically associated with some set of locations on the file system. The main JVM system class loader, for example, is configured by using the classpath argument. Applications also have their own class loaders. While the system classpath can be set to any arbitrary file system paths and archives, WLS and the Java EE specification dictate from where application-specific code can be loaded. In this practice, the MedRec system has been upgraded to utilize a new third-party library. While the application functions properly in the development environment, the production environment has encountered a class loading error. You investigate the contents of the application along with the system classpath. Instructions 1. Set up the practice. a. Locate a Lab Framework prompt or start a new one. Change directories to . b. Execute the following: ant setup_exercise The Lab Framework: − Deploys a new version of the MedRec application − Updates the MedRec domain files c. Kill and restart the server MedRecSvr1. Be sure to continue using the provided startServer1.sh script. 2. Trigger a Class Not Found error. a. Launch a Web browser and access the MedRec application: http://localhost:7021/medrec/index.action b. c. d. e. f.
Log in as the patient Fred with:
[email protected]/weblogic. Click the Profile button. Scroll down and click the Save button. Confirm that the application fails and that you receive a generic HTTP 500 error. Locate the Java error message in the log (or server output) for MedRecSvr1. It should resemble the following: Threads tab. Inspect the contents of the Self-Tuning Thread Pool Threads table. Make a note of the names of threads whose Hogger flag is set to true. Hogger threads are simply suspicious threads that have not yet formally been flagged as stuck. Tip: You can scroll to the right to view this column, if necessary. Alternatively, customize the table. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 8 Chapter 8 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
c.
Analyze thread deadlocks by using the console. a. Click the Dump Thread Stacks button. Tip: If the test client has finished, run it again and then take a thread dump. b. Locate the hogger threads that you identified earlier. Notice that several of these threads meet the following criteria: −
Standard execute threads (not internal)
− −
Active state Running MedRec application code
− Blocked (waiting for a lock) Here is an example: [ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" ... Blocked trying to get lock: com/bea/medrec/web/controller/ ViewingRecordSummaryController$SynchronizedRecord@564290 at ... at com/bea/medrec/web/controller/ ViewingRecordSummaryController.viewRecordSummary ... c.
d.
6.
Tip: Perform a search for the text "blocked". How many threads meet these criteria? Notice that all of them appear to be waiting on the same lock. This indicates a potential deadlock. Try to find a single thread running the MedRec application but not waiting for a lock. For example: [ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" ... at ... at com/bea/medrec/web/controller. ViewingRecordSummaryController.viewRecordSummary (ViewingRecordSummaryController.java:52) ...
This code is likely the cause of the deadlock. e. Scroll down to the end of the thread dump and locate the Blocked lock chains section. Use this information to confirm that these threads are all waiting on the same lock. Tip: For those with some Java knowledge, you can view the culprit code at /resources. Create a custom work manager. While this deadlock is being investigated, we use this opportunity to limit the number of threads this application can consume. This way we can minimize the impact upon other applications on the same server. a. Lock the console. b. In the Domain Structure panel, select Environment > Work Managers. Click New. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 8 Chapter 8 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
5.
Select the Work Manager option and click Next. Name the work manager LowPriority and click Next. Target the work manager to MedRecSvr1 and click Finish. Edit the new work manager. Locate the Maximum Threads Constraint field and click New:
h.
Enter the following values: Field
Value
Name
Max3
Count
3
Click Next. i. Target the constraint to MedRecSvr1 and click Finish. j. Click Save and Activate your changes. Assign the work manager to the application.
a. Inspect the contents of the /resources/medrec-plan.xml file. b.
8.
Copy this file to /applications. Overwrite any existing version of the file. c. Update the medrec application by using the console. d. Restart MedRecSvr1 to destroy any stuck threads. Validate the work manager constraints. a. Start the test client once again. b. Use the console or the kill command to capture a thread dump. c. Analyze the thread dump and confirm that no more than three threads are processing MedRec requests at the same time. d. When finished with the practice, return to the server’s Configuration > Overload tab. Modify the following fields: Field
Value
Failure Action
Ignore, take no action
Panic Action
Ignore, take no action
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 8 Chapter 8 - Page 6
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
7.
c. d. e. f. g.
1. 2. 3. 4.
If the /domains/MedRecDomain location does not yet exist, follow the Solution Instructions for the “Developing a Custom Monitoring Script” practice. Launch the Lab Framework command shell by executing the /bin/prompt.sh file. Change the current directory to . Execute the following: ant setup_solution The Lab Framework: − Makes a backup copy of your current work − Disables overload protection for MedRecSvr1, if enabled − Adds a new work manager to MedRecSvr1 − Deploys the MedRec application by using an updated deployment plan
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 8 Chapter 8 - Page 7
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 8
Chapter 8 - Page 8
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 9
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 9
Chapter 9 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 9
Practices Overview In this practice you will troubleshoot a JDBC datasource.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 9
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 9 Chapter 9 - Page 2
Duration: 40 minutes Skills Learned At the end of this practice, you should be able to: • Correct a JDBC configuration error • Troubleshoot a connection leak scenario • Configure JDBC diagnostic monitors • Configure data source connection timeouts Overview In this practice, you monitor and troubleshoot a data source whose connections do not appear to function as expected. This troubleshooting scenario involves both administrative and application issues. Specifically, this practice focuses on WebLogic’s connection testing and connection leak detection features. You use a combination of the server logs, WLST, and the diagnostic framework. The system under investigation is shown in the following diagram:
Instructions 1. Set up the practice. a. Locate a Lab Framework prompt or start a new one. Change directories to . b. Execute the following: ant setup_exercise The Lab Framework: − Deploys a diagnostic module to MedRecSvr1 if not already present − Introduces a configuration problem in the MedRec data source − Deploys a new version of the MedRec application c. Kill and restart the server MedRecSvr1. 2. Correct a JDBC configuration error. a. After the server starts up, inspect the server output for several minutes. Verify that an error message is logged periodically: Can you guess the configuration mistake? Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 9 Chapter 9 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 9-1: Investigating JDBC Problems
f. g. h.
3.
Launch the console and Lock it. From the Domain Structure panel, click Services > JDBC > Data Sources. Edit MedRecGlobalDataSourceXA. Click the Configuration > Connection Pool tab. Tip: Note the current settings for Initial Capacity and Maximum Capacity. Click Advanced. For Test Table Name, change the current value from PATEINTS to PATIENTS. Edit the following additional fields: Field
Value
Test Connections on Reserve
Test Frequency
0
Click Save. i. Activate your changes. j. Restart the server. Confirm that the error message no longer appears. Observe a connection leak scenario. a. From a Lab Framework prompt, execute /client/runclients.sh to test the MedRec application under load. b. Verify that several exceptions are generated on the server. Locate messages similar to: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool MedRecGlobalDataSourceXA to allocate to applications, please increase the size of the pool and retry. c. d. e. f. g.
h.
Return to the administration console and select MedRecGlobalDataSourceXA again. Click the Monitoring > Statistics tab. Click Customize this table. Remove the JDBC Driver column. Add the following columns: Current Capacity Active Connections Current Count Num Available Click Apply. Verify that not only is the current number of connections in the pool equal to the maximum capacity, but the current number of active connections is the same as well. For example:
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 9 Chapter 9 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
b. c. d. e.
4.
Tip: This message implies that connections were reserved by applications but never released back to the connection pool. k. Restart the server. Configure JDBC diagnostic monitors. a. Return to the console and Lock it. b. Edit the diagnostic module MedRecDiagnostics. c. Click the Configuration > Instrumentation tab. d. If instrumentation is not already enabled, select Enable instrumentation and click Save. e. Add a new diagnostic monitor and edit it by using the following criteria:
f.
5.
Shut down MedRecSvr1 by using Ctrl + C. In addition to the standard server shut down log messages, notice the following warning: .
Field
Value
Monitor
JDBC_After_Reserve_Connection_Intern al
Enabled
Actions (Chosen)
StackDumpAction
Add another monitor, JDBC_After_Release_Connection_Internal. Configure the same action:
g. Remove any other Diagnostic Monitors. h. Activate your changes. i. Execute runclients.sh again. Investigate connection leaks by using WLDF. a. After the clients finish, return to the console and view the EventsDataArchive log file for MedRecSvr1. b. Click Customize this table. c. Display only these columns: Date Method Payload d. Increase the Number of Rows Displayed to 100 and click Apply.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 9 Chapter 9 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
i. j.
f.
Confirm that several events were generated. The Method column indicates "reserve" or "release" while the Payload column displays the stack trace for the event. Customize the table again. For WLDF Query Expression, enter the following: METHODNAME = 'reserve'
g.
6.
Use the browser’s search feature to locate the “ViewingRecordSummaryController.viewRecordSummary” text in the event stack traces. h. Count the number of “reserve” events originating from this code. i. Customize the table a third time so that only "release" events are shown. j. Repeat the previous steps to count the number of “release” events originating from this code. Compare the results. Notice that the viewRecordSummary method seems to reserve more connections that it releases. k. Edit MedRecDiagnostics again and disable all instrumentation. Configure data source connection timeouts. a. Edit MedRecGlobalDataSourceXA once again. b. Lock the console. Click the Configuration > Connection Pool tab. Click Advanced. c. Update the following fields:
d.
Field
Value
Maximum Capacity
20
Inactive Connection Timeout
30
Tip: Inactive Connection Timeout is in the Advanced section. Maximum Capacity is not. Save. Activate your changes. Restart the server.
e. Inspect the contents of the /resources/wlst/monitorJDBC.py file. f. g. h.
i.
j. k.
Launch another Lab Framework prompt and execute the monitorJDBC.py WLST script. Note the current capacity of the database as well as the number of connections in various states (active, available, leaked and so on). Leave the script running. Execute the runclients.sh script a final time and continue monitoring the WLST output. Immediately after the client script finishes, confirm that the number of active connections does not decrease to 0. Verify that after an additional 30 seconds, the number of active connections is reset to 0 and the leaked connection count increases. The previous “No resources currently available in pool” errors should no longer be generated as well. Kill the WLST script. Locate the following warning messages in the server log: Active Destinations tab. f. This practice concentrates on the JMS queue named PatientNotificationQueue. Confirm that the Messages Received count is 0. g. Locate and inspect the MedRec-jms JMS module. Select PatientNotificationQueue. h. Click the Monitoring tab. Confirm that the Consumers Current column is 0. Tip: The Messages Total column is the equivalent of Messages Received. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 10 Chapter 10 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 10-1: Investigating JMS Problems
Observe a JMS producer error. a. From your Lab Framework prompt, navigate to /resources/client. Tip: The Lab Framework prompt has the correct environment to run the supplied JMS clients. Do not use a standard Linux terminal. b. Execute this command: export CLASSPATH=.:$CLASSPATH Why? The client script needs the current folder to be in the classpath. c. Execute the runconsumer1.sh script. d. Monitor PatientNotificationQueue again and verify that the Consumers Current column is now 1. e. Launch a second Lab Framework prompt, add the current folder to the classpath as before, and execute runproducers1.sh. This script simulates multiple systems that produce messages simultaneously. f. Verify that the consumer client is receiving messages via the JMS queue: Message Received: (TextMessage[ID:, Patient=73, ...]) g.
3.
Tip: The consumer client displays the ID of each message along with its payload text. Verify that the producer application generated one or more errors while sending messages. For example: Sender 'PatientNotifier3' encountered a problem. weblogic.jms.common.ResourceAllocationException: weblogic.messaging.kernel.QuotaException: Quota MedRecJMSServer.Quota.1259940548078 exceeded: ...
Can you guess what the cause of this error is? Configure JMS quotas. a. Return to the console and Lock it. b. First, verify the quota resources for the PatientNotificationQueue destination in the JMS module: 1) Click the Configuration > Thresholds and Quotas tab. 2) Confirm that this destination is not currently assigned a quota resource:
c.
4.
Now edit MedRecJMSServer. 1) Click the Configuration > Thresholds and Quotas tab. 2) Increase the Messages Maximum field to 100. 3) Save and Activate your changes. Monitor JMS traffic by using WLST.
a. Inspect the contents of the b. c.
/resources/wlst/monitorJMS1.py file. Launch a third Lab Framework prompt. Execute the monitorJMS1.py WLST script. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 10 Chapter 10 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
2.
5.
Note the current number of messages on MedRecJMSServer (0). Leave the script running. e. Execute runproducers1.sh again. While the producer clients are running, monitor the WLST output. Wait some time after the producer clients finish and both the number of current and pending messages should decrease to 0. f. Verify that the producer now generates no errors and that all messages are received by the consumer. g. Kill the WLST script. Observe a scenario with missing messages. a. Confirm that the consumer client is still running. Tip: You may wish to stop and restart the consumer client between tests to help distinguish the messages for each test.
b. Execute the /resources/client/runproducers2.sh script. c.
6.
Compare the output from the producer script to the output from the consumer script. Notice that the consumer is not receiving the same number of messages that were sent (15). d. Return to the console and monitor PatientNotificationQueue. Confirm that there are currently no messages on the destination. Configure delivery failure settings. a. Lock the console. b. Edit the MedRec-jms JMS module. c. Under Summary of Resources, click New. d. Select the Queue option and click Next. e. Enter the following values:
f. g. h. i.
7.
Field
Value
Name
ErrorQueue
JNDI Name
com.bea.medrec.jms.ErrorQueue
Click Next. For Subdeployments, select DeployToMedRecJMSServer. Click Finish. Edit PatientNotificationQueue. Click the Configuration > Delivery Failure tab. Update the following fields: Field
Value
Expiration Policy
Redirect
Error Destination
ErrorQueue
Click Save. Analyze message events by using JMS logs. a. Click the Configuration > Logging tab. b. Select the following check boxes: Enable Message Logging All Properties Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 10 Chapter 10 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
d.
Save and Activate your changes.
d. Run the /resources/wlst/monitorJMS2.py WLST script. e. f.
Repeat the previous test (runproducers2.sh and runconsumer1.sh). Monitor the WLST script output. Verify that some messages are failing and are being moved to the ErrorQueue destination. For example: PatientNotificationQueue TOTAL CURRENT PENDING 36 3 4
g. h. i. j. k.
l.
ErrorQueue TOTAL CURRENT PENDING 2 2 0
Kill the WLST script. Return to the console. From the Domain Structure panel, select Diagnostics > Log Files. Select and view JMSMessageLog/MedRecJMSServer. Click Customize this table. Display only the following columns: Date Message ID Event Message Browse the different message events. Notice that several Expired events exist:
m. Locate and inspect the Produced events. Notice that some messages include the JMS_BEA_DeliveryTime property while others do not.
8.
Investigate failed messages by using the console. a. Locate and monitor the ErrorQueue destination. b. In the Destinations table, select the check box and click the Show Messages button:
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 10 Chapter 10 - Page 6
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
c.
d.
9.
In the JMS Messages table, click each message ID to view its details. Notice that the body Text of all of these expired messages is similar to one another. View the previous output from the producer clients. Which specific producer seems to be responsible for these messages? For example: Sender 'PatientNotifier1' produced a message for Patient 1. ... Sender 'PatientNotifier1' produced a message for Patient 1. ... Sender 'PatientNotifier1' produced a message for Patient 1.
Now the culprit has been identified. Respond to failed messages. After some discussions with the team responsible for the culprit producer program, you learned that it is taking advantage of the Delivery Time feature of WebLogic JMS. This message header allows the server to hold messages and delay delivering them to consumers. Unfortunately, these messages seem to expire before this delivery time condition is met. a. Make sure your consumer client is still running. b. Return to the console and view the ErrorQueue messages again. c. In the JMS Messages table, click the Move > Move All button. d. Confirm that the MedRecJMSServer is selected and click Next. e. Select MedRec-jms!PatientNotificationQueue as the destination and click Finish. f. View the output from the consumer client. Confirm that the failed messages are now consumed. g. Use the console to verify that both destinations are now empty. h. Lock the console and edit the PatientNotificationQueue. i. Click the Configuration > Overrides tab. j. Lock the console and update the following fields: Field
Value
Time-to-Live Override
3600000
Delivery Mode Override
PERSISTENT
Save and Activate your changes. The time-to-live for all messages on this destination is now 1 hour. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 10 Chapter 10 - Page 7
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
c.
Test runproducers2.sh a final time and verify that there are now no failed messages and that eventually all messages are consumed. Use the console or the WLST script to monitor the destinations. l. Shut down the JMS consumer. 10. Observe a scenario with duplicate messages.
a. Run monitorJMS2.py again. b. Execute the /resources/client/runconsumer2.sh script. c. Execute the /resources/client/runproducers1.sh script. d.
View the WLST metrics and wait for the producer clients to finish. Notice that while there are no failed messages, PatientNotificationQueue indicates that there are still messages pending. For example: PatientNotificationQueue ErrorQueue TOTAL CURRENT PENDING TOTAL CURRENT PENDING 80 0 5 3 0 0
e.
The producers send a total of 30 messages. Inspect the output from the consumer client. Verify that all 30 messages were received. f. While continuing to monitor the WLST metrics, quit the consumer client. Notice that the status of the remaining messages changes from “pending” to “current.” g. Restart runconsumer2.sh. Confirm that it receives these duplicate messages. h. View the latest WLST metrics and confirm that now there are no messages in the queue. Kill the WLST script. 11. Debug JMS communication. a. Return to the console and Lock it. b. Locate and edit MedRecSvr1. c. Click the Debug tab. d. Expand the weblogic > jms > backend debug scope. e. Select the check box for the DebugJMSBackEnd attribute and click the Enable button. f. Activate your changes. g. Repeat the previous test (runconsumer2.sh and runproducers1.sh). Wait for the producers script to finish. Wait for the consumer script to stop writing to the window. h. View the ServerLog for MedRecSvr1. i. Locate debug messages similar to the following: ... ... Tip: You may need to click Customize this table to clear out the WLDF Query Expression from a previous practice for these messages to be displayed. To see only “Debug” messages, set the WLDF Query Expression to: SEVERITY = 'Debug' Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 10 Chapter 10 - Page 8
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
k.
k.
Verify that while 30 messages are pushed to the consumer, only 25 are acknowledged (in batches of 5). When finished with this practice, disable all server debug flags. Also, stop the JMS client.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
j.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 10 Chapter 10 - Page 9
1. 2. 3. 4.
If the /domains/MedRecDomain location does not yet exist, follow the Solution Instructions for the “Develop a Custom Monitoring Script” practice. Launch the Lab Framework command shell by executing the /bin/prompt.sh file. Change the current directory to . Execute the following: ant setup_solution The Lab Framework: − Makes a backup copy of your current work −
Creates a new error queue if it does not already exist
− −
Updates quota settings on a JMS server Updates override and failure settings on a JMS queue
− Disables debug flags, if enabled Note: If you want to run producer or consumer scripts in the Lab Framework prompt under the /resources/client folder, be sure to first execute the command: export CLASSPATH=.:$CLASSPATH
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 10 Chapter 10 - Page 10
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 11
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 11
Chapter 11 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 11
Practices Overview In these practices you setup and analyze SSL and investigate problems with a new authentication provider. Naming Conventions In addition to the variable names listed earlier, these practices use the following variable name to refer to a location on your file system. This value is also defined as a Linux environment variable. Variable
Path
/u01/app/opensource/OpenDS
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 11 Chapter 11 - Page 2
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 11
Duration: 40 minutes Skills Learned At the end of this practice, you should be able to: • Manage and monitor the SSL subsystem • Analyze SSL debug messages • Analyze and update keystore contents • Troubleshoot invalid and untrusted certificates Overview Recall that SSL for Java applications involves keystore files to securely house private and public keys. Identity keystores provide keys to establish one-way SSL connections with clients, while trust keystores provide keys to validate other systems’ identities. Trust keystores are used by the server to establish incoming two-way SSL connections and for initiating outbound SSL connections. Due to the confidential nature of patients’ data, the MedRec application has been updated to require SSL communication between browser clients and the server. Additionally, the application has been updated so that any outbound JMS message exchanges that involve patient data are also secured. This lab environment is depicted in the following diagram:
Instructions 1. Set up the practice. a. Locate a Lab Framework prompt or start a new one. Change directories to . b. Execute the following: ant setup_exercise The Lab Framework: − Generates new self-signed certificates for MedRecSvr1 and MedRecSvr2 − Configures SSL for MedRecSvr1 − Defines a new server MedRecSvr2 that also supports SSL − Adds a JMS server and queue to MedRecSvr2 − Deploys a new version of the MedRec application that requires SSL and also sends JMS messages to MedRecSvr2 2. Observe SSL startup errors. a. Launch the console and view the configuration for MedRecSvr1. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 11 Chapter 11 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 11-1: Investigating SSL Problems
Under Configuration > General, confirm that the server is enabled for SSL:
c.
Click the Configuration > Keystores tab. Note the file name and location of this server’s Custom Identity Keystore. Click the Configuration > SSL tab. Note the Private Key Alias in the keystore that this server is using. Launch another Web browser and access the MedRec application by using the standard server port: http://localhost:7021/medrec/index.action Notice that the server automatically redirects the browser to the SSL port, due to the settings in the application’s web.xml descriptor. Confirm that the browser is unable to access the SSL port. For example:
d. e.
f.
g.
3.
Inspect the latest server output for MedRecSvr1. Verify that several errors were reported when trying to start the SSL port, similar to:
Debug SSL startup activities. a. Lock the console. Edit MedRecSvr1. b. Click the Debug tab. c. Locate and expand the weblogic > security > ssl debug scope. d. Select the check box for the DebugSecuritySSL attribute and click the Enable button. e. Activate your changes. f. Click the Control tab. g. In the Server Status table, click the Restart SSL button. Then click Yes to confirm. h. View the ServerLog for MedRecSvr1. Locate an SSL debug message related to loading the server’s identity certificate: Messaging > JMS Servers. d. Click ProfileJMSServer. e. Click the Monitoring tab. Use the Messages Current column to confirm that a JMS message was received on MedRecSvr2. f. When finished, perform the following: 1) Shut down MedRecSvr2. 2) Delete the file server2.cer. 3) Disable SSL debug log messages.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 11 Chapter 11 - Page 8
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
e. f. g.
There is no complete solution for this practice. If you only partially completed the practice instructions, however, then you need to undo what you have done by following these steps: 1. Launch the Lab Framework command shell by executing the /bin/prompt.sh file. 2. 3.
Change the current directory to . Execute the following: ant setup_solution The Lab Framework: −
Makes a backup copy of your current work
− −
Disables SSL on all servers Disables any SSL debug flags
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 11 Chapter 11 - Page 9
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
Duration: 40 minutes Skills Learned At the end of this practice, you should be able to: • Diagnose a server that fails to start • Work with authentication control flags • Debug the WLS security realm • Analyze LDAP user and group data • Tune LDAP authentication settings Overview The MedRec WebLogic Server infrastructure has been upgraded to integrate with their corporate LDAP, which identifies end users and groups as well as administrative entities. MedRec’s goal is to rely solely on this corporate LDAP for all enterprise authentication, replacing the default WLS authentication provider that uses the embedded LDAP repository. Unfortunately, the latest security realm modifications have introduced problems into the domain. In this practice, you troubleshoot several security issues related to authentication and authorization. Because you are not part of the MedRec enterprise security team, you also use this opportunity to explore the contents of the external LDAP server. This will aid you in diagnosing any LDAP communication issues. The latest MedRec domain configuration resembles the following diagram:
Instructions 1. Set up the practice. a. Locate a Lab Framework prompt or start a new one. Change directories to . b. Execute the following: ant setup_exercise The Lab Framework: − Initializes LDAP data − Adds a new LDAP authentication provider to the security realm Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 11 Chapter 11 - Page 10
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 11-2: Investigating Security Realm Problems
f.
Navigate to /bin.
g.
Execute start-ds. Confirm that the LDAP server is running:
c. d.
... The Directory Server has started successfully 2.
Troubleshoot administrative login failure. a. Kill both servers once again. b. Try to start the admin server (MedRecAdmSvr). Confirm that the server fails with the following error: security > atn > DebugSecurityAtn attribute and click the Enable button. f. Enable the same debug attribute on MedRecAdmSvr. g. Activate your changes. Trace authentication activity. a. Launch a new browser and attempt to log into the MedRec application again. b. View the ServerLog for MedRecSvr1. Locate the security debug messages related to the authentication attempt against the LDAP provider. For example: ... ... ... ... c.
Locate the debug messages specifically related to the communication with the LDAP server. For example: Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 11 Chapter 11 - Page 12
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
a. Launch a browser and access the MedRec application: http://localhost:7021/medrec/index.action Notice that the browser automatically prompts users for their credentials, due to the settings in the application’s web.xml descriptor:
5.
It appears that the configured LDAP filter does not match any people objects in the external LDAP server. Analyze LDAP contents. a. From a Linux terminal, navigate to /bin. b. Run the following to search the LDAP server for all people (users): ./ldapsearch -h localhost -p 7878 -b dc=medrec,dc=com "(objectclass=person)" c.
Browse the returned user accounts. For example: dn:
[email protected],dc=medrec,dc=com givenName: Fred objectClass: person objectClass: organizationalPerson objectClass: inetOrgPerson objectClass: top uid:
[email protected] cn: Patient sn: Winner
Notice that there does not appear to be an “email” attribute for these users. Perform another search, but use the following LDAP filter expression: (&(objectclass=person)(
[email protected])) Confirm that no matching users are found. Tip: Remember that the LDAP expression must be enclosed in quotation marks. e. Now try the following LDAP filter instead: (&(objectclass=person)(
[email protected])) This expression should retrieve the desired results: this user’s account. Modify a provider’s authentication settings. a. Return the console and Lock it. b. From the Domain Structure panel, select Security Realms. c. Edit the security realm and click the Providers tab. d. Edit the MedRecLDAP authentication provider. e. Click the Configuration > Provider Specific tab. f. Locate the Users section. Modify the following fields: d.
6.
Field
Value
User From Name Filter
(&(uid=%u)(objectclass=person ))
User Name Attribute
uid
Click Save. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 11 Chapter 11 - Page 13
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
7.
Click the Configuration > Common tab. Return the Control Flag to its prior value, REQUISITE. Tip: This value essentially bypasses the default authentication provider. i. Save and Activate your changes. Trace successful authentication and authorization. a. Kill and restart both servers. Verify that now the weblogic administrative user is authenticated successfully. b. In the latest log entries from either server, locate the following security debug messages: ... ... Servers tab. In the Servers table, note the machine assigned to MedRecSvr1. h. Try to Start the server MedRecSvr1. This operation fails with an error stating that the node manager is not available Debug node manager client communication. a. From the Domain Structure panel, select Environment > Machines. b. Select MedRecMch1. c. Click the Configuration > Node Manager tab. Verify that the node manager address and port settings for this local machine are correct. d. Click the Monitoring > Node Manager Status tab. Notice that once again, the admin server claims that the node manager is not available. e. Locate and view the latest output from the administration server. f. Notice several error messages related to node manager communication. For example: This type of error typically means that the credentials being sent from the admin server are not those that the node manager expects. Enroll a node manager with a domain. a. From a Lab Framework prompt, navigate to /nodemanager. b. Launch WLST (interactive mode). c. Connect to the administration server and use the nmEnroll command: connect('weblogic','Welcome1','localhost:7020') nmEnroll('/home/oracle/wls11g_trouble/work/domains/MedRecDomain' )
5.
6.
d. Confirm that the command executed successfully, and then exit WLST. Observe remote server startup problems. a. Return to the console. b. Try the Monitoring > Node Manager Status tab again for MedRecMch1. The Status should now indicate Reachable. c. Repeat the previous steps to start MedRecSvr1 from the console. This time the command should succeed. d. Click the Monitoring > Health tab. Verify that the current State of MedRecSvr1 is now FAILED_NOT_RESTARTABLE. Debug server startup problems. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 12 Chapter 12 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Tip: The node manager is started with the –v option, meaning that all node manager log messages are also written to standard output.
View the latest node manager output. Confirm that the node manager logged messages similar to the following: ... java.io.IOException: Server failed to start up. See server output log for more details.
b.
Use the node manager log messages to determine the location of the standard output log file for MedRecSvr1 (it uses the .out file extension by default).
c.
Examine MedRecSvr1.out. Locate the exception that caused the startup failure: Exception in thread "Main Thread" java.lang.NoClassDefFoundError: weblogic/Server Could not find the main class: weblogic.Server. Program will exit.
d.
e.
Tip: The node manager may rotate through several output files while starting a server. You can search multiple files, if necessary. Use either the server output log or the node manager log to determine the command that the node manager ran to launch the server. Identity the classpath being used. For example: java ... -Djava.class.path=... weblogic.Server Are the locations defined in this classpath valid? Compare them to your actual file system. Use the node manager log messages to determine the location of the startup.properties file recorded for MedRecSvr1.
f.
7.
View the contents of startup.properties. Confirm that the classpath value is defined here. g. Return to the console. h. Locate and view MedRecSvr1. i. Click Configuration > Server Start. Note the settings for Class Path and Arguments. Configure node manager to use start scripts. To avoid configuration errors like those above, we update the node manager configuration to simply call the standard server start script, startWebLogic.sh.
a. Edit /nodemanager/nodemanager.properties. b.
Add the following lines: StartScriptEnabled=true StopScriptEnabled=true
c. d. e.
Save and close the file. Kill and restart the node manager. Use the console to try to start MedRecSvr1 a third time. Monitor the domain again. Confirm that MedRecSvr1 eventually starts successfully and is in the “RUNNING” state. Inspect the latest node manager output log for MedRecSvr1. From the server’s output, determine what heap settings the server was started with. For example:
f.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 12 Chapter 12 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
a.
8.
Recall that the original server start settings specified a maximum heap size of 768m. g. Use the console or WLST to stop MedRecSvr1. Develop and use a custom server start script.
a. Create and edit a new file: /domains/MedRecDomain/bin/ startMedRecServers.sh. b.
Add logic to set different heap options for different servers, by using the WLS environment variable USER_MEM_ARGS. For example: if [ ${SERVER_NAME} = "MedRecSvr1" ] ; then export USER_MEM_ARGS="-Xms512m -Xmx756m" else export USER_MEM_ARGS="-Xms512m -Xmx512m" fi
c.
At the end of this script execute the standard WLS start script with the necessary parameters: cd /domains/MedRecDomain/bin ./startManagedWebLogic.sh ${SERVER_NAME} ${ADMIN_URL}
d. e.
Save your changes. Modify the file’s permissions so that it is executable. Edit nodemanager.properties once again. Add the following line: StartScriptName=startMedRecServers.sh
f.
g. h.
i. j. k.
9.
Save your changes and restart the node manager. Tip: Notice that MedRecSvr1 is not automatically started when the node manager starts. This is due to the fact that the server was gracefully shut down and did not fail. Use the console or WLST to start MedRecSvr1 again. Use the server output to verify that it is now running with the correct heap settings. Tip: Notice that for custom start scripts, the node manager also logs all environment variables for troubleshooting purposes. Return to the console and Lock it. Return to the Configuration > Server Start tab for MedRecSvr1. Edit the following values: Field
Value
Class Path
Arguments
Password/Confirm Password
Welcome1
Save and Activate your changes. Test automatic restart. a. From a Linux terminal, determine the process IDs associated with MedRecSvr1: ps –ef | grep MedRecSvr1 b.
Using the terminal, kill these processes: kill -9 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 12 Chapter 12 - Page 6
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
JAVA Memory arguments: -Xms512m -Xmx512m
Observe the node manager output while the server is restarted. Use the console or WLST to confirm that the server is running again.
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
c. d.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 12 Chapter 12 - Page 7
1. 2. 3. 4. 5.
If the /domains/MedRecDomain location does not yet exist, follow the Solution Instructions for the “Developing a Custom Monitoring Script” practice. If not already done, perform the section of the instructions entitled “Set up the practice.” Launch the Lab Framework command shell by executing the /bin/prompt.sh file. Change the current directory to . Execute the following: ant setup_solution The Lab Framework: −
Updates the node manager’s configuration
− −
Adds a custom start script to the domain Enrolls the node manager with the domain
−
Removes any start settings for MedRecSvr1
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 12 Chapter 12 - Page 8
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Chapter 13
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13
Chapter 13 - Page 1
Oracle University and Sentra inversiones y servicios LTDA use only
Practices for Lesson 13
Practices Overview In these practices you work with cluster problems both in the proxy to the cluster and in session replication. Naming Conventions In addition to the variable names listed earlier, these practices use the following variable name to refer to a location on your file system. This value is also defined as a Linux environment variable. Variable
Path
/u01/app/oracle/instances/webtier
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 2
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practices for Lesson 13
Duration: 45 minutes Skills Learned At the end of this practice, you should be able to: • Monitor cluster utilization under load • Monitor runtime statistics for the proxy plug-in • Analyze proxy plug-in logs • Tune proxy connection settings Overview In this practice, you work with a clustered version of the MedRec domain. The cluster consists of three members, which are accessed via a Web server proxy (Oracle HTTP Server). While simulating a user load on the cluster, you use server and proxy tools to monitor and troubleshoot some discrepancies with the distribution of the load across the cluster. The MedRec cluster architecture is shown in the following diagram:
Instructions 1. Initialize the MedRec cluster infrastructure. a. Shut down any running WebLogic Servers, including any started with node manager. b. Locate a Lab Framework prompt or start a new one. Change directories to . c. Execute the following: ant setup_exercise The Lab Framework: − Configures and starts an Oracle HTTP Server instance −
Creates a domain named MedRecClusterDomain
−
Starts all servers in the domain (log in as weblogic/Welcome1 when prompted).
− −
2.
Configures libraries and data sources on the clustered servers Deploys the MedRec application to the cluster (configured to use session replication) Tip: If you must later restart the servers manually, the new domain includes the startServer1.sh, startServer2.sh and startServer3.sh scripts. Tip: If you experience slow performance during this practice, you can shut down the admin server to free up resources. Monitor cluster utilization under load. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 3
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 13-1: Investigating Proxy Problems
Test that the Web server proxy is running by accessing this URL: http://localhost:7777/
b. Inspect the contents of the /resources/wlst/ monitorMedRecApp.py file.
c. d. e.
f.
3.
This WLST script monitors the current number of sessions and total number of requests associated with the MedRec application on each server in the cluster. From your Lab Framework prompt, execute monitorMedRecApp.py. Leave the script running. Launch another Lab Framework prompt and navigate to /client. Execute the runclients.sh script. This script simulates six concurrent users and directs requests through the Web server proxy. While the clients run, monitor the WLST output. Notice that over time the number of completed requests does not seem to be evenly distributed among the members of the cluster. For example: SVR1_SESSIONS SVR2_SESSIONS SVR3_SESSIONS 3 2 3 SVR1_REQUESTS SVR2_REQUESTS SVR3_REQUESTS 18 8 18
Tip: It is also suspicious that the current number of sessions on all servers does not equal the number of users (6). g. After the clients finish, kill the WLST script. Configure proxy debugging.
a. Edit the /config/OHS/ohs1/mod_wl_ohs.conf file. Note the current settings. b.
Tip: You might want to make a backup copy of the file before you change it. Within the block, add the following parameters: DebugConfigInfo ON Debug ALL WLLogFile /proxy.log
c.
4.
Save your changes. Then launch a Linux terminal and restart the proxy: > opmnctl stopproc > opmnctl startproc
d. Verify that the file proxy.log was generated. Analyze proxy output. a. Execute runclients.sh again. Wait for the clients to finish. b. From a browser, access the following URL: http://localhost:7777/medrec/test?__WebLogicBridgeConfig=true c.
Tip: Use 2 underscore characters ('_'). Confirm that during the most recent request to the proxy, all three members of the cluster were currently available:
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 4
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
a.
e.
Verify that although all requests were completed successfully, exceptions were generated. Specifically, there should be one or more READ_TIMEOUT exceptions.
f.
Open the file /proxy.log.
g.
Search for the text, READ_TIMEOUT. Locate log messages similar to the following: *******Exception type [READ_TIMEOUT] (no read after 30 seconds) ... caught exception in readStatus: READ_TIMEOUT ...: no read after 30 seconds ... PROTOCOL_ERROR: Backend Server not responding - isRecycled:0 *******Exception type [PROTOCOL_ERROR] (Backend Server not responding) ... Marking 127.0.0.1:7031 as bad
h.
Use the request ID of these log messages (701126098280723 in the preceding example) to try to determine the server and URL of the original request earlier in the log file. Your results should indicate MedRecSvr2 as the culprit. For example: Using Uri /medrec/patient/viewLoginResult.action ... trying connect to PRIMARY '127.0.0.1'/7031/0
i.
j.
Similarly, use the same request ID to determine which server the proxy failed over to. For example: general list: created a new connection to '127.0.0.1'/7041 for '/medrec/patient/ viewLoginResult.action', Local port:0 ... closeConn: pooling for '127.0.0.1/7041' closeConn: pooling '1'
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 5
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
d.
This implies that the initial connections to the servers did not fail. Find the Runtime Statistics section:
5.
MedRecSvr2 is taking too long to process some requests, and the proxy is timing out. Tip: Because session replication is enabled, a user’s work is uninterrupted after the failover. The use of transactions also helps ensure data integrity when the same request is executed multiple times. Tune proxy connection settings. a. Edit mod_wl_ohs.conf once again. b. c.
d.
e.
Update the WLIOTimeoutSecs property. Change the value to 120 seconds. Repeat the earlier steps to stop and restart the proxy server. Tip: Optionally you can first delete proxy.log to prevent the file from getting too large. You must delete the file before you restart the proxy server, so the file is recreated. Use the WLS administration console (port 7020) to Update the medrec application. Doing so resets the application’s runtime statistics. Tip: If you stopped your admin server earlier, you must restart it. Run the monitorMedRecApp.py WLST script again.
f.
Execute runclients.sh a final time. Wait for the clients to finish. Allow a few minutes for MedRecSvr2 to respond to all requests. Then verify that the load is now evenly distributed, as shown in the following: SVR1_SESSIONS SVR2_SESSIONS SVR3_SESSIONS 2 2 2 SVR1_REQUESTS SVR2_REQUESTS SVR3_REQUESTS 14 14 14
g.
Access the proxy debug page again by using the browser. Confirm that no exceptions were raised. Tip: Network issues are common causes of proxy timeouts. However, for the purposes of this practice, this problem is simulated using code. Those with some Java knowledge may examine the code responsible for MedRecSvr2’s unresponsiveness at /resources. When you are finished, stop the WLST script.
h.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 6
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
request [/medrec/patient/ viewLoginResult.action] processed successfully
1. 2.
Shut down any running WebLogic Servers. Launch the Lab Framework command shell by executing the /bin/prompt.sh file.
3. 4.
Change the current directory to . Execute the following: ant setup_solution The Lab Framework performs the same tasks as described in the “Initialize the MedRec cluster infrastructure” section, but configures Oracle HTTP Server with the solution settings.
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 7
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Solution Instructions
Duration: 50 minutes Skills Learned At the end of this practice, you should be able to: • Configure HTTP session debugging • Troubleshoot a session stickiness problem • Trace session replication across a cluster • Monitor HTTP sessions by using the console and WLST • Troubleshoot a session serialization error Overview Session replication or persistence is a critical component of WebLogic Server’s high availability solution. However, there are several types of configuration and development issues that can lessen the effectiveness of these features. Worse still, most of these problems can be difficult to detect without proper testing. MedRec IT has selected session replication as their high availability implementation for Webbased applications. While simulating a user load on the MedRec cluster, you use server logs, the console, and WLST to monitor and trace session replication and troubleshoot several potential problems. Instructions 1. Set up the practice. a. If you do not have a domain named MedRecClusterDomain, follow the Solution Instructions for the “Investigating Proxy Problems” practice. b. Locate a Lab Framework prompt or start a new one. Change directories to . c. Execute the following: ant setup_exercise The Lab Framework: − Starts an Oracle HTTP Server instance if not already running − Deploys a new version of the MedRec application and deployment plan 2. Configure HTTP session debugging. a. Access the administration console and Lock it. b. Edit MedRecSvr1. Click the Debug tab. c. Select the check boxes for these debug attributes: weblogic > core > cluster > DebugReplication weblogic > servlet > internal > session > DebugHttpSessions Click the Enable button. d. Enable the same debug attributes on the remaining managed servers. e. Activate your changes. f. From your Lab Framework prompt, navigate to /resources/wlst. 3.
g. Execute the monitorMedRecApp.py WLST script. Leave the script running. Observe a session stickiness problem. a. Launch another Lab Framework prompt and navigate to /client. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 8
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Practice 13-2: Investigating Cluster Replication Problems
c.
d.
Execute the runclients.sh script. This script simulates four concurrent users and directs requests through the Web server proxy. While the clients run, monitor the WLST output. Notice that the current number of sessions on all servers does not equal the number of users. For example: SVR1_SESSIONS SVR2_SESSIONS SVR3_SESSIONS 4 4 3 SVR1_REQUESTS SVR2_REQUESTS SVR3_REQUESTS 9 8 8 After the clients finish, kill the WLST script.
e. View the contents of the file /client/logs/out_0.log. f.
4.
At the end of the file, verify that no errors were received by the clients: ... Tests Errors ... Response errors ... Totals 28 0 ... 0
The results so far seem to indicate that the application was able to function correctly due to replication, but that clients were not properly pinned to their primary cluster members. Tip: It may be helpful to disable text wrapping in your text editor, if enabled. Trace and configure session IDs. a. View the most recent debug messages in the /proxy.log file. b. Search for messages with the phrase “Hdrs from WLS.” These values represent the HTTP headers that were received as part of each server’s response to a client request. c. Locate the header that lists the cookies in the response, similar to the following: Hdrs from WLS:[Set-Cookie]=[MEDRECID=c0d3LrCh82QqWCmngS...] d. e.
Given the fact that only one cookie is listed, this must be the current user’s session ID. Direct a Web browser to the proxy debug page: http://localhost:7777/medrec/test?__WebLogicBridgeConfig=true In this list of configuration attributes, locate the value for WLCookieName. It seems that the proxy is searching for a different session ID cookie than the application generates.
f.
Edit the /applications/medrec-plan.xml file.
g.
Confirm that this deployment plan overrides the default session ID cookie name for this application: SessionDescriptor_cookieName /weblogic-web-app/session-descriptor/cookie-name
h.
At the top of the file, change the value of this variable to match the cookie name used by the proxy: Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 9
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
b.
i. j. k.
5.
Save the file. Then use the console to Update the medrec application. Repeat the earlier steps to run the test client and to monitor session usage. Confirm that now upon completion of the test, the servers host the correct number of sessions (4). l. When you are finished, stop the WLST script. Monitor HTTP sessions by using the console. a. Return to the console and Lock it. b. Edit the medrec application. c. Scroll down and select the medrec Web module within this application:
d. e. f. g. h. i.
Click the Configuration > General tab. Set the value of Monitoring Attribute Name to user. Then click Save. Also note the current Session Timeout value. Update the medrec application again and Activate your changes. Return to the medrec Web module page. Click the Monitoring > Sessions tab. There may or may not be sessions remaining from the prior test. Customize this table and add the Monitoring ID column.
j.
Click the Auto Refresh icon. The table refreshes its data every 10 seconds, by default. k. Run the client script once again (runclients.sh) l. As the clients run, observe the list of sessions that are created in the cluster. There should be a total of four. m. Choose any one of the sessions to analyze. Make a note of the session's host Server and its Creation Time. Also use this opportunity to inspect its Monitoring ID:
Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 10
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
SessionDescriptor_cookieName JSESSIONID
c.
Immediately after the message stating that the session was created, find a replication debug message that provides this session’s internal replication ID. For example: Creating primary 1243163881371479560 for application key /medrec>
d.
Search the log for messages related to this replication ID. Try to determine the secondary server for this session. For example:
e.
To confirm this, search the secondary server’s log for debug messages related to either the session ID or replication ID. Determine the process ID of the secondary server. For example > ps –ef | grep MedRecSvr1
f.
oracle 31578 31539
... java ... -Dweblogic.Name=MedRecSvr1 ...
g. h.
7.
Kill the server process (kill -9 ). Inspect the latest server logs to confirm that another secondary server was selected for this session ID. i. Restart the failed server. Observe a session failover problem. a. Obtain and record the current process ID for any one of your clustered servers.
b. Execute the /client/runadminclients.sh script. This script tests a different portion of the MedRec application. It simulates four concurrent users and directs requests through the Web server proxy. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
Practices for Lesson 13 Chapter 13 - Page 11
Oracle University and Sentra inversiones y servicios LTDA use only
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
6.
Tip: The Monitoring ID displays a generated internal ID by default, or the user’s name when found in the session. It does not display the user's real session tracking ID for security reasons. n. When the clients finish, click the Auto Refresh icon again to disable it. Trace replication for a specific session. a. Access the log of the server for your chosen session. b. Search the log for HTTP session debug messages whose timestamp matches your chosen session's creation time. These log messages will also indicate the session's ID. For example:
The client script prompts you to kill one of your managed servers: Kill a server in the cluster...
d. e. f.
Once again, use the kill command to terminate the selected server processes. When the clients finish, repeat the earlier steps to inspect the client log file. Confirm that some client requests encountered errors (HTTP 500): ... -> 500 Internal Server Error, 883 bytes ... ... Tests Errors ... Response errors ... Totals 20 0 ... 6
g.
Search the log file of the failed server. Once again use the latest debug messages to determine which server(s) was selected as the secondary for replication. Inspect the latest log messages of the secondary server(s). Locate the application error:
