Questo sito utilizza i cookie per migliorare l'esperienza di navigazione. Continuando la navigazione acconsenti all'utilizzo dei cookie.

A REAL CASE of high CPU consuming analisys by a java process

The scenario is High CPU usage by a java process, It pretty often appens did you know? This is my java process:

$ps -o pid,cmd -e | grep java | grep prod| tr " " "\n"
13605
java
-Xmx8192m
-Xms8192m
-XX:MaxPermSize=1024m
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=5601
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-XX:+UseMembar
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing
-Dsun.rmi.dgc.server.gcInterval=3600000
-Dsun.rmi.dgc.client.gcInterval=3600000
-Doracle.jdbc.timezoneAsRegion=false
-Djava.security.egd=file:///dev/urandom
-DgpInstance
-server
-jar
XXXXXXXXX.jar
-n
GC01
-p
production
-m
TEST

The goal is to identify which threads of the process is consuming the most CPU of the host. In this perspective the top command could be very usefull,just add some awk stuff to limit the column displayed ( with -b flag ) and use the -H flag to print lwps.

$ top -b -n 1 -H | awk 'NR>6 && NR<13 {printf "%6s %-4s %-4s %-s\n",$1,$2,$9,$NF}'
   PID USER %CPU COMMAND
 13608 gp   26.5 java
 13607 gp   24.6 java
  5121 gp   13.3 java
  5290 gp   13.3 java
  6060 gp   11.4 java                                                                                                            

As a result we can say that only two thread are consuming CPU:  13608  and 13607 .

The thread id  number is printed in decimal format and we need to convert it in exadecimal

$echo "obase=16; 13608"|bc

3528

the next step is to make a thread dump of the jvm:
$kill -3 13605

the last thing is to find in the standard output of the java process the exadecimal number foud before
$grep 3528 nohup.out
 "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f9ea0017800 nid=0x3528 runnable

In this case we found out that the cpu is consumed by minor collections, So We probably have to tuning the jvm heap.

DECRYPT WEBLOGIC ADMIN USER PASSWORD


create the following script : $DOMAIN_HOME/security/decrypt_password.py

from weblogic.security.internal import *
from weblogic.security.internal.encryption import *
encryptionService = SerializedSystemIni.getEncryptionService(".")
clearOrEncryptService = ClearOrEncryptedService(encryptionService)
 
# Take encrypt password from user
pwd = raw_input("Paste encrypted password ({AES}fk9EK...): ")
 
# Delete unnecessary escape characters
preppwd = pwd.replace("\\", "")
 
# Display password
print "Decrypted string is: " + clearOrEncryptService.decrypt(preppwd)
 
 
 
Get crypted password aand username  from boot.properties : 
 grep password  $DOMAIN_HOME/servers/AdminServer/security/boot.properties|  sed -e "s/^password=\(.*\)/\1/"
 grep username  $DOMAIN_HOME/servers/AdminServer/security/boot.properties|  sed -e "s/^username=\(.*\)/\1/"
 
set environment : 
.   /ltm/oracle/Middleware/user_projects/domains/base_domain/bin/setDomainEnv.sh 
 
 
launch the command and paste the string that you want to decrypt:
java weblogic.WLST decrypt_password.py
({AES}fk9EK...):{AES}noXFmhZdQI7xOf8Hs2zx6Vu/cTWbvi53mRnch04eLNk=

Bash Scripting

Yesterday, at work i had to solve a little problem: how can i verify that a text file A is conteined on another text file B.
IN ather words i have to take every row of file A and check if it is also a row of file B.

In a BASH script.

Easy? Yes it is. And then because it took me two hours? It was not quite so easy. At least for me.

Leggi tutto: Bash Scripting

WLST - delete JMS messages

If you need to purge all messages from a JMS Queue, from command line  this is the WLST code that do the stuff for you : the only thing you have to do is find the path to the jms destination in the serverruntime tree :-)


ulrm1='t3://managedListenAddress:7011'
def conn(admurl):
   usr='weblogic'
   pwd='*******'
   try:
      print 'usr='+usr
      print 'pwd='+pwd
      print 'ulrm1'+ulrm1
      connect(usr, pwd, url=admurl)
   except ConnectionException,e:
      print 'Unable to connect to server...'
      exit()
conn(ulrm1)
serverRuntime()
cd('/JMSRuntime/<server>.jms/JMSServers/JMSServer1/Destinations/<QueueSetName>!<JMSServerName>@<QueueName>')
cmo.deleteMessages('')
disconnect()





BASH MAGIC

Talking about awk.

As you certainly know, there is  a function in awk that can return a portion of a string, this function is called substr. The  syntax is : substr(s,c,n)
where s is the original string, c iis the start posizion of the substring you want to get, and n is the length of the substring.

The problem I should face some days ago was to investigate in the logs of a Jboss server to get  the long executions time of a certain remote call, logged as below :  
2014-20-05 14:25:01,004 ...... Time:[245]

So I wanted the substr of "Time:[245]" to print the lines which the execution time was for example > 500 ms.

The solution is pretty easy: you want to get the substring of Time:[xxx] starting after the char "[" and finishing before "]" , than  you have to perform a numeric comparision between the substring obteined and the number 500. To get the position of the square brackets you can use another simple awk function  called "index".

but, as you can imangine,  substr return a string, so how do you cast this string in a number?
Below is my solution :

awk '{
b=substr ( $10, 7 , (index( $10,"]") - index( $10,"[") -1 )  )  + 0
if (  b > 500 )
        print  b  
}'



 I just added a zero to the result of substr. I know it is not very elegant but it works Fico

of course, if you have a better solution just post it!



full text Search  in a bash script

As you perfecly know, the find command can search files , and you have so many options to decide what files you want to find, and execute commands on every file.
This is done with the exec option. That said, you can easily perform some kind of full text search over a set of files.

For instance let's suppose we want to search all the strings that represent IP address on all the configuration files of a servlet engine like TOMCAT. This is easily done when you
have a regular expression that macth IP addresses : ([0-9]{1,3}\.){3}[0-9]{1,3}

Leggi tutto: BASH MAGIC