As a developer in a service industry we often learn a lot during a particular phase of project, this post is something similar where in we learnt more apart from writing simple java code. Below shown is one of the process which was developed as a part of Java project,
Pseudo-code of the process is
Step 1: Fetch data from the table
Step 2: Loop through all the records which are fetched
Step 3(While Loop): For each record try to fetch the column information and append to string buffer and perform some replace operations.
Step 4: Write this string buffer into a file after completion of the loop
Step 5: Repeat from Step 1 by changing the cursor size.
Problem with the above process!!!!!
There is a query which is provided by the customer which returns huge data and which has to be written into a file by using the above process. This process is not that difficult but while performing this process the ESB layer is utilizing lot of resources than needed. Below is the history about this process how it has made efficient and made to use less resources.
"Indexes" the Initial Problem :
We have experienced a huge time lag in firing a query probably around 5 mins, if this is the case then obviously it will kill the performance of the process. This is the initial hurdle which has to be passed. We found out that indexes were missing in the major table which was used in the query. Once we had fixed that we could be able to fetch data instantly (prob 2 mins).
"Smart Usage of WsApp properties" :
I have already explained about this topic in the other post please refer this i.e,
"Out Of Memory"(replacing content of huge String Buffer object):
out.write(sbf.toString().replaceAll("_X005B_", "[").replaceAll("_X005D_", "]"));
out : FileOutputStream
sbf : String buffer which is as huge as enough to crash a container
AS mentioned sbf was a huge string and as we have did operations on this crashed the container since there was no sufficient memory available to do this operation.
As we have got an out of memory exception, we have added a JRE property to increase the NOM memory
-Dbus.xml.vm.maxsize=1072M
But no LUCK :(
We had tried logging memory occupied at various stages and we had observed few things
-- The total memory allocated for the container is increasing exponentially. We felt like the memory was not being released, on analyzing we found out that the memory leak was at query.getObjects() line, we also created a busObject which will be having the iterator.nextElement() as the value. As these are platform level objects normally the Java layer would release soon after the execution of the method. But as we are still in the method for a longer time, hence its not being released. So there was a need to delete them manually which can be done as below
if(Node.isValidNode(bObj._getObjectData()))
Node.delete(bObj._getObjectData()); //This step is added while reading each row of the iterator element.
As expected this worked. The reason being Java layer would delete platform level objects after completion of the method and not in between.
Lessons Learnt :
Pseudo-code of the process is
Step 1: Fetch data from the table
Step 2: Loop through all the records which are fetched
Step 3(While Loop): For each record try to fetch the column information and append to string buffer and perform some replace operations.
Step 4: Write this string buffer into a file after completion of the loop
Step 5: Repeat from Step 1 by changing the cursor size.
Problem with the above process!!!!!
There is a query which is provided by the customer which returns huge data and which has to be written into a file by using the above process. This process is not that difficult but while performing this process the ESB layer is utilizing lot of resources than needed. Below is the history about this process how it has made efficient and made to use less resources.
"Indexes" the Initial Problem :
We have experienced a huge time lag in firing a query probably around 5 mins, if this is the case then obviously it will kill the performance of the process. This is the initial hurdle which has to be passed. We found out that indexes were missing in the major table which was used in the query. Once we had fixed that we could be able to fetch data instantly (prob 2 mins).
"Smart Usage of WsApp properties" :
I have already explained about this topic in the other post please refer this i.e,
http://cordyslearning.blogspot.in/2014/02/smart-usage-of-ws-app-connector.htmlKeeping the query cache of wsapp container only for 30 secs resulted in fetching the data from DB layer multiple times even after implementing cursor. Hence this is avoided by making the query cache refresh interval to 300 (5 Mins).
"Out Of Memory"(replacing content of huge String Buffer object):
out.write(sbf.toString().replaceAll("_X005B_", "[").replaceAll("_X005D_", "]"));
out : FileOutputStream
sbf : String buffer which is as huge as enough to crash a container
AS mentioned sbf was a huge string and as we have did operations on this crashed the container since there was no sufficient memory available to do this operation.
As we have got an out of memory exception, we have added a JRE property to increase the NOM memory
-Dbus.xml.vm.maxsize=1072M
But no LUCK :(
We had tried logging memory occupied at various stages and we had observed few things
-- The total memory allocated for the container is increasing exponentially. We felt like the memory was not being released, on analyzing we found out that the memory leak was at query.getObjects() line, we also created a busObject which will be having the iterator.nextElement() as the value. As these are platform level objects normally the Java layer would release soon after the execution of the method. But as we are still in the method for a longer time, hence its not being released. So there was a need to delete them manually which can be done as below
if(Node.isValidNode(bObj._getObjectData()))
Node.delete(bObj._getObjectData()); //This step is added while reading each row of the iterator element.
As expected this worked. The reason being Java layer would delete platform level objects after completion of the method and not in between.
Lessons Learnt :
- While using a query inside Java which returns huge data, make sure you have indexes available in order for faster retrieval.
- Make sure you implement cursor in such cases @ Java layer.
- Since this service is used as a web service make sure you provide sufficient timeout for this service in BPM activity.
- Make necessary changes to the container so that it will be beneficiary in this scenario.
- And Last but not least with the help of JConsole make sure you avoid memory leaks even if you have to delete platform level variables like query.getObjects(), itr.nextElement() etc.,.
No comments:
Post a Comment