MQ-related CRON job setting

Recently my colleague was faced with the crontab issue. Below is the description:

1. CRON Tab

We developed a JAR where it connects to the Queue Manger and puts message into the Queue to kick start the WMB flow. We have created a Shell script to send message to the queue. It is successful by executing the Shell script manually where it connects to the Queue Manager and successfully put the message into the Queue.

Unfortunately, the CRON job failed to execute this script. After a few rounds of investigation we found out that this is due to the AIX Environment Setting. We need to set the environment variables at our Shell script as below before connecting to MQ:
. /opt/IBM/mqsi/6.1/bin/mqsiprofile

The default environment setting for CRON does not have the MQ Specific environment.
Therefore, when we are using server binding, we need in include this into our Shell script.
If the Shell is running externally as client, we do not need to include this.

Modify WSDL to get the java client generated right

I have to generate java client for this WSDL. However the bean generated is not what i expect. The reason is the SOAP response format actually looks like this:

<data>
<field id="field1"><field>
<field id="field2"><field>
...
</data>

The bean generated is like this:

public Class Response{
Field[] data;
....
}

I failed to get the value of each . The Field class does not provide a getValue method. Instead it provides getId method which returns the field id.

I had 2 solutions:

1) Modify the generated java web-service client code. (Those serializer, deserializer etc)
2) Modify the WSDL to get the code generated as according to what i want.

I chose the second option.

I commented this segment of the WSDL:

<xsd:element maxOccurs="unbounded" name="field">
<xsd:complexType>
<xsd:sequence maxOccurs="1" minOccurs="1" />
<xsd:attribute name="id" type="xsd:string" />
</xsd:complexType>
</xsd:element>

I replace the above segment with:

<maxoccurs="unbounded" minoccurs="0" name="field" nillable="true" type="xsd:string"/>

Done!

Issue with JMS & MQInput node in WMB

The XML is sent by JMS as below format:

reqMsg.setJMSType("mcd://mrm//" + msgType + "?format=CwXML");

WMB trace file throws the below error:

See the following messages for details of the error.
2010-03-31 16:49:05.949233 13942 ParserException BIP5313E: Message format ''CwXML'' is not defined for message set ''RR_MVNO_Port_In_MessageSet (H74SG78002001)''.
The broker received a message for processing within the MRM domain.
However, the message cannot be processed because the message format specified in the physical format identifier ''CwXML'' has not been defined for message set ''RR_MVNO_Port_In_MessageSet (H74SG78002001)''.
The message format describes the physical representation of the message, and you can specify this format either in the properties on an input node, or dynamically within an MQRFH2 message header.

The solution is to add one more "XML Wire Formats" at the Message Set.

Also, need to make sure the MQInput node's Message Domain & Message Set are set with the correct values.

Another cert issue

We have this standalone java program running on a AIX box calling a web service which is running on https. The web service provider provided us the cert and we imported this cert to the cacerts keystore using IBM Key Management tool(Alternatively you may use the java keytool command).
(Anyway this is the first time i am exposed to this, so please correct me if i am wrong)

At first the web service called failed. We got the below error:

unable to find valid certification path to requested target

We checked the shell script where we started this program. In the shell script, the java command path is specified as /usr/java5_64/bin and the cacerts actually resides at /usr/java5_64/jre/lib/security

Therefore we changed in our script from /usr/java5_64/bin/java to /usr/java5_64/jre/bin/java and now it is working well.

Please find the 2 URLs as below

Link1

Link2

Issue with Java Security Provider

While working on one enhancement. I encountered a weird problem. The program within WBIA framework failed in calling web service. It was throwing me a NullPointerException from Axis library as below:

at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
at org.apache.axis.utils.SessionUtils.generateSessionId(SessionUtils.java:62)
at org.apache.axis.SOAPPart.<init>(SOAPPart.java:164)
at org.apache.axis.Message.setup(Message.java:377)
at org.apache.axis.Message.<init>(Message.java:246)
at org.apache.axis.client.Call.invoke(Call.java:2425)
at org.apache.axis.client.Call.invoke(Call.java:2366)
at org.apache.axis.client.Call.invoke(Call.java:1812)

I made the investigation as below:

1) I put a java.security.Security.getProviders() in my program to list all the security providers. I compared the results from successful call (standalone java program running at the same AIX server) and unsuccessful call (program extending from WBIA framework running at the same AIX server).

Here is the result:

//Unsuccessful call.
- IBMJSSE2
- IBMJGSSProvider
- IBMCertPath

//Successful call
- IBMJSSE2
- IBMJCE
- IBMJGSSProvider
- IBMCertPath
- IBMSASL

Obviously some security providers are missing hence caused the unsuccessful call. I double confirmed this by modifying my java.security file at local and tested the same program from my local. Yes the web service call actually failed due to the missing security provider. The missing security provider is actually com.ibm.crypto.provider.IBMJCE

Therefore, i did the below the solve this issue:

1) Set the Java home path to the JRE folder instead of the JDK folder itself.

2) I added the security provider at runtime.
java.security.Security.addProvider(new com.ibm.crypto.provider.IBMJCE());

Finally it worked. One question is, how come it somehow did not refer to the java.security file that i set everything correctly else i do not have to do the 2nd step as mentioned. I still need to find out this.

Load Test

Past few days i encountered one serious production issue, which the web service timeout before the backend process sent back the message to the SOAP response node in WMB flow. This was due to the high load in production that caused the delay in the backend process.

The whole process is: WMB flow extracts SOAP request message, turns it into a MQ Header message, puts the message to Java adaptor's request queue, then Java adaptor processes the content and puts a response message to the response queue, then WMB flow will read the message in the response queue and will turn it into a SOAP response.

In the progress of investigation, we decided to move one database query out from the java adaptor program to the WMB flow. The reason being is sometimes the db connection will just drop and the java adaptor program has to spend some time reinitiating the db connection. In fact according to the time recorded in the adaptor's log the process was still quite fast, it's even within miliseconds. Well we just gave it a try.

After making all the necessary changes we deployed the works to the testing environment. I then performed some load tests to the web service. The result was negative. The tps was around 3.5 but the average time taken was > 10 seconds and we had to achieve < 10 seconds! So still plenty of work to do :(

We performed a few cycles of investigation, trial and error & testing until we found the solution - increasing the WMB flow instance. This is a setting in the WMB Toolkit. So we increased the number of instances to 3. This time we managed to achieve an average time of 3 seconds! We were so happy and excited!

What about the cause? See below:

There were many messages piling at the response queue which is supposed to be read by the WMB flow. These messages were sent in by the Java Adaptor after it finished processing the data. Apparently these messages were not picked up by the WMB flow in time hence causing the timeout error. By increasing the WMB flow instance, concurrently there are 3 identical WMB flows reading the content of this queue, hence problem resolved!

I was really relieved!

Classloading in Websphere Administrative Console

I have this web service server application which has been running fine at a Websphere Process Server. One day, my application's log was writing to another application B's log. That is really weird. I checked and found out that this B was recently deployed to the server only. The cause for my application to write to B's log is due to B's log4j jar file was put in the server's common directory ! If we do not change the classloading policy, which is default to PARENT_FIRST, then Wepshere's classloader will load that common directory first only will load the jars in our application's WEB-INF/lib directory.

Hence i logged on to Websphere Administrative Console, went to my application's war file and changed the classloader policy to PARENT_LAST. By doing this Websphere will find and load the neccessary classes from your web module's WEB-INF/lib(web module classloader) first, also you must ensure that the jars you put under your application's WEB-INF/lib folder do not clash with the one at server's common directory, else you will get a java.lang.LinkageError exception due to classloader is trying to load duplicate class.