I have become a Manager

I have become an Engineering Operations Manager since early 2020 (of course with the help from my Special Mind Power Technique derived from Ancient Text of Confucianism) 

I am now managing 12 Senior Engineers, including 4 Leads, for ID&F products, hosted services & cloud services. 

Since then, I spent most of my time managing the people and the services. I have not got much time in developing something. Hence this space has been idle for some time. 

Today, I get to know there was one programming competition organized by Panasonic back in year 2007. Someone provided the winner list in Lowyat forum. Therefore I further studied the current status of the winners and came out with the table below. 

Rank	Name	College	Current Status	Current Company	Location	Remark
1st prize winner	Tan Aik Keong	MMU	Founder	Agmo Studio	MY
2nd prize winner	Chooi Kah Wai	APIIT	Technical Lead	PathDAO	MY
3rd prize winner	Simon Lim Hao Wooi	UTAR	Software Engineer	Canva	AU
Outstanding winner	Gan Eng Chin	UM	Senior Software Engineer	Automattic	MY	One of the early engineers in Experian CheetahMail
Outstanding winner	Fong Kha Chun	UM
Outstanding winner	Kwan Toh Choong	Curtin U of Tech	Digital Platform Design Manager	Shell	MY	11 years in Shell
Outstanding winner	Lee Chee Cheong	MMU				Last job as Technical Manager at iRadar Sdn Bhd
Outstanding winner	Tee Shu Hui	UTM	Senior Firmware Development Engineer	Micron Tech	SG
Merit Winners	Tan Kian Tat	MMU	Self-Employed			SAP Consultant since grad
Merit Winners	Lim Fang-Yin	UTAR				Active in Quora
Merit Winners	New Chin Jian	UM
Merit Winners	Choong You Qi	MMU	Staff Android Engineer	Setel	MY
Merit Winners	Chan Kin Meng	MMU	Founder CEO	Gameconomy	MY	Web3.0 Gaming
Merit Winners	Siew Lead Choon	UM				SAP?
Merit Winners	Yeoh Yan Pore	USM	GM	IHS Group
Merit Winners	Ernest Eg Ket Lung	USM
Merit Winners	Chan Chen Shyang	Curtin U of Tech	IP Consultant	Nokia	MY	Nokia till today
Merit Winners	Ahmad Irshad B. Abdul Hamid	UniTen	Software Engineer	Sage	MY
Merit Winners	Chua Fook Ching	MMU	Business & Integration Arch Specialist (SAP)	Accenture	MY
Merit Winners	Khor Yit Kean	TARC

Construct complex nested JSON file based on a list of JSON path

What happened was my QAs did not want to change their mind to prepare data in Excel, using 'path' concept. They're too used to that method already since the XML time. They'd like to continue the same. Row contains Path and Cell/Column contains value. Each Cell/Column represents one test case. Sounds good?

I googled and googled and tried so many solutions suggested by a lot of kind men in StackOverflow, none of the solutions really gave me what I wanted. No one really did this before that building a complex/complicated nested JSON file. I decided to write the algorithm myself. 

How does the so called complex/nested look like? It looks like below. And I had to build the JSON from this list of paths. 

$.JSONDoc.OC.AL.data_type
$.JSONDoc.OC.AL.value
$.JSONDoc.OC.SIG.data_type
$.JSONDoc.OC.SIG.value
$.JSONDoc.IN.MCPL.PRODTYPE.data_type
$.JSONDoc.IN.USER.COUNTRYCODE.data_type
$.JSONDoc.IN.USER.COUNTRYCODE.value
$.JSONDoc.IN.MCPL.LOCATION.COUNTRY.data_type
$.JSONDoc.IN.MCPL.LOCATION.COUNTRY.value
$.JSONDoc.IN.USER.FRSCR.data_type
$.JSONDoc.IN.USER.FRSCR.value
$.JSONDoc.IN.MCPL.ITEMS.value[0].QUANTITY.data_type
$.JSONDoc.IN.MCPL.ITEMS.value[0].QUANTITY.value
$.JSONDoc.IN.CLU.SPLT180.data_type
$.JSONDoc.IN.CLU.SPLT180.value
$.JSONDoc.OUT.TXN.FRC.data_type
$.JSONDoc.OUT.TXN.FRC.value[0]

The algorithm below works well so far for any JSON path that consists of JSON Array as well. I know it's not perfect but I really hope it's helpful to you guys who've bumped into the similar issue that I had. One thing to note is that the 'has' method from JSONObject does not really give accurate result if the JSONObject has multi-level of nested JSONObject. Therefore I also wrote one for myself. 

public static JSONObject createJSONObject(String[] keys, int index, String value, JSONArray jArray, JSONObject masterJObj) {
        if (index < keys.length) {
            String key = keys[index];
            String nextKey = keys[index + 1];
            if (key.contains("[")) {
                return createJSONObject(keys, index + 1, value, jArray, masterJObj);
            }
            if ((index < keys.length - 2) && nextKey.contains("[")) {
                JSONArray jsonArray = new JSONArray();
                JSONObject jobj = new JSONObject();
                Object obj2 = getObject(masterJObj, key);
                if (obj2 != null) {
                    if (obj2 instanceof JSONObject) {
                        jobj = (JSONObject) obj2;
                        if (jobj.has(nextKey)) {
                            jsonArray = jobj.getJSONArray(nextKey);
                        }
                    }
                }
                createJSONObject(keys, index + 1, value, jsonArray, masterJObj);
                jobj.put(nextKey, jsonArray);
                return jobj;
            } else {
                JSONObject jsonObject1 = null;
                if (jArray != null) {
                    if (!jArray.isEmpty()) {
                        for (int j = 0; j < jArray.length(); j++) {
                            jsonObject1 = jArray.getJSONObject(j);
                            jArray = null;
                            break;
                        }
                    } else {
                        JSONObject jsonObject2 = new JSONObject();
                        jsonObject1 = new JSONObject();
                        jsonObject1.put(key, jsonObject2);
                        jArray.put(jsonObject1);
                    }
                } else {
                    Object obj1 = getObject(masterJObj, key);
                    if (obj1 != null) {
                        jsonObject1 = (JSONObject) obj1;
                    }
                }
                if (jsonObject1 == null) {
                    jsonObject1 = new JSONObject();
                }
                if (index == keys.length - 2) {
                    JSONObject jsonObject2;
                    if (jsonObject1.has(key)) {
                        jsonObject2 = jsonObject1.getJSONObject(key);
                    } else {
                        jsonObject2 = new JSONObject();
                    }
                    if (nextKey.contains("[")) {
                        nextKey = nextKey.substring(0, nextKey.indexOf("["));
                        JSONArray jArray1 = new JSONArray();;
                        if (!jsonObject1.isEmpty() && jsonObject1.has(nextKey)) {
                            jArray1 = jsonObject1.getJSONArray(nextKey);
                        }
                        jArray1.put(value);
                        jsonObject1.put(nextKey, jArray1);
                    } else {
                        if (!jsonObject1.isEmpty()) {
                            if (jsonObject1.has(key)) {
                                JSONObject jsonObject3 = jsonObject1.getJSONObject(key);
                                jsonObject3.put(nextKey, value);
                            } else {
                                jsonObject1.put(nextKey, value);
                            }
                        } else {
                            jsonObject2.put(nextKey, value);
                            jsonObject1.put(key, jsonObject2);
                        }
                    }
                    return jsonObject1;
                } else {
                    JSONObject returnedJObj = createJSONObject(keys, index + 1, value, jArray, masterJObj);
                    if (returnedJObj != null) {
                        if (returnedJObj.has(nextKey)) {
                            jsonObject1 = returnedJObj;
                        } else {
                            if (jsonObject1.has(nextKey)) {
                                if (!returnedJObj.isEmpty()) {
                                    String nextNextKey = returnedJObj.keys().next();
                                    Object jObj3 = returnedJObj.get(nextNextKey);
                                    if (jObj3 instanceof JSONObject) {
                                        jsonObject1.getJSONObject(nextKey).put(nextNextKey, jObj3);
                                    }
                                }
                            } else {
                                jsonObject1.put(nextKey, returnedJObj);
                            }
                        }
                    }
                }
                return jsonObject1;
            }
        }
        return null;
    }

    public static Object getObject(Object object, String searchedKey) {

        if (object instanceof JSONObject) {

            JSONObject jsonObject = (JSONObject) object;

            Iterator itr = jsonObject.keys();

            while (itr.hasNext()) {

                String key = (String) itr.next();

                if (!searchedKey.equals(key)) {

                    Object obj = getObject(jsonObject.get(key), searchedKey);

                    if (obj != null) {

                        return obj;

                    }

                } else {

                    return jsonObject.get(key);

                }

            }

        } else if (object instanceof JSONArray) {

            JSONArray jsonArray = (JSONArray) object;

            return getObjectFromJSONArray(jsonArray, searchedKey);

        }

        return null;

    }

Camel-Context(DSL) only, No Coding! How to Log? How to Handle Error? How to Configure Retry? How to work out the Property Placeholder?

I'm excited to have found new way in Apache Camel to build specific feature without writing any single code, which has always been my way due to time constraint and flexibility (to make everything configurable and every exception handled properly). It was a long winding process with many many rounds of trial and error.

What I'm trying to achieve here is very simple, just read file and SFTP to the other side and Backup the original Source File! If this would have been done in a Java/Groovy and registered as a Spring Bean, then it shall be very fast. However I chose to do it the Camel-way!

1. In your Camel-Context.xml (Spring DSL), declare a Route.  
2. The <From> should be the File EIP. Remember to set a "Delay" for the file polling interval. 
3. Next, use the SFTP EIP for the <To>

Done. So simple! However...

I want to LOG the entire process, such as:  

1. What Are the Files Being Picked Up? 
2. Whether the SFTP is Successful or Failed?  
3. Was the SFTPed file moved to a Backup folder? 

For my own knowledge, I was using 2 ways to log:

1. Camel's Log EIP - To log Route process + Error to the Application-specific log. 
2. Camel's Log - To log Error to the General Error log. 

**For Error, will be written to both logs. 

Camel's Log EIP 

<camel:log message="[MyTimer2] SFTP Begins ${file:name}" loggingLevel="DEBUG" logName="myTimer2Logger" />

**${file:name} here is a kind of Reserved Keyword inside Camel's. It's called the File Expression language and this specific one will output the name of the file picked up in the log.  

Camel's Log

<camel:to uri="log:error-rollingFileAppender?level=ERROR&amp;showCaughtException=true&amp;showStackTrace=true" />

** First Parameter (in this case - error-rollingFileAppender) refers to the Appender setup in my Application's LogBack.xml

Question, where should I place my LOG Error Block? I should only LOG Error upon Exception right? 

First, I tried the <DoTry> - <DoCatch> way, I put my LOG Error function within <DoCatch>. However, there is a flaw here! Upon SFTP Exception, the File:// will still regard the SFTP as successful and continued to move file to the Backup folder. 

Finally I found the right way, use <OnException>! So just put the Log Error block inside your <OnException>, which is inside the <Route>! 

<camel:exception>java.lang.Exception</camel:exception>
<camel:to uri="log:error-rollingFileAppender?  level=ERROR&amp;showCaughtException=true&amp;showStackTrace=true" />
<camel:log message="[MyTimer2] File Transmission Error ${exception.stacktrace}" loggingLevel="ERROR" logName="myTimer2Logger" />
</camel:onException> 

Is that all? Nope.

I wanted to Retry for a fixed number of times upon any kind of Exception! 

Easy. Just declare the below:

<camel:redeliveryPolicyProfile id="myRedelivery" retryAttemptedLogLevel="ERROR" maximumRedeliveries="${H2MaxRetry}" redeliveryDelay="${H2RetryDelay}" />

Then add this as the "redeliveryPolicyRef" to your Route.

Last but not least, I wanted to set all the properties, such as the Input Directory, the SFTP host, username, password and etc in a single properties file! 

Here's the trick:

Use <Endpoint> for your File:// and SFTP:// 

Somehow only Endpoint allows Spring Property Placeholder! 

<camel:endpoint id="send_source" uri="file://${app.dir}/filewatcher/anz_output?delay=${myTimer2}&amp;move=${app.dir}/filewatcher/anz_backup/$simple{file:name}.done&amp;moveFailed=${app.dir}/filewatcher/anz_bad"/>

<camel:endpoint id="send_remote" uri="sftp://${H2hostname}${H2Path}?username=${H2user}&amp;password=${H2password}&amp;soTimeout=${H2SocketTimeout}&amp;binary=${H2Binary}"/>

**These properties placeholder are stored in the properties file configured at the PlaceholderConfigurer

Run Software in Hidden Mode

Many years ago I was building a Car Workshop Management System. The client is a Car Workshop in Johor. Basically the client uses this software to take order, to track work progress and to generate receipt to customers. The software is able to generate "Account Summary" as well showing the total sales and receipts over a period of time.

One day my boss came to me with a new requirement. The client wanted to have a "secret" mode of the software. Under this secret mode, the client can show to Tax Officers different series of sequence number for invoices and receipts.

So in the software we keep two IDs with running number as the value, one is the "real" invoice number and another one is the "fake" invoice number. So when the client runs the software under "secret" mode, the Account Summary report will show only those transactions with fake number.

Ha ha. What a fun memory!

Oracle SOA Composite SCAC-50012 error during Build

Earlier on my head was spinning for trying to resolve this SCAC-50012 error. JDeveloper throws out this error while building the SOA Composite. 

The compiler suggests me to go to a specific location for the scac.log file. It said the log should contain the detail for the error. 

Well yes there is error detail in the log. It says "could not initialize variable" and "the schema processor cannot find the element " blar blar. So I go ahead and check the blar blar WSDLs + XSDs. They all look perfect. So what the fuck is going on?

So I started all the "long troubleshooting process" which basically means multiple times of "Trial and Error". I removed this WSDL reference, removed that "Invoke Activity". For each change I will compile the whole thing. So I repeated this for every change. Just to narrow down where exactly has gone wrong. 

I did not perform all this in one day as I have other higher priority tasks to take care of. But on and off I will come back to this one as this problem slowly becomes a needle that keeps poking me every time the user asking for an enhancement for this particular SOA Composite. 

So today finally I managed to 'gotcha' the culprit that caused this bloody issue. It is due to the reference to a specific XSD in one of the WSDLs consumed by my SOA Composite is invalid. 

Lesson learnt. Next time the same issue comes to you, please don't "TRUST" the scac.log file. Just examine every WSDLs or XSDs in your Project. Some references may be invalid already.

Something funny has happened!

I am not sure why. Somehow the XML message that was passed to my request queue contains response tag which is not supposed to be attached to it.

What annoyed me was my WBI adapter for Kenan kept processing the same XML message over and over again! Thank to this and only i found out that there was one database connection remained opened thus causing more and more database sessions created!!! This could be a serious issue and luckily i tackled the problem before the system goes for production! However, i still need to find out why sometimes the XML message is redirected to the request queue even tough it has been processed by the adapter without encountering any Tuxedo connection error! The only clue i know now is this kind of 'problematic' XML message contains response tag! Could it be an error from the portal side? Could it be they acquired the processed XML message from response queue and then sent to request queue again? Well, it is doubtful.

Scratching head... continue tomorrow....