Service Readiness Checklist (SaaS)

 Architecture & Scaling

• Is the service's architecture documented?
• Can the service tolerate machine failures whilst preserving SLO?
• Are the service components easily scalable?
• Can users consume unbounded resources through your service?

Code

• Where is the code hosted?
• If this is a our company's OSS (Open Student Society) project, does it have a CLA setup (Contributor License Agreement), is there an appropriate license and governance doc?

CI & Testing

• Do you have CI?  Do you use an existing "blessed" CI solution?
• Does the CI build the final production assets?  Where are the built assets stored?  Are they versioned? [We expect all assets run in production to be built by CI, not on a developers laptop.]

Release process

• Is the release process documented?  How regularly do you release?  [You should document the process of getting a new build of your service into production; putting these docs in the infrastructure configuration repository is a good idea.]
• Is there a staging environment?  A dev environment?
• Can updates to the service be rolled out without downtime?  Can releases be safely rolled back?

Config management

• Is the service config version controlled?  Is the config in the infrastructure configuration repo?
• Are resource requests and limits appropriately configured? [In very general, requests should be set to the 95th quantile of the usage over the last week.]
• Are your jobs restarted when config changes?
• If you have external dependencies, are they configured with Terraform?
• If the service required secrets, are they in an approved reliable secret store?

Security

• How is access to your new service controlled?
• Is TLS used over all untrusted networks?
• Is sensitive data encrypted at rest?

SLA/SLO

• Is there a published SLO for the service?
• How is the SLO monitored/calculated and who owns it?
• Is the SLO defensible?

Observability

• Does the service export metrics?
• Do the exported metrics allow for RED Method style analysis?  [The RED method: Key metrics for microservices architecture - Rate/Error/Duration]

Alerts

• Do you use Prometheus alerting?  Are the alerts version controlled?
• Are alerts routed to the correct please?
• Do you use SLO-based alerting?
• Have alerts been tested and validated in lower environments?

Dashboards

• Are dashboards version controlled?
• Are these RED Method-Style dashboards for the service?
• Are the dashboards available in our monitoring system?

Logs

• Does the service emit logs?
• Are all log error codes documented?
• Do your jobs emit credentials or secrets in their logs?

Back-up and recovery

External Services

• Are the external dependencies sufficiently monitored and is alerting set up?

On-call & Incident Response

• Do you have follow-the-sun on-call shifts setup?
• Is the on-call rotation adequately staffed?
• Are you using PagerDuty/OpsGenie/VictorOps/…?
• Have the individuals on-call received adequate training on how to handle the specific alerts?
• Does the service have an entry on the Cloud status page?
• Is there an escalations channel for the customer support enquiries?

If the service has been in production for more than one month:

• Have recent outages been followed up with a post mortem?
• Did the outages have documented recovery steps/guides?  If not, then they should be created
• Over the past month, has the service generated less than 2 pages per day on average?

State Management (If you're service uses a databases e.g. GCS, RDS etc)

• Are there (sufficiently frequent) backups?  Have you tested restore?
• Should the data in the database be exported to BigQuery/Redshift/… for BI?

Feedback

• Did you find this checklist useful?
• What do you think is missing from this checklist?

I have become a Manager

I have become an Engineering Operations Manager since early 2020 (of course with the help from my Special Mind Power Technique derived from Ancient Text of Confucianism) 

I am now managing 12 Senior Engineers, including 4 Leads, for ID&F products, hosted services & cloud services. 

Since then, I spent most of my time managing the people and the services. I have not got much time in developing something. Hence this space has been idle for some time. 

Today, I get to know there was one programming competition organized by Panasonic back in year 2007. Someone provided the winner list in Lowyat forum. Therefore I further studied the current status of the winners and came out with the table below. 

Rank	Name	College	Current Status	Current Company	Location	Remark
1st prize winner	Tan Aik Keong	MMU	Founder	Agmo Studio	MY
2nd prize winner	Chooi Kah Wai	APIIT	Technical Lead	PathDAO	MY
3rd prize winner	Simon Lim Hao Wooi	UTAR	Software Engineer	Canva	AU
Outstanding winner	Gan Eng Chin	UM	Senior Software Engineer	Automattic	MY	One of the early engineers in Experian CheetahMail
Outstanding winner	Fong Kha Chun	UM
Outstanding winner	Kwan Toh Choong	Curtin U of Tech	Digital Platform Design Manager	Shell	MY	11 years in Shell
Outstanding winner	Lee Chee Cheong	MMU				Last job as Technical Manager at iRadar Sdn Bhd
Outstanding winner	Tee Shu Hui	UTM	Senior Firmware Development Engineer	Micron Tech	SG
Merit Winners	Tan Kian Tat	MMU	Self-Employed			SAP Consultant since grad
Merit Winners	Lim Fang-Yin	UTAR				Active in Quora
Merit Winners	New Chin Jian	UM
Merit Winners	Choong You Qi	MMU	Staff Android Engineer	Setel	MY
Merit Winners	Chan Kin Meng	MMU	Founder CEO	Gameconomy	MY	Web3.0 Gaming
Merit Winners	Siew Lead Choon	UM				SAP?
Merit Winners	Yeoh Yan Pore	USM	GM	IHS Group
Merit Winners	Ernest Eg Ket Lung	USM
Merit Winners	Chan Chen Shyang	Curtin U of Tech	IP Consultant	Nokia	MY	Nokia till today
Merit Winners	Ahmad Irshad B. Abdul Hamid	UniTen	Software Engineer	Sage	MY
Merit Winners	Chua Fook Ching	MMU	Business & Integration Arch Specialist (SAP)	Accenture	MY
Merit Winners	Khor Yit Kean	TARC

Construct complex nested JSON file based on a list of JSON path

What happened was my QAs did not want to change their mind to prepare data in Excel, using 'path' concept. They're too used to that method already since the XML time. They'd like to continue the same. Row contains Path and Cell/Column contains value. Each Cell/Column represents one test case. Sounds good?

I googled and googled and tried so many solutions suggested by a lot of kind men in StackOverflow, none of the solutions really gave me what I wanted. No one really did this before that building a complex/complicated nested JSON file. I decided to write the algorithm myself. 

How does the so called complex/nested look like? It looks like below. And I had to build the JSON from this list of paths. 

$.JSONDoc.OC.AL.data_type
$.JSONDoc.OC.AL.value
$.JSONDoc.OC.SIG.data_type
$.JSONDoc.OC.SIG.value
$.JSONDoc.IN.MCPL.PRODTYPE.data_type
$.JSONDoc.IN.USER.COUNTRYCODE.data_type
$.JSONDoc.IN.USER.COUNTRYCODE.value
$.JSONDoc.IN.MCPL.LOCATION.COUNTRY.data_type
$.JSONDoc.IN.MCPL.LOCATION.COUNTRY.value
$.JSONDoc.IN.USER.FRSCR.data_type
$.JSONDoc.IN.USER.FRSCR.value
$.JSONDoc.IN.MCPL.ITEMS.value[0].QUANTITY.data_type
$.JSONDoc.IN.MCPL.ITEMS.value[0].QUANTITY.value
$.JSONDoc.IN.CLU.SPLT180.data_type
$.JSONDoc.IN.CLU.SPLT180.value
$.JSONDoc.OUT.TXN.FRC.data_type
$.JSONDoc.OUT.TXN.FRC.value[0]

The algorithm below works well so far for any JSON path that consists of JSON Array as well. I know it's not perfect but I really hope it's helpful to you guys who've bumped into the similar issue that I had. One thing to note is that the 'has' method from JSONObject does not really give accurate result if the JSONObject has multi-level of nested JSONObject. Therefore I also wrote one for myself. 

public static JSONObject createJSONObject(String[] keys, int index, String value, JSONArray jArray, JSONObject masterJObj) {
        if (index < keys.length) {
            String key = keys[index];
            String nextKey = keys[index + 1];
            if (key.contains("[")) {
                return createJSONObject(keys, index + 1, value, jArray, masterJObj);
            }
            if ((index < keys.length - 2) && nextKey.contains("[")) {
                JSONArray jsonArray = new JSONArray();
                JSONObject jobj = new JSONObject();
                Object obj2 = getObject(masterJObj, key);
                if (obj2 != null) {
                    if (obj2 instanceof JSONObject) {
                        jobj = (JSONObject) obj2;
                        if (jobj.has(nextKey)) {
                            jsonArray = jobj.getJSONArray(nextKey);
                        }
                    }
                }
                createJSONObject(keys, index + 1, value, jsonArray, masterJObj);
                jobj.put(nextKey, jsonArray);
                return jobj;
            } else {
                JSONObject jsonObject1 = null;
                if (jArray != null) {
                    if (!jArray.isEmpty()) {
                        for (int j = 0; j < jArray.length(); j++) {
                            jsonObject1 = jArray.getJSONObject(j);
                            jArray = null;
                            break;
                        }
                    } else {
                        JSONObject jsonObject2 = new JSONObject();
                        jsonObject1 = new JSONObject();
                        jsonObject1.put(key, jsonObject2);
                        jArray.put(jsonObject1);
                    }
                } else {
                    Object obj1 = getObject(masterJObj, key);
                    if (obj1 != null) {
                        jsonObject1 = (JSONObject) obj1;
                    }
                }
                if (jsonObject1 == null) {
                    jsonObject1 = new JSONObject();
                }
                if (index == keys.length - 2) {
                    JSONObject jsonObject2;
                    if (jsonObject1.has(key)) {
                        jsonObject2 = jsonObject1.getJSONObject(key);
                    } else {
                        jsonObject2 = new JSONObject();
                    }
                    if (nextKey.contains("[")) {
                        nextKey = nextKey.substring(0, nextKey.indexOf("["));
                        JSONArray jArray1 = new JSONArray();;
                        if (!jsonObject1.isEmpty() && jsonObject1.has(nextKey)) {
                            jArray1 = jsonObject1.getJSONArray(nextKey);
                        }
                        jArray1.put(value);
                        jsonObject1.put(nextKey, jArray1);
                    } else {
                        if (!jsonObject1.isEmpty()) {
                            if (jsonObject1.has(key)) {
                                JSONObject jsonObject3 = jsonObject1.getJSONObject(key);
                                jsonObject3.put(nextKey, value);
                            } else {
                                jsonObject1.put(nextKey, value);
                            }
                        } else {
                            jsonObject2.put(nextKey, value);
                            jsonObject1.put(key, jsonObject2);
                        }
                    }
                    return jsonObject1;
                } else {
                    JSONObject returnedJObj = createJSONObject(keys, index + 1, value, jArray, masterJObj);
                    if (returnedJObj != null) {
                        if (returnedJObj.has(nextKey)) {
                            jsonObject1 = returnedJObj;
                        } else {
                            if (jsonObject1.has(nextKey)) {
                                if (!returnedJObj.isEmpty()) {
                                    String nextNextKey = returnedJObj.keys().next();
                                    Object jObj3 = returnedJObj.get(nextNextKey);
                                    if (jObj3 instanceof JSONObject) {
                                        jsonObject1.getJSONObject(nextKey).put(nextNextKey, jObj3);
                                    }
                                }
                            } else {
                                jsonObject1.put(nextKey, returnedJObj);
                            }
                        }
                    }
                }
                return jsonObject1;
            }
        }
        return null;
    }

    public static Object getObject(Object object, String searchedKey) {

        if (object instanceof JSONObject) {

            JSONObject jsonObject = (JSONObject) object;

            Iterator itr = jsonObject.keys();

            while (itr.hasNext()) {

                String key = (String) itr.next();

                if (!searchedKey.equals(key)) {

                    Object obj = getObject(jsonObject.get(key), searchedKey);

                    if (obj != null) {

                        return obj;

                    }

                } else {

                    return jsonObject.get(key);

                }

            }

        } else if (object instanceof JSONArray) {

            JSONArray jsonArray = (JSONArray) object;

            return getObjectFromJSONArray(jsonArray, searchedKey);

        }

        return null;

    }

Xpath Evaluation On XML where there are multiple Namespace Declaration and there's XMLNS Declaration Without Prefix

The XML contains multiple or more than one XMLNS namespace declaration. One of them is with declaration without prefix (http://www.ilog.com/rules/param) which causes issues where xpath can't read the Xml element.

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ds="http://www.ilog.com/rules/DecisionService" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<DecisionServiceResponse >
<DecisionID>RLOS001PL62000205DRAFT_2019-07-05-13:35:57:48957</DecisionID>
<underwritingRequest xmlns="http://www.ilog.com/rules/param">
<ns0:underwritingApprovalRequest xmlns:ns0="http://www.tmbbank.com/enterprise/model">
<application xmlns="">
<projectCode/>

I have a Java class that implements javax.xml.namespace.NamespaceContext which later will be instantiated and set in the setNamespaceContext of the class javax.xml.xpath.XPath. Below showing how I override the 3 methods from the NamespaceContext interface. 

    /**
     * This method is called by XPath. It returns the default namespace, if the
     * prefix is null or "".
     *
     * @param prefix to search for
     * @return uri
     */
    public String getNamespaceURI(String prefix) {
        if (prefix == null || prefix.equals(XMLConstants.DEFAULT_NS_PREFIX)) {
            return prefix2Uri.get(DEFAULT_NS);
        } else {
            return prefix2Uri.get(prefix);
        }
    }
    //This method is not needed in this context, but can be implemented in a similar way.
    public String getPrefix(String namespaceURI) {
        return uri2Prefix.get(namespaceURI);
    }
    public Iterator<String> getPrefixes(String namespaceURI) {
        // Not implemented
        return null;
    } 

Apart from the above, there is another method as below: 

    private void putInCache(String prefix, String uri) {
            prefix2Uri.put(prefix, uri);
            uri2Prefix.put(uri, prefix);
    }

    private void storeAttribute(Attr attribute) {
        // examine the attributes in namespace xmlns
        if (attribute.getNamespaceURI() != null
                && attribute.getNamespaceURI().equals(XMLConstants.XMLNS_ATTRIBUTE_NS_URI)) {
            // Default namespace xmlns="uri goes here"
            if (attribute.getNodeName().equals(XMLConstants.XMLNS_ATTRIBUTE)) {
                putInCache(DEFAULT_NS, attribute.getNodeValue());
            } else {
                // Here are the defined prefixes stored
                putInCache(attribute.getLocalName(), attribute.getNodeValue());
            }
        }
    }

The above method would not work because are multiple xmlns declarations, and prefix2Uri is a HashMap which does not allow duplicates, hence the prefix DEFAULT will be set to the most latest xmlns="" declaration than the one highlighted in yellow as above. 

Let me print something below for your clarity: 

ns0, http://www.tmbbank.com/enterprise/model
DEFAULT, http://www.ilog.com/rules/param
DEFAULT, ""

As you can see, eventually the prefix2Uri HashMap contains only 2 entries: 

ns0, http://www.tmbbank.com/enterprise/model
DEFAULT, ""

In order to fix this, just need to add one line in the putInCache method that looks like below, highlighted in light blue: 

    private void putInCache(String prefix, String uri) {
        if (prefix2Uri.get(prefix) == null || prefix2Uri.get(prefix).equals("")) {
            prefix2Uri.put(prefix, uri);
            uri2Prefix.put(uri, prefix);
        }
    }

Besides that, the XPath query that is used to query the XML, also needs to add DEFAULT as the prefix. 

Initially, the method that builds the XPath is as below: 

    public static String getPath(Node n) {
        StringBuilder path = new StringBuilder();
        do {           
            path.insert(0, n.getNodeName());
            path.insert(0, "/");
        } while ((n = n.getParentNode()) != null && n.getNodeType() == Node.ELEMENT_NODE);
        return path.toString();
    }

Now, it should look like the below , added lines highlighted in purple:

    public static String getPath(Node n) {
        StringBuilder path = new StringBuilder();
        do {           
            if(n.getNamespaceURI() != null && !n.getNamespaceURI().equals("") && (n.getPrefix() == null || n.getPrefix().equals(""))){                path.insert(0, UniversalNamespaceCache.DEFAULT_NS + ":" + n.getNodeName());
            } else {
                path.insert(0, n.getNodeName());
            }
            path.insert(0, "/");
        } while ((n = n.getParentNode()) != null && n.getNodeType() == Node.ELEMENT_NODE);
        return path.toString();
    }

Apache Camel DSL Sample That Does A Few Things

Below piece does a few things. It's useful if you are using Camel DSL like me. The explanation is in the form of comment below. 

<!-- Bean instantiation of Sql Server driver which will be referenced by below DSL -->

    <bean id="sqlServerDS"

        class="org.springframework.jdbc.datasource.DriverManagerDataSource">

        <property name="driverClassName" value="${mssql.driverClassName}" />

        <property name="url" value="${mssql.url}" />

        <property name="username" value="${mssql.username}" />

        <property name="password" value="${mssql.password}" />

    </bean>

<!-- Setting up camel endpoint for XSLT which will be used to transform the received XML into another XML -->

<camel:endpoint id="someXSLT" uri="xslt:file://${xslt.location}/someXSLT.xslt"/>

<!-- Setting up File endpoint to output the generated output flat file -->

<camel:endpoint id="InformationFile" uri="file://?fileName=${output.location}/$simple{file:name}&amp;fileExist=Append"/>

<!-- Camel Route starts -->

<camel:route id="LalaLand">

                        <!-- Listening to someQueue which has been setup in ActiveMq -->

                        <camel:from uri="tcpActivemq:queue:someQueue"/>

                        <!-- transform the Xml picked up from someQueue using someXSLT -->

<camel:to ref="someXSLT"/>

                        <!-- define file name for the output file and stores it in Header -->

                        <camel:setHeader headerName="CamelFileName">

<camel:xpath resultType="java.lang.String">concat('somefile_', //code, '.txt')     </camel:xpath>

</camel:setHeader>

                        

                        <!-- logging -->

<camel:log message="SPLIT BEGINS" loggingLevel="DEBUG" logName="lalaland" />

                       <!-- Split the Xml transformed by someXSLT into lines -->

<camel:split>

<camel:xpath>//FileRecord</camel:xpath>

<camel:setHeader headerName="ctcRef">

<camel:xpath resultType="java.lang.String">//contactRef</camel:xpath>

</camel:setHeader>

                                <!- The payload will lose in the next section so needs to store the original message -->

<camel:setProperty propertyName="oriMessage">

<camel:simple>${body}</camel:simple>

</camel:setProperty>

                                <!-- if header variable ctcRef is not empty, Camel DSL will call 3 SQL to finally get the Date Of Birth, which is the requirement of this. Why 3 Sql? Because over here it does not support JOIN query -->

<camel:choice>

<camel:when>

<camel:simple>${header.ctcRef} != ''</camel:simple>

<camel:to uri="sql:SELECT ID FROM CONTACTS WHERE CONTACTREF = :#ctcRef;?dataSource=sqlServerDS" />

<camel:log message="body: ${body[0].get('ID')} " loggingLevel="DEBUG" logName="com.experian" />

<camel:setHeader headerName="contactId">

<camel:simple>${body[0].get('ID')}</camel:simple>

</camel:setHeader>

<camel:log message="contactId: ${header.contactId} " loggingLevel="DEBUG" logName="com.experian" />

<camel:to uri="sql:SELECT ACCOUNTID FROM ACCOUNTCONTACTS WHERE CONTACTTYPEID = 102 AND CONTACTID = :#contactId;?dataSource=sqlServerDS" />

<camel:setHeader headerName="accountId">

<camel:simple>${body[0].get('ACCOUNTID')}</camel:simple>

</camel:setHeader>

<camel:log message="accountId: ${header.accountId} " loggingLevel="DEBUG" logName="com.experian" />

<camel:to uri="sql:SELECT CONVERT(varchar,DATE_OF_BIRTH,23) AS DOB FROM CS_PERSON WHERE ACCOUNTS1 = :#accountId;?dataSource=sqlServerDS" />

                                                <!-- Store Date Of Birth in Header -->

<camel:setProperty propertyName="dob">

<camel:simple>${body[0].get('DOB')}</camel:simple>

</camel:setProperty>

<camel:log message="dob: ${header.dob} " loggingLevel="DEBUG" logName="com.experian" />

</camel:when>

</camel:choice>

                               <!-- Set the Body back with the original payload stored above -->

<camel:setBody>

<camel:simple>${property.oriMessage}</camel:simple>

</camel:setBody>

                                

                                <!-- Retrieves field value using xpath, add the dob from Header, and print in one line -->

<camel:transform>

<camel:xpath resultType="java.lang.String">concat(//accountRef, '|', //contactRef, '|', //title, '|', //givenName, '|', //middleName, '|', //familyName, '|', ${header.dob}, '&#xD;&#xA;')</camel:xpath>

</camel:transform>

                                

                               <!- The below will be executed after last line by checking based on CamelSplitComplete from Header -->

<camel:when>

<camel:simple>${header.CamelSplitComplete} == true</camel:simple>

                                        <!-- Below will add below wordings and number of records based on CamelSplitSize from Header to the output flat file -->

<camel:transform>

<camel:simple>adding new line here ${header.CamelSplitSize} </camel:simple>

</camel:transform>

</camel:when>

                                <!- content will be output to a physical file -->

<camel:to ref="someFile"/>

</camel:split>

</camel:route>