The importance of having a Integrated Dashboard for better IT management

 

Having an Integrated Dashboard is helpful in terms of IT Management. At least it is very very useful in terms of standalone program management according to my experience.

What kind of support we can get from the Integrated Dashboard?

1) Have a glance of how many "Interfaces" or "Services" we have. There will be a search feature to search for the Interface or Service  that we want to go to.

2) To configure Interfaces' or Services' variables at run time. For e.g. SQL, SQL Input Parameters, Database Connection, FTP Connection, Email Connection and etc.

3) To deploy latest code to the specific interface or service. Any Source Repository tool such as Mercurial, SVN, Github, and build such as Maven and Jenkin will be integrated to this Dashboard. The dashboard will provide a button to build the latest source code.

4) To provide scheduler function to schedule the launch the specific interface or service.

5) To provide server health (CPU, Memory) monitoring options using existing API such as JavaMelody or others. 

All the above are some important features that an Integrated Dashboard can have. Of course we can add more things such as monitoring , audit and etc. And of course we have to try to make sure each and every of our "interfaces" or "services" will be registered under this Dashboard. To build this Dashboard from scratch is going to take tremendous effort.

My thought on future IT Work Process

Nowadays in a lot of organizations, we have the "Application Team", "Middleware or SOA or ESB Team", we have the "Infrastucture Team", "DBA Team" and so on.

Imagine a project needs to involve so many different teams just to implement a solution. I would not say this working model is bad or not efficient or whatever negative term you can think of. Because so many IT solutions were implemented this way. The thing is, however, usually when working with different teams, communication becomes very very important and we've got to deal with a lot of human factors due to this. Human factor is more difficult to deal with than any technical problem.

Many times a team needs to depend on another team to complete the necessary task first. Such dependency creates a lot of troubles when the dependent team works in silo and does not respond in time using different kind of excuses. For e.g. if the Development team wants to increase server memory or to request to open certain port or things like that, in this case the Development team has to approach the Server team and Network team, and this may end up requires the Development team to fill in dozens of form, to attend to dozens of meeting to provide justification and etc. From the business perspective, this is actually not "acceptable", you know, every minute counts in the business world.

So I'm thinking, why not we have "development unit" only, or we remain only the "application team", then we remove the Middleware team, DBA team, Server team and Network team. Each application team will have their own Middleware, DBA, Server Admin and Network Admin.  And we'll set up the "Architect Team" to govern the Middleware, DBA, Server and Network standards. The Architect Team will consist of Enterprise Architect, Application Architect, Middleware Architect, DBA Architect, Server Admin Architect and Network Architect to coordinate with each other. They are the one who set the standards to the whole organization and each projects, they have to provide the architectural details for each project and review the architecture constantly at different project phases.

Project operation wise, architects should be not involved at all. Project manager will have full power on his/her team member consisting of BAs, Developers, System Admins, Network Admins, DBAs and etc. In that case we can remove the "form filling bureaucracy" and improve the project speed.

I'm not sure if any organization out there is already doing this way.

Old school Axis 1.0 way of consuming Web Service :(

Ok. Although nowdays we have a lot of advanced web service libraries such as CFX and etc, we still have to admit that sometimes we do bump into situation that we have no choice but to use the grandpa way of consuming web service! (well at least we are lucky not to use the great-grandpa way. LOL)

Yeah somehow I have to use the Axis 1.x library to generate Web Service Java Clients or Stub for me. Well that is not a big problem but but but in my case the Web Service Provider needs SOAP Header which contains all the username token information.

That's a real headache ya!

Well, the solution is quite easy anyway. It looks like below:

import org.apache.axis.client.Stub;
import org.apache.axis.message.SOAPHeaderElement;

SOAPHeaderElement security = new SOAPHeaderElement("http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd","Security")

SOAPHeaderElement usernameToken = new SOAPHeaderElement("http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd","UsernameToken");

SOAPHeaderElement user = new SOAPHeaderElement("http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd","Username", "xxx"); 

SOAPHeaderElement password = new SOAPHeaderElement("http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd","Password", "xxxyyy");
usernameToken.addChild(user);
usernameToken.addChild(password);
security.addChild(usernameToken);
            
Execute_ptt exec = ws.getexecute_pt(new URL("http://localhost:8001/soa-infra/services/SampleWebServiceProvider"));
((Stub)exec).setHeader(security);

Next, just make your web service call using the stub.

Fixed Length File Without Delimiter and BeanIO

What happened is I have this IBM Mainframe's CobolCopyBook and I need to read a number of fields from there.

For your information the CobolCopyBook is a FixedLength file which has NO Delimiter.

My problem is that in my environment I have to stick to the BeanIO framework for the parsing. That means I cannot use the normal BufferedReader + substring way which is considered NOT SO ELEGANT way to solve this problem.

Also this is the first time I am using BeanIO. I found that the existing FixedLength utilities in BeanIO, such as the FixedLengthReader expects each line has a Carriage Return (CR or \r) or Line Feed (LF or \n). But it also allows the user to specify a "delimiter" to determine when to assume a Line.

Unfortunately in my use case, as I mentioned earlier , my file does not have any delimiter. Basically it is one string of characters. Worst still is one of the field always has the CR or LF character in it! What will happen is the BeanIO's reader will consider whatever after the CR or LF character is a brand new Line but the fact it is NOT!

To summarize, I have 2 problems here.
1) No delimiter to determine a new line
2) One of the field contains Carriage Return and Line Feed 

I was cracking head for these two problems and my colleague came out with some brilliant ideas. They suggested me to use OuterBean which contains the Target Bean as a Segment. Segment as one annotation is another BeanIO feature that does all the grouping or nested bean stuff. This approach will solve the "No Delimiter to break line" issue.

Another colleague suggested to create a custom java.io.Reader that extends from java.io.Reader and implements my own "read()" method. Because in the end BeanIO is going to invoke the "read()" method from the Reader object that is passed into its internal classes. So inside this read() method I will check if the character is either a Carriage Return or a Line Feed then I will return a space means I am actually replacing \n and \r with space. This will solve my second problem where the Carriage Return or Line Feed will cause BeanReader to take the characters after it as a new line.

So now I managed to fix these two problems with the OuterBean approach and Customized java.io.Readar approach. However this is not so complete because I don't like the replacing carriage return and line feed character with space.

public class CustomerOuterUserBean {

    @Segment(name="customerUserBeans", collection=ArrayList.class, minOccurs=0, maxOccurs=-1, type=CustomerUserBean.class )

    private List customerUserBeans = new ArrayList();

public class NpiFilterReader extends FilterReader {

    public NpiFilterReader(Reader in) {
        super(in);
    }

    @Override
    public int read() throws IOException {
        int read = super.read();
        if (read == '\r' || read == '\n')
            return ' ';

        return read;
    }

So I continued the journey and in the end I found a very very simple solution for these two issues.

What I did is to create a custom RecordParserFactory which manipulates a custom RecordReader and I overwrite the read() method for the custom RecordReadear. The read() method in the custom RecordReader  looks like follows:

public class MyFixedLengthRecordReader extends FixedLengthReader

    public String read() throws IOException, RecordIOException {

        char[] buffer = new char[300];

        if(in.read(buffer, 0, 300) != -1){

            return new String(buffer);

        }      

        return null;

    }

Note that in my use case my line is fixed at 300 characters per line.

This is the most elegant yet simple solution! Thank Divine!

Secure Web Services using Oracle Enterprise Manager Fusion Middleware Control

Ok. Recently I have been working on Web Service Security.

My web services are running on Oracle Enterprise Manager Fusion Middleware Control.

The security part has the following break down:

1) Authentication
Whether the service consumer is a trusted one. I am using the username-token to achieve this.

2) Authorization
In this case the service consumer is trusted. However it may not have the right to invoke the particular web service operation.

3) Transport Layer Security
TLS or SSL is generally used for transport layer protection. The idea is to encrypt every data during the transmitting process. I am yet to implement this.And I am not sure if I really need to do this in my enterprise environment. The reason is our web services are in a secure network environment. Our service consumers are all internal applications. So I am not going to cover the detail for this part in this post.

Ok. What we need to know next is all the WS-* stuff. This could take some time for you to read. And along the reading process you might get even more confusion. So I will tell you what not to be confused here, as follows:

WS-Policy VS WS-Security. WS-Policy and WS-Security are two different entities. WS-Policy is a Language, whereas WS-Security is a Conceptual Framework. 

Next we are going into the technical detail. Diagrams are not available because I am lazy to remove all the sensitive company and application name:

Scenario:
Create two users to call a particular web service and to invoke different service operations respectively.

1) You need to be aware, that for Oracle Enterprise Manager Fusion Middleware Control, the "User and Group" setup are controlled by Oracle Weblogic. The whole Oracle Enterprise Manager itself is running on Oracle Weblogic. So we are going to access two different "Portals", one is the Oracle Weblogic Administrative Console and another one is the Oracle Enterprise Manager.

2) Now our first step is to create a new user. Go to Oracle Weblogic Administrative Console -> Security Realm -> User and Group. Create your user there. Very easy. Just fill in the blanks only.

3)  I am not going to cover the "Group" because I want to keep this as simple as possible. Everything will be just "User". The "Group" is useful if you want to group different users into the same group and then instead of granting permission to multiple users you just need to grant permission to one group.

4) Now you don't need the Oracle Weblogic Administrative Console anymore. Close it kill it leave it fuck it. Our next focus is to login to the Oracle Enterprise Manager for the remaining steps.

5) I assume you have deployed your web service (to be exact, it is SOA composite exposed as a web service) using the Oracle Enterprise Manager.

6) Right click on your "domain" that hosts your SOA composite,  you will see "Web Service" appearing in the context menu. Go there and look for your particular web service and click it to go to the detail page. 

7) Go to the "policies" tab. Here you can attach any policies you like to your web service. In our case we should attach "oracle/binding_permission_authorization_policy" for Web Service Endpoint Authorization.

8) At the same tab, we must also attach a username token policy for "Authentication". Just attach any policy with the name username token. But of course you have to do this carefully. 

9) Ok. we are done with the "Attaching Policy" now. Next, go to your "domain" again. Right click on your domain and you will see "application policies" and "application roles". Using the "application roles", you can create a "role" that can consists of multiple users or groups for easier management. In our case I will just skip the "application roles" to make things simple. So we will just go straight to the "application policies".

10) At the application policies, Create an Application Grant. choose the user that you set up at step 2 as your "Grantee".

11) At the same page again, there is a section for "permission". Put something as follows:
Permission Class - oracle.wsm.security.WSFunctionPermission
Resource Name - your web service NAMESPACE/service name. Remember, it's the namespace, not endpoint address.
Permission Actions - your web service operation

Done. Now the Authorization part is completed.

12)  It is testing time now. I suggest you to create another user who is not granted any permission to the particular web service.  In fact I successfully made two different users to access to the same one web service but each of them are only authorized to invoke one particular service operation.

13) Where do you pass the username and password? In your SOAP Header. Remember to include the below part in your SOAP message.