Net-Centricity, SOA, and Web 2.0

Net-centricity, Web 2.0, and SOA relationship

As I mentioned in a previous blog, I see Net-centricity as a "collaboration" umbrella concept where the mindset revolves around leveraging the power of the collective whether it is people or systems. I view SOA and Web 2.0 as the two sides to the net-centricity coin.

I remember the words used by one of my favorite high school teachers when trying to express to us students that although physics and chemistry are mostly treated as different disciplines they are actually very closely related. He used this definition that I found very useful, "Physics is the study of mass and energy with an emphasis on energy while chemistry is the study of mass and energy with an emphasis on mass."

Following this model I offer that SOA is the construction of systems and human interactions with an emphasis on systems while Web 2.0 is the construction of systems and human interactions with an emphasis on humans.

No definition is perfect but I think this explanation captures that they both center on interactions between entities each with its own bent but also involving a good amount of overlap. Mash-ups are a great example of the overlap between SOA and Web 2.0.         

Top Heavy SOA

I recently read a CIO Magazine article called, "SOA’s True Challenge—It Ain’t Technology." This is a good SOA article (it is surprising how few there are). I would be more specific than stating "SOA: It ain't technology" but more like "SOA: It ain't about products/vendors." I think this is what they mean. The article leans heavy on architecture which is right-on but designing SOA without knowledge of technology is like designing a skyscraper without knowledge of materials. I say this because I'm afraid those building SOA will downplay knowledge of technology, fail to bring in the right skills, and then build a very poor SOA architecture because "hey, it's all about the business right, we don't need those technology people."

That said, I think it does capture the need for bigger picture business alignment (i.e., the consumer design piece) but also gives some credence, although not enough in my opinion, to the transformation of select current systems functionality into leverage-able services (i.e., the provider design piece). These need to be done in parallel with the latter delivering value in the very short term. From all I've seen in the past 8 years of doing SOA and EAI, leaders rarely have the inclination/endurance to sit through a 12 to 18 month design phase before even starting development and implementation that won't deliver value for another six months or more. I run into this issue with my current SOA projects where people are saying, "We don't have time for this fantasy [SOA] that may deliver value who knows when, we need to solve problems today."   

The article states "But it also introduces new challenges for CIOs—challenges that stretch IT's abilities in areas that have been chronically underdeveloped: process and architectural planning. Their staffs will also be stretched. In short, CIOs who expect that they can do SOA the same way they've always done technology implementations risk being blindsided."

You may already be tired of me harping on the importance of the bottom-up (i.e., provider side) aspect of SOA design and development. I hope not because I'm sure there is more to come. I'm worried that the industry is placing too much emphasis on the heavy top-down design effort. Over emphasizing it is a sure way to slow down SOA adoption and create a great deal of unnecessary pain. Organizational change is always been an inhibitor to new technology and it is no different for SOA. Don't get too caught up on top-down reengineering of business and/or IT or you risk the wrath of change management beast which has killed more IT efforts than bad technology and poor business alignment combined.

I'll say it again, start small and build momentum.

Grady Booch is Mistaken

I recently read an unfortunate interview with Grady Booch (the famed modeling language guru) entitle "Grady Booch | Avoid the ‘stupid’ SOA approach." In this interview Mr. Booch espouses a top-down approach to SOA and labels a bottom-up approach as "stupid."  This is a very surprising position for a technologist who has made his career in top-down systems design modeling. Not! What is surprising is that he seems unable to see SOA from other perspectives. Mr. Booch exposes his ignorance of SOA. This ignorance probably stems from his ivory tower position with IBM where he must have lost touch with those, like me, who build SOA systems.

To state that top-down is the only way to approach SOA and that bottom up is "stupid" is simplistic and shortsighted. More unfortunate is that it perpetuates the perception that SOA is a top heavy, drawn out process that doesn't deliver results for years and only after prodigious investment. This is counter to the desires of CIOs to solve the problems they face today. Too many people think of SOA only as an enterprise architecture play tied to the "big picture." SOA is best implemented at a systems level or a controlled system-of-systems level. Let it evolve into an enterprise architecture.

My experience is that SOA must be both a top-down and a bottom-up effort. Top-down to ensure it moves towards longer term objectives but bottom-up to solve today's problems and remain grounded to the systems that provide the services in the near term. A bottom-up approach enables a migration path from the existing systems to the objective systems. Few organizations have the luxury of scrapping all they have to start building a pure SOA.

He misses the critical division of labor between consumer and provider treating SOA systems just as any other traditional systems development effort rather than approaching SOA from a market driven perspective where exchanges can be created by organizations "exposing" valuable data and functionality to the marketplace. Think of the top-down approach as the service consumer perspective and the bottom-up approach as the service provider perspective. Using market vernacular, think of top-down as the customer perspective and bottom-up as the merchant perspective (notice I avoided the term "vendor" so not to confuse).

He also goes so far as to state that "The idea of services is not a means of abstraction. It is simply a mechanism for reaching into systems." I am stunned. I am hoping the interviewer just simply got this quote wrong. First of all, what does "the idea of services" mean. I can't remember the last time I implemented "the idea of services." I suppose you could implement a service without the web services means of abstraction - but you would be wrong! Creating a service is all about delivering leverage. How can you really deliver leverage without an abstraction layer? SOA is all about leverage. It is all about abstraction. It is far more than simply a mechanism for reaching into systems. We have been reaching into systems with non-abstracted, tightly-coupled mechanisms for years and where has that gotten us? Without the loosely-coupled abstraction, SOA would not be experiencing the hype it now enjoys. In fact, this blog wouldn't even exist (believe it or not).

I may be a little hard on him but he should realize that his cavalier remarks make it more difficult for us to make SOA success a wide spread reality.

As far as his "hype" comment goes; name a significant technology that didn't go through the hype stage. Hype is a natural reaction to something with great potential. Hype is simply the time lag between our imaginings and the world's realities. SOA will catch up to the hype regardless of Mr. Booch's careless comments.

I just ran spell-check on this blog entry and the dictionary wanted to replace Booch with Botch. It also wanted to change SOA to soar. Coincidence, I think not.

SOA and Modularity

I was having an interesting discussion with Nick, a good friend of mine and fellow SOA professional (Nick's Blog) and he provided me the best analogy I've come across concerning SOA and modularity. Object oriented programming aficionados and old timer component proponents will often ho hum SOA modularity stating that modularity has been around since Moses wrote the Ten Commandments on two tablets. Of course the basic modularity concepts behind an object, component, or service remain intact but the devil is in the details so here is the analogy.

A traditional module is analogous to a jig saw puzzle piece. Modules can be assembled to deliver a product but the way they fit together is predetermined and static. This can be very useful. For example, if you want to deconstruct a product into modules, to facilitate transport (as in the Moses example), and then reassemble it at its destination ensuring it is put together exactly as it was before transport. However, jigsaw puzzle pieces are a great example of "tightly coupled" interfaces and they do not facilitate reuse.

 

A tangram puzzle (Tangram Definition-Wikipedia) is analogous to a SOA service (i.e.,module). Tangram pieces are "loosely coupled" and provide the flexibility to create a wide variety of products using the same pieces. This is very illustrative of composing SOA services to serve multiple business processes. Abstraction utilizing industry standards (i.e., ReST and Web Services) delivers the ability to create interface specifications that will allow for business process flexibility. Additionally, these interfaces are discoverable. This means that anyone wanting to use the "piece" can find it and understand how to use it. Abstraction and discover-ability are fundamentally what SOA adds to the age old concept of modularity. These are also the primary reason for the current buzz and investment around SOA. 

Think of jigsaw as tightly-coupled and inflexible. The puzzle is built to deliver one picture just as many traditional applications (even modular ones) were built to satisfy one business process. The tangram puzzle or service "modules" are constructed with loosely-coupled interfaces to allow for business process flexibility and use in multiple business processes. 

Thanks Nick. This analogy has long legs.

The Truth About SOA

I read a CIO article called "The Truth About SOA" by Christopher Koch. Overall it is a good article but just as you would expect from anyone who claims to know the truth about SOA, there are many debatable points. He openly states that SOA is far from a proven concept then goes on to say that a significant number of companies have two years experience experience with SOA and that large companies have had success. He is right, many companies have had success. Doesn't this mean that SOA is a proven concept? I think it is. Ask Amazon and eBay who's business model pretty much is founded on SOA concepts. However, just because SOA is proven doesn't mean that it can be used by any organization to do anything. A hammer is proven to drive nails into lumber. Using only a hammer to build an entire house may not be proven.

A fundamental underpinning of his argument seems to be that SOA requires a large, top down, enterprise architecture approach to building cross-organizational systems. Admittedly, that is one way to do it, however, it probably is not the best way. I don't believe that you can design and build a portfolio of services. You design and build services. You grow a portfolio of services. I espouse the market approach to SOA. Who sits down to design and build a market?

If you set out to re-design your companies business processes as part of your SOA effort then pack your bags for the train to failure. You will never get past go. In my experience any significant change is driven from the bottom-up (with senior level support) and momentum dictates success. Use SOA to create more flexible and open systems to allow the organization to evolve its business processes. Don't set out to transform your business. Enable transformation to occur by solving real problems with an eye towards creating value not only for your primary "customer" systems but to the larger organization. The old adage, "Think globally, act locally" is key to SOA effectiveness. SOA efforts should be held to no more than 180 day delivery cycles.

SOA is best accomplished by examining your current and future systems interoperability needs against your current interoperability capability (including maintenance costs). If there is no mismatch, then don't do SOA. Well, it isn't that simple but it certainly isn't as hard as transforming your organization. You should also look at the need for distributability of systems and your requirement for more modular systems that need to evolve quickly and maybe even in unanticipated directions. The power of SOA is that I can use it to build something to meet a need that I know I have (or better yet, that I know the "market" has) and yet build it in a manner that facilitates its utilization in ways I did not foresee or by users I would not otherwise be able to reach.

I agree with Mr. Koch that SOA requires more architecture and design rigor, however, the reason for that rigor is to facilitate distribution of effort. This means that because I have taken the time to design this service I now have more options in how I source its fulfillment whether I build, buy, or leverage. Therefore, I strongly disagree with his assertion that SOA requires a "centralized development methodology" and a centralized staff. We are building SOA design packages that we ship out to industry as part of an RFP. Industry can propose a solution that is built from scratch or one that leverages a current existing system or one that involves a packaged application. This is Net-Centric systems delivery.

As far as ROI goes. Of course it is difficult to calculate the ROI of transforming your business. If you set this as your goal then the risk of failure is extremely high and probably will have little to do with SOA technology and more to do with culture and politics. However, if you have a system that interfaces with multiple other systems and integration maintenance costs are killing you, then an ROI might be easier to calculate.

It is also interesting how people seem to treat SOA investment as a new problem. The dilemma has existed since the first time a human being built anything. Do I build it quick and cheap realizing it will only last for a short time or do I spend a little more time and money for something that will last longer and require less maintenance. Nothing new here.

I could go on and on here but I need to get back to building something.

Bottom line: Don't ask SOA to transform your business.            

Enable Net-Centric Interoperability Prototyping and Design Testing

While the SOA architecture greatly reduces the overall Net-Centric system complexity through the decomposition of system functions into modular services and the development of Community Service Interface Specifications, it does call for increased attention to the process of prototyping, testing, and validating system design decisions.  These efforts are best completed within a joint prototyping lab that is chartered to focus on systems integration through effective interoperability architecture and design. This is necessary because:

·                     To achieve overall system agility and adaptability, system functions that were once performed by a monolithic component are now implemented as a set of loosely coupled services interacting with one another.  Each individual service interface must effectively enable many-to-many interoperability. Some more difficult and challenging service interface specifications require prototyping to enhance the design prior to development. This will enhance the quality of the design, reduce the need for subsequent modification, and decrease the interoperability implementation schedule.

·                     Net-Centric systems shift focus from application-centric performance to the “composite” system behavior where capabilities on the Net are orchestrated together to deliver a mission capability. The prototyping environment enables enhancement of system interoperability design through the investigation of expected aggregated system behavior.  It also enables the testing of  how services behave when orchestrated in multiple configurations to meet several mission threads in a dynamic environment where mission needs may change rapidly.

·                     A prototyping environment with simulation capabilities enables experimentation and prediction of aggregate system performance in a variety of Net-Centric physical environments from fixed sites with significant and reliable bandwidth to edge computing with bandwidth constraints and occasional connectedness. This will enhance interoperability design and also provide valuable information on the computing hardware and software infrastructure required to achieve Net-Centric systems success throughout the GIG.

·                     The prototype environment also provides a means to demonstrate a Net-Centric system in a controlled environment. This capability is crucial for gaining senior leader champions of the Net-Centric SOA approach and in building momentum and enhancing the change management aspects of this critical technology transformation initiative. 

Although service interoperability testing is normally the job of the service consumer in a large and open marketplace, in smaller communities a leveraged CoI prototyping and testing environment can significantly advance SOA/Net-Centricity adoption. However, this capability is basically shared infrastructure and therefore faces all the challenges associated with funding and managing shared infrastructure that supports cross organization utilization.

Brokering Shared Services

Creating and exposing services on the Net is a critical and significant achievement but success does not end there. True Net-Centric success is measured in the numbers of service consumers and the leverage achieved. An important value-added activity for the CoI leadership involves brokering the leverage of services across the CoI participants and other potential organizations outside the CoI. This activity reaches beyond the design and development phases to the consumption of operational services available on the Net. Value-add activities includes supporting services discovery and efforts to facilitate use of existing services available on the Net versus buying or building new. Some of these are:

·         Ensure the services discovery infrastructure is in place and properly utilized so that organizations can readily find, understand, and effectively consume services on the Net.

·         The CoI leadership can facilitate cross-organization requirements for Net-Centric service enhancement. A service that is built for a primary user base may, with relatively minor adjustments, become more valuable to the Net and consumers that were unanticipated when the service was originally designed. The CoI can utilize their work with the DCGS Service Interface Specifications to recommend service changes to enhance leverage-ability in the Net-Centric Environment.

·         The CoI leadership can provide assistance to CoI participants by delivering education and outreach assistance, documentation, collaborative capabilities, and human expertise targeted at promoting understanding, awareness, and leverage of existing services. Changing organizational behaviors in this transformation initiative is crucial to the success of Net-Centricity. This involvement is necessary up front and may subside as the transformation matures and the culture of service discovery and utilization emerges.

·         Assist CoI component organizations in the development of cross organization security and data access policy and procedures. A more centralized group such as the CoI leadership can support inter-agency interoperability by establishing standard policy, shared processes, and best practices for cross-organization service access.

·         Assist CoI component organizations in building cross organization relationships and in establishing and maintaining memorandums of understanding (MOU) and memorandums of agreement (MOA).

·         The CoI leadership can represent the community as a whole in facilitating interoperability with other communities such as national agencies and coalition organizations in DoD's case.

Facilitating Community Service Interface Specifications

Promoting Web Services industry standards is a first step. Developing a set of standards-based Community Service Interface Specifications that describe how systems share functionality and data with each other and the Net-Centric GIG is the primary endeavor of the CoI. Accomplishing this involves establishing a multi-level CoI structure that develops and evolves the service interface specification enabled by a more centralized team that manages the CoI and the interoperability architecture artifacts. The centralized team is divided into the management team and the design team.

The CoI management team is responsible for determining the multi-level CoI organization and managing the productivity of CoI efforts. The multi-level CoI involves organizations and members that participate at varying levels including a core team, an extended team, and the general participation. The CoI management team must carefully determine the make-up of the CoI to ensure productivity and rapid returns. The core team must be chosen strategically. It should only consist of those organizations that are required to gain critical mass and no more. In industry service interface specifications are often established by a single organization like Amazon and E-bay who are large enough in themselves to drive a service interface specification into wide spread adoption. Although the DoD and other communities may not enjoy this luxury, by strategically connecting a few “eight hundred pound gorillas” a community can build a Community Service Interface Specification that can quickly deliver value not only to the gorillas but also to other organizations who are willing to adopt the specification. If the CoI leadership chooses the core team correctly then the demand for the services provided by the gorillas will drive specification adoption. The community, through the extended team and the general participants, can evolve the specification to meet the “20%” of specific needs. Assembling a core team that is too large can stall progress and lead to a specification that is too “specific” to have general applicability. Conversely, choosing a core team that is too small will inhibit adoption of the specification. The general rule is to assemble the “market makers” or those organizations that offer the most value to the “net” by exposing their capabilities as Net-Centric services.

The design teams leads the technical work and captures the CoI efforts into systems and service design artifacts. It subsequently collects input and manages configuration control on the design artifacts. These artifacts include the Community Service Interface Specifications themselves as well as the critical design documents (use cases, sequence diagrams, etc.) upon which they are based. The technical team can also manage interoperability compliance and exception handling of developed services through conformance with the Community Service Interface Specification.      

         

Promoting The Use of Open Industry Standards

The CoI leadership should be the champions of promoting the use of Web services open industry standards (XML, WSDL, UDDI, etc.) for community systems interoperability. The community CoI can make it clear that interface standards based interoperability is the best practice approach  over platform heterogeneity, point-to-point, or hub-an-spoke message broker based interoperability.

Although some of the newer more specialized standards are still maturing, the Web Services standards as a set have gained wide spread utilization throughout the world and the risk of their abandonment is minimal. SOA can be achieved through the use of other interoperability standards (e.g., CORBA IDL, JMF, XML RMI), but no alternative to Web services has the wide spread applicability or adoption for system to system interoperability.

It is important to understand the distinction between Web Services interoperability standards and community Community Interface Specifications. Web services standards are the general language constructs used to create community Community Interface Specifications. For example, the WSDL standard is used to describe the operations that a service can perform and the XML\XSD standard is used to describe the community data dictionary that services utilize. Creating a single common set of operations and data, instantiated in a single Community Interface Specification as the definitive vernacular for community interoperability, is ideal and can be achieved over time but it is very difficult to accomplish during the early stages of Net-Centricity. As history proves, getting a very large community of organizations to agree on a single data model is laborious and time consuming with a high risk of failure. This is where the use of Web Services standards delivers significant benefits.

Even if two organizations do not use the same Web Services based Interface Specification, because they are based upon the Web Services standards, interoperability can be achieved through Web Services translation using XML Schema Language for Translation (XSLT) or semantic mediation services. This translation can easily be automated and often does not require any coding development. Translation is unavoidable and is a healthy part of Net-Centric interoperability maturation. If all community Community participants agree to use Web services then this is a major step forward that will lead to faster interoperability results. A single common Community Interface Specification is the goal and the community must progress towards that end but agreeing on utilizing Web Services interoperability standards enables incremental progress as a function of “marketplace” evolution rather than working group decree.   

   

The CoI leadership can promote the use of Web services industry standards through policy, financial incentives, architecture guidance, implementation efforts, governance, and any other means by which they can exert influence. It is imperative that CoI leadership place the interface specification driven approach as the centerpiece of community interoperability.

Additionally, the community CoI and the component organizations should strategically and actively participate in industry standards bodies such as OASIS, OMG, and W3C. Active participation ensures that the community will influence the continuing evolution of the standards to meet their needs. Participation also will greatly assist the community in developing, propagating, and maintaining expertise in open industry standards that will translate into more effective systems interoperability implementation.

Using CoIs to Promote Net-Centricity and SOA

Net-Centric effectiveness depends not only on technology but also on process, people, and organization. The US Department of Defense is embracing communities of interest (CoI) as virtual organizations to facilitate collaboration between organizations in the march towards Net-Centricity. An effective way to achieve a scalable, multi-agency interoperable systems capability is through a well managed CoI. There are four main efforts the CoI leadership can undertake to spur transformation and achieve Net-Centric interoperability by making it easier for the community component organizations to build interoperable systems, minimize impact on their central operations, and maximizing leverage between organizations:

1. Promote the use of industry open standards for systems interoperability
2. Facilitate community service interface specifications
3. Enable Net-Centric interoperability prototyping
4. Broker shared services to maximize systems reuse\leverage