Managing high-value service artifacts

Organizations invest thousands, if not millions of euros developing processes and services that are deployed within service-oriented infrastructures yet mostly fail to have a coherent view of those assets, where they are deployed, who built them, etc.  I use the word "asset" very deliberately as, if similar sums were spent on capital equipment you can bet that there would be an asset register in place that recorded what the asset was, how much it cost, where it was being used, etc.  Yet for some strange reason our industry takes a more "laissez-faire" approach to the software assets created through investments of similar and often larger sums.  Is it because these assets are intangible?  Clearly not.  From the CIO and CTO down these assets are known. Further they are extremely concrete. They reside on physical systems and drive many of the key operational processes of the businesses they serve.  So why are they not as well managed as the average laptop PC costing a fraction of the price?

Maybe it's because practical mechanisms to manage such assets are not widely available nor well practiced.  In thinking about this problem and researching various approaches I came across the work of Thomas Erl, specifically his book on SOA Design Patterns (Prentice Hall Service-Oriented Computing Series from Thomas Erl) (ISBN 13: 978-0-13-613516-1) which contains a section describing Service Inventories. Summarizing this work is out of the scope of this entry but suffice to say I found the ideas contained within the enterprise service inventory and inventory implementation patterns very stimulating reading and would recommend adding it to your reading list.  It led me to conclude that these ideas fitted the notion or our metadata registry, Spaces, very well.  Spaces is designed to allow software assets to be classified in any number of dimensions.  Spaces includes an flexible organizational hierarchy that allows for enterprises and business units to be modeled and any number of projects to be associated with those organizational entities.  A classification dimension for "SOA Layer" to represent the conceptual purpose of a service or process asset is available out of the box.

The combination of service classification and organizational mapping allows us to provide a framework to manage and report on your enterprise service inventory.  All that was needed was a new UI plugin to be implemented to render the SOA Inventory summary based on the existing classifications.  You can see the results in the screen shot below:

This is a UI plugin for Adjoovo Spaces.  It represents the process and service assets that are stored in the Spaces registry by the two dimensions: "soa-layer" and "project-affiliation".  An asset can have a single "SOA layer" classification but have multiple "project affiliations".  The UI widget is designed to encourage the classification of assets by representing unclassified instances as "unknown" and rendering them in red (bottom row and far-right column).  In addition to being a quick visualization of your enterprise service inventory, this view is also interactive and navigable.  Clicking on any dimension (row, column or cell) will present a list of the assets in the selected classification. Clicking on the link takes you directly to the service or process asset summary where you can see all the collected knowledge the registry holds about that asset and, of course, enrich that knowledge further using the built-in wiki, attaching links, etc.

So if you want to find all the services being used in the context of a particular project, just click the relevant column and you will see the whole list of classified assets.  Similarly by SOA layer or by combination of both of those dimensions.  Since the registry has knowledge of the relationship between projects and organizations or projects and developers, additional reports can be run e.g. display all the processes that a business partner organization is using, or show me all of the knowledge workers that work on Project Cirrus.

As discussed above, you can create any organizational / project structure you need to reflect your business.  The soa-layer classification is configurable through the Spaces registry dictionary (it is an enumerated list that I modified to follow the layers provided in Thomas Erl's book).  So if you want to start to add service inventory management to your projects, simply download Adjoovo Spaces, import project artifacts and in a few clicks you can have your own service inventory available for your team to share or even your boss or bosses boss to view. Overlaying these views with additional asset dimensions such as life-cycle stage and milestone achievement can provide a more fine-grained report for the CEO who wants to dig deeper.

To me, taking business view of these service-oriented initiatives is key to helping stakeholders (your CEO or business partners) track the benefits that their investments are making.  It is much easier to show progress using a simple dashboard of the asset inventory then walk them through dry project Gantt charts or process models.

Find out more about Adjoovo Spaces, here.

Spaces is a knowledge management product for software assets

A powerful combination of a metadata registry, collaboration framework, rich UI and broad range of "asset" discovery filters allows users to quickly document there enterprise applications in a centralized knowledge base.

This knowledge base can be used to immerse those new to your projects on the key assets, encourage re-use, reduce duplication and waste and reduce the risks to your projects when key staff leave.

in reference to: Collaborative Metadata Registry for SOA, REST and much more - Adjoovo (view on Google Sidewiki)

Nurturing the Roots of Service-Oriented Delivery

Many failed projects lamented by analysts and commentators do so because of the lack of effective metadata management. Read on to find out why this can happen to your project and simple steps to avoid these common mistakes.

Be it SOA or WOA, WS-* or REST, these strategies offer a way to vastly increase the agility of the information technology framework within an organization and yield many downstream benefits. Like all systems, natural or otherwise, Service Infrastructures are built on roots which if ignored will lead to a weak, under-nourished and poorly performing system. The important “roots” in question here is the metadata that underpins the service framework, that describes the messages and services that nourish your Enterprise.

Root or Branch?
When people think of service-orientation, the focus naturally falls to the services and processes that they wish to create, compose and automate. Our metaphor of roots being metadata might also describe the orchestrations and services as branches and leaves. This straightforward approach of focusing on the “big things” seems obvious yet risks ignoring the much larger mass, of albeit smaller entities, the metadata that supports them. For every service or process, exists many hundreds of other smaller components. Each piece of metadata a potential asset or a hindrance.

Figure 1 (left/above) depicts the landscape of service-oriented design and the relative size of the problem posed by the metadata layer. Much time is spent creating this root ball yet its existence largely hidden as it is often below the conscious-level of the knowledge workers that busily build out the service infrastructure. This is ironic as the cost of building up this mass of information and the implications it has on the larger system make it one of the most valuable resources the I.T. organization possesses (if used effectively). If ignored it can multiple the cost of service delivery, greatly extend the complexity of the service and orchestration layers, and drastically reduce the overall “agility” of the whole service infrastructure.

How these negative effects permeate through the service infrastructure is a simple as can be, yet happen slowly, quietly and unseen in the day to day activities within most infrastructure projects. When written down and exposed this causal-chain is obvious which makes it even more surprising that it precipitates the failure or undermines much value from countless project investments:

1. The direct cost of duplication (lack of re-use)
2. Fragmentation of architecture (leading to complexity)
3. Increased complexity slowing the development system (compromising agility)
4. Loss of agility leading to slower development cycles, increased costs and overall failure to deliver proposed benefits of service-orientation

Duplication
The problem starts innocuously enough... a developer either under pressure to meet a deadline or oblivious to the assets available around them creates a new message form to represent a business concept. This form is similar but critically is not the same as other forms of the concept and messages used elsewhere. Initially no symptoms are observed but the first impact to the service initiative has occurred. That is the increased costs as our developer spends time thinking and building something that already existed. Hard to measure maybe but it is a tangible cost and the tip of an ongoing series of costs that permeate from that one event. By developing that duplicate asset, the need to maintain it has also been created. Whereas the act of creation was a one-time event, the process of maintenance, additional documentation and complexity introduced lives on for many years.

Fragmentation and Complexity
The next symptom likely occurs at the process layers where various services and messages are being combined. Having similar but not identical forms of a particular data or service concept leads directly to more complexity in the form of transformations and other interface code. Message transformations make service orchestrations far more complex than they need be, magnify the amount of code involved, and therefore significantly increases complexity which in turn bloats the testing and maintenance headaches. An obvious effect of increased complexity is slower implementation and update cycles. However the the fragmentation that was introduced also directly erodes the agility of the whole infrastructure by making it harder to combine services together due to the need for complex handling of similar but different metadata.

In short, increased size, complexity and fragmentation of the metadata layer drives additional costs into the development, deployment and maintenance of service-oriented solutions and critically reduce the agility of the service and process layer. More metadata translates into a larger quantity of required code and other artifacts which in turn leads to higher development and maintenance costs. Multiple entity representations (or fragmentation) leads to higher process complexity. Fragmentation and higher complexity leads directly to less agility. Less agility means longer implementation cycles and undermines the ability of the business to respond to its changing requirements.

A Better Approach
The solution to these detrimental effects is simple to state, harder to achieve but starts with providing “extreme transparency” into the assets that make up the service infrastructure. The use of metadata registries, knowledge organization systems and collaborative tools, ideally integrated together, provide a firm foundation on top of which to build a methodology that optimizes the use of supporting metadata. The benefits of such tool come down to a single word: “visibility”. By allowing that earlier developer effective insight into what existed, the duplication and fragmentation most likely could have been avoided. The result being less code to maintain. The services derived from that now shared asset would be more easily combined, composed and re-used. The complexity of those process compositions would be less. The cost of development and maintenance also reduced.

However, no amount of “visibility” will solve the problem completely, there also needs to be a shared taxonomy by which assets are classified and a methodology to categorize new assets, and also pro-actively look for opportunities to reuse and consolidate. Building re-use into the delivery teams goals from the outset establishes awareness. Monitoring re-use by deep dependency analysis ensures that these goals are understood and progressed.

Practical Steps to Service-Oriented Agility:

1. 1.Create a centralized knowledge base of service and metadata assets.
   (You need a place where assets can be documented and easily located.)
• You can use disparate tools, open source, or look for a tailored solution
• A wiki and a process to encourage documentation and knowledge sharing is a good start
2. Adopt a taxonomy and classify at a granularity to suit your scale.
   (The process of discovery is aided by having mechanisms to “tag” or describe assets in a uniform way thereby aiding discovery and reuse)
• 1.Thomas Erls' “Inventory Patterns” provides a good starting point
• 2.UDEF and other semantic models are an emerging technology that may help
• 3.Look at one of the canonical message libraries as a potential base to build upon
3. Measure re-use. Consider specific goaling to encourage it.
   (The old project managers' adage “what gets measured gets done ...” comes in handy here)
   
   
4. Prune and consolidate often.
   (Duplication and fragmentation is a practical reality no matter how good your methodology so build in a process for dealing with it)
• Regular “asset” reviews at both the service and metadata layers

Knowledge Organization
There are multiple solutions already in place that assist an organization in maintaining a “map” of the operational service layer and managing the process of taking a service through the testing phase into production use. However these tools tend to ignore the issue of coherent metadata management during design and build, linking the service and process layer to the assets that support them and the process of collaborating to enrich the description of these assets. A home-grown combination of service registry, wiki, other group-ware may suffice but the more tightly integrated the better.

Taxonomy
Adopting a taxonomy helps an organization relate assets to business concepts and more easily share ideas between teams of knowledge workers. Many different approaches have been written about and can be used in individually or in combination. The aim being to provide a framework that allows each asset to be classified in order to promote its visibility when it is needed, i.e. the developer looking to wrap a legacy stock control system should easily be able to find existing assets associated with the “inventory” domain as it is understood by that specific organization. It could be as simple as “tagging” assets with well known keywords (e.g. inventory) or by using more sophisticated ontology approaches.

Measurement
Measurement is key to understanding the operational effectiveness of the methodology allowing you to learn and fine-tune. Having a system of deep-dependency analysis that can report on asset reuse will allow you to find areas of concern and highlight examples of success thereby allowing you to evolve your own best practice that fits your organization.
Finally, implement a process of frequent review of assets (by service domain for very large organizations). Delegate this process of review to your architects, analysts and senior developers and goal them to optimize the asset base by aggressive pruning and refactoring. You do this in the knowledge that assets that add mass to your metadata roots, add costs to your processes as well. There is no such thing as benign metadata, only metadata that is productive and metadata that hinders.

Further Reading
Interesting readings on related subjects ...

1. Adjoovo Spaces – a collaborative, service-oriented, metadata registry
2. Thomas Erls' – SOA Design Patterns (inventory patterns)
3. Canonical Message Libraries and Ontologies
• UBL – Universal Business Language
• Semantic Web – on ontologies
• UDEF – Universal Data Element Framework

Adjoovo Spaces Community release v1.1.2

Adjoovo is pleased to announce a new, free-to-use community release of its integrated registry and repository offering. Adjoovo Spaces is a web-based application built on open standards that evolves the traditional concept of a metadata registry to make it possible to extract and store detailed information and relationships from a variety of different technical artifacts in logical groups, or 'spaces'.

The Adjoovo Spaces registry enables developers, architects, business analysts, compliance groups and other knowledge workers to maximise their participation and productivity in development projects by allowing them to quickly search, explore, share and annotate the artifacts and important services that they expose. The rich context-sensitive views, tag clouds, summary displays, dependency graph analysis tools and built-in wiki capabilities collectively provide unparalleled degrees of visibility, all within a single runtime environment.

Spaces includes an extensible set of adapters to import and discover the applications, services, resources, message formats and potentially complex relationships exposed from a collection of artifacts and sources.

The latest release offers a revamped WADL tree-view for REST developers, new summary views and one-click web import capability along with several enhancements for developers and architects of SOA/ESB and REST-based solutions who are looking to understand, analyse, document and share information about complex integration projects containing WSDL/WADL/XSDs, BPEL processes, JavaSE, JavaEE, Spring, SCA and JBI composite applications.

Adjoovo is confident that Spaces can provide immediate insights and benefits to your development project at any stage of its lifecycle. To learn more, try the free-to-use community download, available today at:

http://adjoovo.com/cms/products/spaces

Adjoovo Spaces comes with a simple warning - continued use will fill the visibility void in your development project.

Been busy ...

and haven't blogged in a while due to working on the first release of our product. Well Adjoovo Spaces Community Release went out on Friday as planned. It is the first release of a project that has been incubating for a couple of months. The aim is to build a comprehensive metadata registry / knowledge organization system built around the needs of developers, architects and designers who need to work with complex development artifacts.

The objective is to provide a rich environment for folks to find our much more, in a shorter space of time about their existing and new projects. It is designed to work in "brown-field" as well as "green-field" projects. Brown-field scenarios are by far the most common scenario today (seldom do we get the opportunity to build something new in isolation). Building on-top or by the side of something that already exists, that probably isn't well documented, and for which you may not have access to source code seems to the norm. Well if that's a situation you often face, Adjoovo Spaces may be able to help ...

How does it help? Well in three different ways:

• it can inspect a wide range of technical artifacts and extract metadata from them
• it creates relationships between the metadata that can be navigated and reported on
• it allows you to "annotate" or "enrich" the metadata in a non destructive way

If its not obvious, here are some of the usecases that we support now and are improving all of the time:

allows quick discovery when starting a new project. By pointing spaces at your subversion tree or uploading the project archive files, Adjoovo Spaces will quickly inspect the contents, find all of the Java, WSDL, XSD, WASDL, Maven, Spring artifacts and provide a navigable web site that helps you understand the contents of the project.

non-destructive annotation allows developers and architects to enrich the information about services, resources even java classes using the integrated wiki capability. Annotations "attach" rich content to existing metadata and artifacts without modifying the original content.

dependency analysis is provided in multiple views that allows rapid navigation through metadata relationships allowing for a better understanding of the relationships or to answer specific questions such as what would be impacted if a specific change was made.

These are just a few examples of the use-cases that the current snapshot version allows. Many more are available and are being added with each release. You can download and use Adjoovo Spaces for free and find out more from http://adjoovo.com/cms/products/spaces

DTD parser solved...

After I wrote the previous entry, one of my ex-colleagues pinged my to point me in the direction of Oracles XML package. A quick look at the web site shows that it probably has everything I needed except one thing ... a compatible license. So I had to continue on to write my own which I have now completed!

The 0.1 version of it is running now in my development environment and I am feeling pretty okay about it. Not too bulky and quite fast. I must admit that until now I was blissfully unaware of the intracasies of DTD files. My parser is a schema metadata extraction tool, not a validating instance parser. I have used that fact to cut a few corners in my implementation.

Now of to look at Jersey ....

Wot no DTD parser ???

I cannot find a Java-based DTD parser and I cannot believe it! Of course there are parsers out there but none that seem to be able to deal with my usecase or the example DTDs that I need to process! I have tried a bunch:

• Castor
• Matra
• Wutka

But none of those worked for me on these DTD's. So off to write one I guess. Oh well I must be going and I might be a while ;) ....