Web services market - wide open for open source

The current push by big players in the software world to create and/or control the emerging SOA or WebServices market via MetaStandard standardisation seems to be a tactic learned from the producers of consumer electronics. The consumer electronics markets are managed, technology is presented in an orderly, even evolutionary manner. Revolutionary innovation is kept from the masses until the existing markets are saturated, the innovation is then rolled out across the board. (Ok, it is not that bad, antitrust guidelines prevent it getting too much like collusion, see Carl Shapiro's standards paper for a nice discussion of the issues.)
It makes good sense, the market is huge; the pie is big enough for all the players and the customers like the illusion of choice. Sure there are exceptions once in a while but in the main, there is order.
I imagine it works like this, the major players share much of their R+D. They decide and standardize in advance, the technologies that they will support and license from each other. They publish the standards (to ensure the network effect), market, manufacture and release the products, continue R+D and repeat the process.
In consumer electronics, this model works because the capital costs required to develop, manufacture and distribute a new product are real barriers to entry. The R+D embodied in the standards is very hard to reproduce by some one outside the club.

In software, much of this is turned on it's head. The value of software is in the design. Good design takes experience, domain knowledge and skill. Once the design is embodied in code, the R+D job is mostly complete. The costs associated with manufacturing and distribution are close to zero. Marketing too, can be very cost effective with the web amplifying the word of mouth effect.

The emerging specifications for WebServices are very close to design, this is part of the nature of software and part of the nature of an interoperability specification. By standardisation, the "men in black" are creating a market but they are also designing a solution, one that is very easy to replicate.

Why then, are they taking control of the standardisation process?
Is is simply because they realize that the end game is domain knowledge and customisation or are they just ensuring that the market gains momentum fast, a rising tide to lift all boats (and hence their super tankers)?

Using Standards to create the future

Thinking about the place of open standards, how they are created and evolve, how they sometime lead, sometimes evolve with and sometimes follow a market, lead me to consortium.org. This is a great resource for fact and opinion on standards bodies and consortia.
The OMG consortium, one of the first, had a vision, produced a specification and worked through the evolution of the specification. In the beginning the specifications lead the market in a new direction, then the specifications followed the market; taking innovation (used as a differentiator between competing standard products) back into the standards. What followed was a period of 'evolution with the market', fixing problems, clarifying issues and needs etc. The next phase was extending the OMG model into new ground, up the stack, towards the applications. This required taking the lead on the market again, but this second phase was not as successful. The same "clear problem focus" was missing, the original vision was being diluted. In addition the process (and vendors embedded interests) were getting in the way, the result is captured in the view expressed in a survey on participating in standard development organisations(SDOs):

"Firstly, we (Sun) give very little consideration to SDOs, in large part because the rules are so arcane that we find that we get specifications with "maybe bits", rather than on/off bits. (possibly due to the fact that too many SDOs believe that a compromise where everyone is disenfranchised is a legitimate way to achieve "politically acceptable technical standardization"

The lesson may be, that a clear problem focus is key to any standards effort and that efforts to build past this initial problem, to fully capitalize on the first success, are best left until the next clear problem arises. There appears to be a time to "let go" that comes once a solution to the original problem has evolved. Hold on past this evolution stage and you smother the opportunity to build on the original success.

The latest edition of standard bodies (MetaStandards around Web Services and SOA) appear to be taking a very different approach. Rather than being focused on a common problem, they are focused on a common market. They are using the standards to give credibility and cohesion to the market and to build momentum and awareness about the technology. They are building the implementation and developing the standards at the same time. It is jumping straight to the evolution stage of the standard but without a clear statement of the problem; the "use cases" are being generated on the fly, in reaction to the markets response to the marketing messages. Maybe this is the perfect iterative design metaphor, produce a working trio of implementation, standards and marketing, present it to the market for review, evaluate the response and try again. What is clear is that the so-called "Men In Black" (Microsoft, IBM and BEA Systems) are really taking Alan Kays quote to heart

"The best way to predict the future is to invent it."

The capability and power of the "Men In Black" is not in question, if any group can create the future then this combination can. It will be interesting to see how far the solution goes past the interoperability play, how much control and scope do these standards want to have. Will they learn from the OMG case and stop once a clear problem and solution have evolved or is this new model of creating the future simply better and can accomplish more?

The fruits of the Open Source community may provide an alternative. Open source is firmly based on solving real problems. It may be in the interest of the market to take some control back, to support initiatives that have the problem solution rather than the market opportunity at the core. Our needs will be better met by a problem solved rather than the opportunity for more problems being created.
Is this just a classic tail of technology churn, all be it a very well executed one?

New Shape of Knowledge

More goodness from one of the co authors of the cluetrain, the web is changing how we think:

"That's because we've thought of our minds as containers. But the Web is made of links - pages pointing outside of themselves to other pages - each a little act of generosity."

It is an embodiment of how we learn by talking with others, exploring ideas and concepts, the notion of knowledge as a conversation that is now captured in the web in an open manner is very compelling, this blog entry and blog is worth a read

"Conversation is a paradox because it iterates difference on a common ground. That a paradox happens every day is a miracle."

The insight about the way the web changes the politics of knowledge by severing the links between knowledge, organisation of knowledge and ownership is useful. That we can still make sense of all this knowledge is the real miracle.

Grady Booch on Technology Churn

From an MSDN chat with Grady Booch some straight forward good advice

Technology churn is always a challenge.... ...I would therefore offer the general guidance to those teams to focus on some of the best development practices, architecture first, the use of modeling, controlling your change management; these are fundamentals that will prepare you to absorb technological change, which you'll have to do over time.

Then a word on adoption in the context of emerging Web Services:

"I expect we will see organizations struggle to build their own kind of Web Services because they would initially choose to not build upon public services out there, unless they are provided by a platform member such as Microsoft. It simply has to do with trust, availability, security, and scalability. And so they will probably dabble by building their own services, and later build systems upon public services."

Just trust, availability, security, and scalability; these are so important, so real and the reason why software engineering as a profession has so far to go. Sure we need to innovate, but as consumers we need to take some control, it is fair enough to have to dabble, to experiment, but not all the time. To develop a full understanding of a system takes time, stability of purpose and context, in the constant churning world of Software we have neither. So ensure we have arcitecture and models and ensure we know what our system does and how we validate that it does it. That is, write real tests for it. Only then are we in a position to evaluate the next wave that churns our way and empirically decide whether it is of benefit.

Utility solutions and the IT Hierarchy

Again a lively debate follows from a publication by Carr implying that there will be no need for low level corporate IT; IT functions will be available as a utility service. While APSs, outsourcing, shared data centers etc., are all a reality today, can all IT eventually be served in this way?
I think the hierarchy approach is a useful construct to help answer this question. It causes us to look at the computing needs of an organization in an ordered fashion. Recent details of IT spending discussed in The End of Corporate IT? Not Quite - Computerworld allude to a the need for some separation, in Carr's argument; a move from the 'one size fits all' approach:

In many ways, basic IT infrastructure has indeed become a commodity that should be treated as a utility where cost reductions reign. However, lumping all IT investments into the commodity category is the critical oversight in Carr's argument.

The un-lumping model is then described:

Much like Maslow's Hierarchy of Needs in human development, the IT Hierarchy of Needs segments IT spending into four progressive levels.

The first level is basic IT infrastructure—the core foundation for corporate computing including servers, networking, storage, desktops, mobility and telecommunications.

The second level includes the tools to automate manual tasks and processes, streamline transactions and foster creativity and collaboration.

The third level includes all applications to support the collection, visualization and application of information to measure the business and drive improved performance.

The fourth and highest level is how a company uses its information to change the playing field by creating different relationships with suppliers, partners and customers, as well as applying competitive insight.

Looking through this IT Hierarchy lense, I think the limits of utility computing will be tied to the lower layers of the hierarchy and then coupled with the evolution of standardised application specific solutions. The space occupied by ASPs today (web hosting, payroll, CRM), will grow to encompass the tools/services that are of utility along and across industry sectors as best practice becomes apparent.
However the middle and higher order functions will remain allusive because of their specificity and their value.

Looking for cost reductions for the middle order functions may require a pooling of resource, an openness to sharing best practice and a resistance to technology churn. The use of information is the key, how it is obtained, maintained and shared can become common practice, a standard or an open solution for that particular community. The open innovation model, coupled with an open source approach at the solution level could provide a framework here.

The highest order functions will remain out of the realm of community and utility because they will be the main stay of competitive advantage and are too valuable to an individual organisation to share.

We know more than we can tell.

Jon's Radio references a telling phrase: "The Tacit Dimension of Tech Support, refers to The Tacit Dimension, a 1967 book by the scientist/philosopher Michael Polanyi. One of his touchstone phrases was: 'We know more than we can tell.'"

This is one reason why the agile 'workwith' is so important. Seeing, sharing, doing with another person, is the best way to learn. It takes time to assimilate in this way, but also some stability, in both context and purpose; elements common to a traditional apprenticeship.

The open source meritocracy model - move it to the systems/solutions level

Dave Thomas and Andy Hunt do a wonderful job of describing the opensource ecosystem, identifying key motivators and practices that make it work. The mix of capitalism and communism, through meritocracy and community, provides an interesting balance.
Adopting some of these practices in commercial product development makes good sense and has been successfully demonstrated by the agile community. Going further I would like to see the same open practices applied higher up the food chain, in systems or solutions development. The move up the food chain is already occurring with open source in the web and application development arena, projects like ruby on rails and spring are taking on more and more of the infrastructure tasks of the developer. More solution oriented projects are also gaining traction in the CRM space.

An open solution oriented approach need to be limited to open source. Building open solutions with commodity components is also a possibility. Once the key capabilities of the commodity are identified and well understood, opening up the ways in which these capabilities can be used and reused in different configurations and contexts could provided a valuable community resource. Taking from the cluetrain mantra

#39 The community of discourse is the market

the community can determine the net value of a commodity, the appropriate level of functionality and the appropriate life span. Systems can be build from software that is proven, by the market, to be fit for a purpose. The meritocracy would reveal its self as best practice, the community would facilitate the diffusion of knowledge to allow the replication and ongoing maintenance of open working systems. It could be a powerful force in the reduction of technology churn that plagues aspects of the application portfolio.

When piracy works ....

Using formal economic modeling, professors Pankaj Ghemawat and Ramon Casadesus-Masanell consider the competitive dynamics of the software wars between Microsoft and open source, an interesting side effect of piracy is observed

"We also look at the effect of piracy and ask whether piracy can ever be beneficial to Microsoft. This extension was motivated by analyzing data on a cross-section of countries on Linux penetration and piracy rates. We found that in countries where piracy is highest, Linux has the lowest penetration rate. The model shows that Microsoft can use piracy as an effective tool to price discriminate, and that piracy may even result in higher profits to Microsoft!"

It effectively reduces the cost to zero but increases the network effect. Understandable but surprising.

The need for proximity in building collective intuition

Yet another interesting and relevant working paper that looks at the factors, and in particular 'proximity', that effect the building of collective intuition in an organisation.

The purpose of this paper is to examine the conditions under which intuitive forms of reasoning emerge to accelerate complex problem solving in product innovation teams. The paper originates from an ethnographic field study of four product teams in two companies: a hardware and software project from both a U.S. and a Japanese computer firm. This inductive, theory-building study was designed to use multiple cases to examine closely the learning and problem-solving behavior of product innovation teams. A problem-solving lens offers an insightful way in which to view product innovation, offering rich descriptions of interpersonal communication, project coordination, and the associated context for project team member interactions (Brown & Eisenhardt, 1995). Nonetheless, the empirical research on product innovation as problem solving has sometimes neglected to consider the challenges faced on the human side of problem solving in product teams (e.g., motivating people, creating the conditions for cross-functional teams to work together) (Brown & Eisenhardt, 1995). This study provides a response to this gap by focusing on the work practices associated with product innovation teams (Brown & Duguid, 1991; Engestrom & Middleton, 1998).

Communities of practice, global idea exchange, Open innovation, together these all help to deal with the proximity issue using internet technologies, exchanges and forums. For something as broad as innovation however, face to face meetings may be a prerequisite, proximity cannot be modeled or synthesised. On the otherhand, for something more concrete and specific like the replication of an IS/IT systems in another context, real proximity may not be needed. The internet tools may be rich enough because they are closer to the medium of IS/IT.

Open innovation and Idea Exchange - how it works

Mohan Babu writes about the reality of open innovation in the Indian context with some concrete examples, the concept is simple:

The model behind such global exchanges is simple: Companies or R&D groups post their problems on online Idea Exchanges and invite solutions from a pre-qualified global pool of candidates or companies. In return for a verifiable solution, the solver gets a substantial monetary reward.

The opportunities for distributed participation are huge, but the reuse of this model in the broader IS community, based around solutions to well known problems provides another angle on innovation. Innovation through replication of best practice or evolution of best practice. It provides a means to allow tech laggards to benefit from each other by sharing their experiences of technology adoption. The experiences then mould the predominant usage patterns for the laggards. Some input form the early adaptors would be of value, it may even be worth paying for, but the real value would come in the shared solution; buying power, known issues, known limitations, complementary vertical processes and systems.

The role of the knowledge broker...

From The Theory and Practice of Knowledge Brokering in Canada

One of the most consistent messages from the national consultations was that people whose job description actually says 'knowledge broker' are rare and that the situation is not likely to change. That's why the foundation was told to shift its emphasis from the idea of the individual knowledge broker to the activity of brokering.

It is very much about doing rather than being. It is a very active role, some what of a viral catalyst.

A broker's main task is to bring people together; they are catalysts who, through diligent network-building and solid background, can create a mix of people and even organizations that will stimulate knowledge exchange, the development of new research and the interpretation and application of solutions.
Brokers search out knowledge, synthesize research and scan for best practices, useful experiences, and examples from outside their own organization. They may also act as advocates for the use of research-based evidence in decision-making and have a role in supporting and evaluating changes they have helped to put in place - although the literature only mentions a generic 'follow-up' role.

Making it happen is very much part of the role, it requires a finisher rather than a starter. Collaboration needs to be nurtured and supported; it must be followed-up with metrics and rewards, taking into account the tacit nature of the mutual benefits.

legal protection, possibly the true value of ASF for new projects

In an interview with the server side, Greg Stein Chairman of the Apache Software Foundation (ASF) talks about one of the key value propositions:

"..., but what they wanted was sort of the legal oversight, the legal protection that the Apache software foundation provides to its committers. So, at the ASF all the committers are shielded from like lawsuits and other types of claims against them by the ASF. The ASF will be the party in any potential lawsuit. "

The PMC "oversight" process ensures that there is open IP disclosure and no hidden legal rat holes. Of course there is the Apache brand, but that means more to the users of open source than to the producers. The incubator and community ensures that the projects have value in them selves, the ASF ensures that the license can be trusted. For corporate IT, this sort of trust is vital.

The Six Degrees World of Inventors

Sara Grant, HBS Working Knowledge, writes about her research : "'Our work and more recent work on knowledge diffusion demonstrates that knowledge flows along these collaborative relationships, even years after they were formed,' says Fleming. At the same time, the world of inventors 'is getting smaller,' he says, 'inventors are more connected to their colleagues in outside firms, and that knowledge is diffusing in both directions.'"

People. and men in particular, often touch base around work, what is going on now, how things were in the past and so on. It provides a common link or a fabric into which conversation can evolve. Technology, the internet, PC's etc are also part of the fabric, they provide another common touch point. The opportunity for cross pollination are immense, as each new industry sector gets to grasp with technology, new boundaries are explored and common ground uncovered. The "world of inventors" may be getting smaller but the scope of invention is broadening. I think the personal fabrication model has applicability for industry as sectors fabricate solutions for them selves in ways that were before unheard of.

Innovation will be about seeing what should be out of what can be and making it happen with what is. This will be a role not for a smaller group but for the community at large as the language of pervasive technology becomes more understood.

IS/IT practitioners are struggling with the concept of a common language but progress is being made. With consumer electronics we are a lot closer, plugging and playing, sharing and manipulating, downloading and feeding. The current or next generation of children that take things like this for granted (without need ing to know how it works :-)) will have a freedom to explore and innovate like never before.

Ward and pepper on the IS Capability

Here is a nice presentation of the IS capability theory. A good reference point. The definition of the capability:

Essentially represents organisations ability to "connect ....technology to its business performance" (Marchand et al, 2000)

provides a pithy synopsis.

While the fact that the capability must "transcend organisational boundaries" is clear, going forward, I think it must also transcend institutional boundaries. Looking outward for inspiration, example and know how; then building new partnership or community structures to share in, evolve and profit from the understanding.

Self-Actualization Trends in the Digital World

Self-Actualization Trends in the Digital World: " Assuming that most of us can rest assured that there's a roof over our heads and food on the table, the democratization of technology means that more and more people can have opportunities to author and create aspects of their lives in entirely new ways."

There is a new freedom once the basics are taken care of. This begs the question of how best to take care of the basics, but the answer is, I think, in the community. Where we have community source for software, we should have community tacit knowledge about how to use software. Often, technology, finds it's feet in ways that were not originally envisioned. Only through the trial and error process of engineering, use and reuse does the real capability emerge. A community infrastructure is the best way to quickly identify, nurture and evolve the capability of a new technology. If the technology satisfied one of the basic needs then the motivation is with us all to share in the community, we can then get back to the more important self-actualization.

For the business of course this means bringing more innovation to the market :-)

Maybe WS provides a common language for open innovation at the IS/IT level

One of the daddies of documenting what Open Source is, wrote this back in 2003:

Hacking and Refactoring: "we can imagine a hacker culture speaking a common tongue other than Unix and C (in the far past its common tongue was Lisp), and we can imagine an explicit ideology of open source developing within a cultural and technical context other than Unix (as indeed nearly happened several different times)."

Possibly today, that time is comming, the common language and context may be here for business heads to use. The future of hacking may be with the business bods, working with shared business concepts and using the tools of IS/IT to evolve open, shared, trusted working systems. The solution to the personal problem of one business head could easily emulate successful open source projects if the problem is wide spread, the technology easily accessible and the context easily adaptable.

The key enabler is a shared language, BPEL, ebXML, Web services, WSDL and the WS-* stack can provide the shared framework. Open Innovation can provide the strategic initiative and Open Source can provide the historical background from which to learn.

While hackery is still treated as a craft and will remain so in many domains, the new craft is business agility, way up the software stack and closer to the fully evolved practioner. Where the business process is a mundane, day to day or regulatory task, it makes sense to share the burden of keeping the process up to date using the open source, community paradigm.
A shared solution will induce better understanding, foster alliances and increase buying power. Technology has its place as a contributor to strategic advantage, but it also has its place as a commodity tool that serves a business function in an stable, evolving and proven fashion. Open initiatives are the path to facilitate this distinction and implement the latter.

Some nice ideas about innovation...

This presentation, though a bit bright :-) has some good content. The rate of technology adoption is a handy (slide 17) but the idea that innovation is networking is cool.

From technology package delivery
to
innovation package delivery

you can take on the whole process, not just the technology. First see the technology at work in context and then make it work for you through partnership, colaboration or brokering. The demise of the industrial research lab (IRL) is also interesting.

Real life practitioners of Open innovartion

This industry week article is proof of that the open model, finding existing proven innovation and bringing it to market through collaboration, can work!. They have developed an innovation strategy around what they call the Connect + Develop (C+D) initiative. Looking outwards to see relevant innovation, then connecting to collaborate on developing and profiting from the idea.

To accommodate P&G's accelerated innovation process, Cloyd emphasizes a fast cycle learning methodology to help P&G identify winners earlier and develop them at lower cost. 'Remember,' he adds, 'that the challenge is that most innovations fail!'

If the innovation is already partly proven, evolving it through incremental improvement or modifying it to a more appropriate context can make it a real success. Only a new pair of eyes can see these possibilities!

The case for openness is stated through:

• Not all the smart people work for you. There is a need to work with smart people both inside and outside the company.
  
  
• External ideas can help create value, but it takes internal R&D to claim a portion of that value for you.
  
  
• It is better to build a better business model than to get to market first.
  
• If you make the best use of internal and external ideas, you will win.
  
• Not only should you profit from others' use of your intellectual property, you should also buy others' IP whenever it advances your own business model.
  
• You should expand R&D's role to include not only knowledge generation, but knowledge brokering as well.
  

The last two points may be hard to quantify and implement, how much to pay for an idea? How to make the brokering of knowledge work?

They big lesson seems to be that the key to success is in the process employed to make the initiative work.

For those motivated to emulate P&G's lead in open innovation, Brez offers recommendations. "Start with a formal strategy that includes implementation and tactical plans. Comprehensive planning produces the greatest value. Put all of the elements together -- with senior management committed to the strategy, the implementation, the organization and the funding."

"fab labs" (either fabulous, or fabrication, as you wish).

Edge: PERSONAL FABRICATION: A TALK WITH NEIL GERSHENFELD: "In one of these labs in rural India they're working on technology for agriculture. Their livelihood depends on diesel engines, but they don't have a way to set the timing. The instrument used in your corner garage to do that costs too much, there is no supply chain to bring it to rural India, and it wouldn't work in the field anyway. So, they're working on a little microcontroller sensor device that can watch the flywheel going by and figure out when fuel is coming in. Another project aimed a $50 Webcam at a diffraction grating to do chemical spectroscopy in order to figure out when milk's going bad, when it's been diluted, and how the farmers should be fairly paid. Another fab lab is in the northeast of India, where one of the few jobs that women can do is Chikan embroidery. The patterns are limited by the need to stamp them with wooden blocks that are hard to make and modify; they're now using the lab to make 3D scans of old blocks and 3D machine new ones."

This is fantastic stuff, simple practical and open. The notion of personal fabrication is set for huge growth. Chatting recently with a colleague who predated punch cards he referred to his children who have no fear of technology and just get on with it, making things work together rather than figuring out why they work. As engineers we often need to fully understand how something 'hangs together' before we will trust it, but our inability to command or grasp the variety of fields that technology touches now becomes a limitation. We simply can't understand everything! But like our children we need to just get on with it, make things work and get to grips with the new literacy that is cheap open commodity hardware and software.

The work of the fab labs will have fabulous effects in developing countries who simply see possibilities and don't' fully need to understand the whole history or background design, just the principals. In stead of asking "How did that work", they will ask "How can I make it work for me".

Strategic Advantage through Capability building

We build on the insights of the dynamic-capabilities school of business strategy by extending it especially across enterprise boundaries The relative pace of capability building matters most. Companies that embrace their edges will develop their own capabilities much faster than those that simply defend and extend their core operations and core markets.

Looking at the edges, at the competition, at the way things are changing around you, can help focus the innovation effort. While your core competencies are crucial, evolving these competencies in the context of your changing market and with the help of partners is key to sustainability.

One of the biggest challenges that executives face is to know when and how to leap in capability innovation and when to move rapidly along a more incremental path.

This can be very relevant to technology; in some cases it may makes sense to replicate a system that is proven to give value at a fixed cost. Replicate the entire process if it works for someone else, partner to share the maintenanceand increase buying power.

It builds on the insight widely attributed to Bill Joy, one of the founders of Sun Microsystems, that "there are always more smart people outside your company than within it"

. When these smart people have a working solution that they are willing to share, it makes sense to put it to work for you!