SOA and COTS Orthogonality – a follow up and clarification – Original post 2007-07-08

Thanks to some very creative and insightful feedback on the original post from a few colleagues of mine, a bit of clarity and expansion on the dimensions is warranted.

For the sake of illustration in the article, I polarized the camps in the post as “SOA” and “COTS”. The intent of the SOA label was a best-of-breed, service-oriented connectivity model in an organization that is as likely as not to use a different vendor for every business core application. I believe that aspect was reasonably clear, but I blew it and mislabelled the “COTS” approach. Obviously, the label has a nod to IBM, Oracle and Microsoft in that they clearly offer integrated, custom, single-vendor stacks of software that work quite well emwithout/em needing to do very much integration. The list goes on with SAP, Sun and many more. The bias isn’t clear at this point, as an SOA organization can use these stacks quite well, and those stacks can be adopted completely and still integrate with yet other applications by SOA.

The labels really weren’t the point. They were intended to illustrate the bias in an organization towards a number of product specialists, being developers that will take a stack and be very capable at extending it and pushing it in all the required dimensions, vs. the generalists which in general don’t know the products to a great degree, but are very capable at putting very different and foreign systems together and making the integration the strength of their skill and of the enterprise, and can cobble custom code and applications together reasonably well.

Now, with a big nod to my colleagues, obviously, the vast majority of organizations have both types of developers (including ours) and most enterprises especially those of middle to large size will have experts of both types within their boundaries. It’s quite healthy indeed to have such a mix, and enables the best features of both approaches.

Flexibility and such multi-faceted capability matrices don’t come across very well at the higher levels of executive logic. Thus you wind up with a bit of a polarization. In one, the executive either has buzzworded themselves and you’re chasing SOA (likely with some preferred vendor), or you need to reduce and consolidate on a single vendor stack for efficiency and getting a unified set of developers and resources. (Work disclaimer, our executives are emnot/em as strongly afflicted by this condition as many, fortunately!)

That brings up the core discussion point. Is “best of breed” an approach to wander off and try to get the perfect vendor and thrash with continuous re-implementation of core business systems? Is “single stack” a simple procurement optimization that lets you have one company to pay the bills to and yell at when things go off the rails? Why does this either-or conceptualization exist?

Most often, when communicating the strategy of an IT organization, a “mission statement” of sorts is what is needed. That often grinds down into specific aspects of the implementation, and what shows up on the accounts payable. That simplification is problematic at best, for as noted earlier, the vast majority of teams is comprised of both aspects, and both types of developers. So then to the executive level, the teams are disorganized and inefficient, which is patently incorrect.

Really, good development teams need to get a job done, and they will use the tools they know, followed by the tools they are interested in, followed by the tools people they trust talk about, and finally, followed by the tools they are told to. Dire employment threats may elevate the use of the mandated tools higher in the stack, but that introduces a legislative inefficiency in the development process much as economists complain about legislative interference in the market creating inefficiencies. So the cases all boil back down to a question. What is the goal?

Getting the job done in the quickest way possible creates an inefficiency in that as changes are required, they usually require more effort than if some flexibility around that need was built into the solution in the first place. The trade was speed for flexibility. That’s most often the COTS approach. You get a “good enough” fit from pulling a pile of ready-to-rock code off the coat rack, and adding a nice shirt, tie, belt and decent shoes to go with it from your closet, and you’ll survive the presentation to the board just fine, it didn’t cost you a pile, and it can be replaced by next year’s style without too much pain. Unless the shirt and shoes start looking aged…/p

If you need the job to be done and each piece to be near the top of the game, say for competitive advantage, then you want a bit more out of the box. You want a top of the line suit, tailored to your specific needs, and not just anybody in a 42 Tall, you want it to fit your business down to the inseam. The tie is selected to go with the suit, and complement it precisely. The shirt and shoes and belt likewise. The goal is the ultimate ensemble to present to the world and take the headlines at E3. I’m stretching the metaphors again, but this gets to the central piece of the COTS vs. SOA, or specifically, COTS vs. Custom integration debate.

What’s the business advantage? If you aren’t making money off of your accounting department (and if you are, the SEC may wish a word), then your accounting system is a COTS piece with minimal integration into your other enterprise systems. If your accounting is part of an integrated ERP approach because your business is driven by operational efficiencies, and that defines at least a reasonable amount of your competitive advantage, then you want that system to fit precisely, and drive more benefits to the corporation directly. That’s the payback and that’s the decision fulcrum.

So I’m going to backtrack from my original assertion of bias on a team. It’s bias on a component, especially in the SOA world of today and tomorrow. In some cases you want your COTS/Stack integration team to ripple through and keep the overhead low on necessary but non-advantageous systems, and in others, your custom integration and customization teams will take the a product and adjust it to fit the company exactly, and bring competitive advantage and leverage to the table in a system that is critical to the nature and core business of the organization.

Just don’t do a custom job when the simple stack will suffice, and don’t settle for off-the-shelf when your core business depends on it.

SOA and COTS Orthogonality – original posting 2007-03-30

I’ve been diving much more deeply into SOA and moving an enterprise towards SOA. In that process I’ve been considering an idea that seems to have a certain truism to it.

An Enterprise strategy that relies on moving heavily to using COTS (Commercial, off-the-shelf) products is not the same enterprise strategy that would fully embrace an SOA (Service Oriented Architecture) strategy. As a great number of companies claim to embrace both in their IT organizations, let me more fully explain what I mean.

The essence is in what ability your seek to enable to the greater degree in your organization. SOA and COTS have biases and while not perhaps completely orthogonal, you certainly can’t take full advantage of both simultaneously without having a rather large conceptual dividing line in the team doing the work.

COTS strategy embraces a stack in most cases, or out-of-the-box-instant-integration. Systems are configured by default to work with each other in certain dimensions, and those dimensions are most often walled garden integration paths either not open to outside systems or not surfaced in a way that effectively enables interoperation with other stacks. The benefit and bias is one of getting up to a certain level in a very short period of time. Most people would think Microsoft with this strategy, and they are perhaps the best at enabling the integration in the stack, but Oracle, BEA, IBM, Sun and many, many other vendors to this in varying degrees, markets and technical layers.

The price most often is in the ability to swap out a piece of the stack if it doesn’t suit your business needs or if it needs to be customized beyond the ability of that integrated stack. At that point the benefit most often has to be unwound to a large degree and portions reconstructed to gain the original integrated functionality back as well as the desired extension. Tightly integrated systems in their very nature are less flexible outside of their designed purposes and capabilities. That is the trade-off in all software. Tight integration with speed and efficiency in exchange for a loss of flexibility and openness. If somebody says you can have both, check your wallet. They should be telling everyone how to change the art of the design trade-off in software engineering worldwide rather than selling you a product stack.

The other side or axis if you stick with my observation, is SOA. SOA is designed around a loose-coupling, highly flexible approach. Swapping a component out is much easier in this case, as the integration is abstracted and highly flexible in nature. The system can be tailored to bring many technology pieces together, and arrange and integrate them in many ways. The approach also seeks for the services to be somewhat granular. Discrete, and at any rate smaller than a COTS stack or large and highly capable product set. Vendors sell more of the enablers of this, and components in those COTS stacks also expose themselves and pieces as services.

The trade-off with SOA is that you don’t get a whizzy-integrated system out of the box, and it’s going to take some work and discipline to get the systems work decently well. The advantage you are taking the hit for is flexibility and agility and the ability to bring entirely new components, services and capabilities into the mix with very little pain and rework comparatively. You also, as a side-effect, are not nearly as beholden to a single vendor as you are if you wire everything into a single COTS stack, as the COTS integration points are not, as a rule, generic.

That doesn’t say why the two are orthogonal though. As I note, the COTS products can expose capabilities and features as services to be used in the stack. But by orthogonal, I was talking organizational strategy. One of the two needs to take precedence to really be able to get the proper leverage out of either solution.

The COTS solution requires the team to become expert at building and customizing that vendors products, and building on that stack. This isn’t a C# vs. Java argument, this is getting a team of developers expert with Exchange Server, Sharepoint, Commerce Server, BizTalk and other vendor systems. Expert in building solutions on top of those systems, and taking advantage of all the capabilities they create in each other. Each of these systems has a set of capabilities, and you learn to use and extend them and the stack.

The other side is a team that is expert at learning the boundaries of systems, often but not always smaller systems or sets of services, and integrating, sequencing and orchestrating them together. If the invoice system isn’t doing what the business needs, the approach looks at how to augment it with another system or replace it entirely without deconstructing the enterprise.

Where COTS would extend and customize the existing system to fix the issue, and thereby become expert in extending and partially writing/customizing the inventory system and generation process (something I would note you expressly bought the COTS piece so you would not have to do it), the SOA approach looks at getting the tool that does solve the needs for the invoicing working within the enterprise, but that process is around getting information in and out, and controlling the system. It does not get into the system itself.

The experienced ones will know full well the environment is not this cut and dry. I am not claiming that it is, but polarizing the backdrop to better see the trade-off. The trade-off is what kind of team and enterprise you want as the main driver and focus. Most decent sized enterprises will have both sets of people, as there is always a bit of COTS customization to do (it’s not that big a deficit to throw the product out) and there’s always a bit of SOA or Application Integration (The SAAS provider for CRM doesn’t talk our COTS stack) but the bias is a conscious decision that affects the team. Do you want the IT group to bring in solutions fast and cheap (COTS) by default and specialize in a specific set of vendor tools as their primary skill set, or do you want the IT group to work to put together new and legacy systems that fit your business and spend the time putting those pieces together in flexible ways, taking a bit more time and money to do so? There is a long list of advantages and disadvantages to both. I contest that you need to pick one of the two as your primary thrust (or neither if that’s your preference), but you can’t honestly pick both.

Currently playing in iTunes: On My Way by El Chupacabra

Eating of the Dogfood – original post 2006.09.19

There is much that has been said about “eating your own dog food“. Indeed, there is no better way to ensure that what you are building is of real use. It’s not always possible mind you, as is the case in our company, where we provide data and analysis software for the energy and engineering segments, and we aren’t in either of those businesses. As a result, our products keep our customers very close in design and implementation for our best success.

My head did a bit of a turn sideways to contemplate a very odd thing the other day. I’ve been working in C# on .NET 2.0 creating a relational database walker to do a transform/load into a custom schema and management system we need to work with. As I’m creating this system, I need to discover the primary keys in the Microsoft SQL Server 2005 tables I’m crawling. Oddly, the metadata doesn’t seem to contain any such information. There are some constraints, procedures, and the table and columns of course, but the primary key is absent. As the SQL Server Management Studio shows the primary key, I’m pretty sure I’ve done something wrong. So into Google and MSDN we go.

The part that made my brain do a serious double-take, and say “I emmust/em have read that wrong, was looking at the ADO.NET 2 documentation, specifically this page on MSDN. It states quite clearly that the SQL Server provider doesn’t do primary keys. But emOracle’s/em provider does. Apparently SQL Management Studio has a connection with the greater cosmos that allows it to magically divine primary keys from the fabric of space-time. After some more searching, I’ve found this gem from the Program Manager of SQL 2005/Whidbey. I appreciate it “bothering him”, but the management studio obviously wasn’t eating the dog food of the Whidbey release on the schema collections, and is instead using some alternate mysticism to achieve the desired results.

By the looks of the XML file Carl provides, it appears the magic is within the mystical (and decidedly specific) System Tables. This is obviously up there with a hack, as it’s a graft into the .NET config files of the workstation you install and run on, but given the relative simplicity of the configuration change, I’m rather baffled as to how this was missed in the product release. Again, from a dog food perspective this wouldn’t have been missed without disabling the entire SQL Server Management Studio from being useful at all if the only public remote interface was the one that the ADO.NET system provides. It would have improved the .NET system and the SQL Server Client metadata at a minimum.

So the short of it is, thanks to blogs and other “out-of-channel” communication, there is a very awkward work-around to what seems to be a very fundamental oversight. But again, if you want a better product, dig in an use what you build as much as possible. Adhere to your own published interfaces, and if you have an API, make sure your other products are using it, and not some other obscure method for integration.

Convention vs. Configuration – original post 2006.09.05

Unless you’ve stayed removed from the open Internet and general development news the past year, you’ve been inundated with articles about Ruby on Rails. There has been an immense level of hype, and more recently, established systems and frameworks have been almost counterattacking for lack of a better word against Rails.

Much like the polarized debates I attended at the DDJ Architecture and Design World conference in Chicago a few months ago, the screaming is always about volume, not content. At the conference, those who believe Model Driven Architectures and such are dead-end academic nonsense shout louder each year with more invective and never engaging in meaningful explorations. It’s not that they don’t believe any of the ideas, in fact, some are on both sides of the debate quite deeply, but the public discussions too often degenerate into a shouting match of x vs. y.

With Rails, it gives another view of the increase in capability of a developer, much like MDA allows in some scenarios. It’s no panacea, and like MDA, Rails has it’s solid successes, it’s marginal areas, and areas where it just isn’t up to the task compared to other frameworks today.

The point is the core idea that you will come across in article after article and book after book on Rails. Convention over Configuration. In ASP.NET and Java web applications, you have quite a serious amount of configuration in the development of the application. Some of the IDEs automate portions of this, but in every case, the power of the environment hits you square in the head when you’re using it, even when you don’t need the power. That’s where Rails changes the model, and finally, some of the Java frameworks (and possibly others) are taking notice. They give intelligent, most-often-used cases as defaults. Conventions of capability and use rather than an open box of parts. You can still, sometimes with a great deal of effort mind you, change that aspect and configure things specifically to your needs. That’s the right thing. But needing a simple CRUD web app for a database, you should not need to configure the three tiers, and distributed transactions and processing systems for a 3 day project.

I made the case that it is precisely because we are still writing in C# and Java and these decidedly C-like languages that so much work IS getting outsourced, and why we as IT professionals continue to fail in satisfying our users. We are using tools and languages that represent the same incremental abstraction from the processor that C and Fortran did in the 60’s. Our representations of components, modules, concepts and constructs are still linked directly to the idea of index variable loops and branching constructs of comparisons. It’s not that we need to do away with those. Some of the decision and looping logic is fundamental to software development. The point is we are continuously trying to achieve more sophisticated systems, richer interaction models and more scalable and distributed systems with the same tools we were writing command line utilities and our first compilers with. Seems rather inefficient. I regard it as a unique failing of our profession to date, and it’s why I will continue to explore technologies like Rails, like Ruby and MDA, and even go back to examine the more powerful technologies of LISP and other conceptual languages to find a way to get out of the rut we are trapped in.

It’s not about which framework is “best”. It’s about taking ideas to enable us to develop more sophisticated systems more easily. Components, modules, services, these are all some level of wrapping and abstraction, but even those are all built with these crude tools. In 30 years, we’ve basically achieved P-code and automated memory management. Not good enough.

Currently playing in iTunes: Pacific Coast Party by Smash Mouth

Be the Ronin with your Languages – Original post 2006.08.21

For anyone who has been in the industry more than a few years, and for everyone that really enjoys software and computing systems, odds are you’ve learned more than one language in your travels and studies. If not, I suggest you’ve done yourself a disservice.

The lesscode blog isn’t too active any more, but it has much sage advice, and a good post from last year that’s been referred to many times. “The Philosopher’s Song” is a ranging muse on the efficacy of languages, and the rage of Ruby vs. Python and the sameness of now to history.

At any rate, a Ronin is a samurai without a master. A warrior that seeks a purpose in some ways. When it comes to developing software, do not forge yourself into a single tool. Further, do not sit within the pure OO or pure procedural languages, and further don’t stay within a single family or style of languages. Try LISP, try Smalltalk, try Python, try Ruby, try C#, try Java. Extend into Javascript, and understand assembler on at least one architecture if you can. Touch Perl and a shell script, look into languages fringe and mainstream. But most of all, understand how different families and types of languages approach programming differently. LISP does things in such brevity and simplicity for certain problems that you will wonder why some problems are done in any other way. Ruby yields higher constructs that C# can seem downright tedious for many tasks. All languages have a purpose, and a strength, and a weakness.

Branch, enrich, learn, but do not stay beholden to one language in your skills. When all you have is a hammer….

Currently playing in iTunes: The Ride of the Rohirrim by Howard Shore

Joel Says it more Clearly Than I – original post 2006.08.03

I’m a Mac user. I work on Windows. I have a Linux server. I’ve administered every one of those platforms, plus Solaris, and programmed decent sized apps on all of them as well. I get asked a great deal “Why did you switch to Macintosh?”. Because I can concentrate on using my computer rather than fixing, updating or configuring it. Windows just doesn’t have access to the open source software in the way I like to get at it. Mac does (Fink anyone?). Linux was actually really cool and has a pile of configurable pieces, amazing arrays of doing anything you want, and the ability to swap just about anything out or tweak it just so. Problem is, I spent a lot of time making it “just so” and there wasn’t that much benefit over adapting to a mainstream OS.

The interesting thing is that there is a perception that the Mac has a better UI and is easier to use. It is if you have never used a computer before. If you have, you iwill/i have some unlearning to do. Once you get used to it, there are many things I do find more intuitive on the Mac, and in general I believe the quality of the software on the platform is higher. (I cite Omni on that count primarily, who make some absolutely top-notch software) Drag and drop is really pervasive in so many ways. Windows is coming along for sure, but I still find the Mac a much more creative place to reside on in my personal time, and even for some aspects of my job. I’m comfortable in Windows, and there are a lot of unique strengths and advantages there as well, different from the other two platforms, but personal taste rules personal dollars.

The real point though, is made by Joel Spolsky, which is what you expect. So a long-time Windows user is going to be put off and find the Mac awkward initially, as I did. And anyone that isn’t a Linux buff is going to be downright boggled by the do it yourself nature of the OS until they adapt. I’ve got a friend that has ranted eloquently on that a few times, but the refs are down on his blog at present, so no link. Sorry Norm! At any rate, check out Joel’s shot across the bow on this one: “Usability in One Easy Step (First Draft) – Joel on Software

Currently playing in iTunes: Code Monkey by Jonathan Coulton

A new beginning

I’m gradually getting things together for my own venture of Digital Katana Technologies Ltd. An Alberta corporation, consisting of myself for the technical software, networking and design activities independent of my other activities and work.

I’ll be moving the archives of my old Digital Katana blog archives over for posterity as they are relevant to the software industry in general, and we’ll carry on from there.