The Content of Systems
December 31, 2017
What better way to emerge from a year of distractions than to tackle an impossible topic. Even if this attempt was only tangentially successful, it would feel like being a phoenix emerging from the ashes. Hence my choice for a signature graphic and hence my choice of a case study with which to illustrate, or at least gesture towards, what the content of systems might be.
My choice of case study is a mega software project within the Canadian Federal Government - one appropriately called the Phoenix Pay System. It may sound a little too provincial to be instructive beyond the insular world of Canada's capital, Ottawa, but it is loaded with lessons for everyone as it is steaming towards the unenviable title of worst software project ever. When we are building software, we usually have a vision of what failure would look like, and sometimes of what success would look like too. Few on this project would have foreseen billboards being erected publicly demanding that the system be fixed or masses of unpaid, or inaccurately-paid, public servants taking up placards and pitchforks in protest. Somewhat maliciously, I have been revelling in the spectacle. Somewhat depressingly, I have been ruminating on the fact that this debacle is an exemplar of how large software projects typically go and that the lessons from this disaster will be lost on everyone who really should be paying attention. Why these lessons will be lost can be traced back to two sources: the politicians and bureaucrats on the customer side of the equation are effectively incapable of learning from experience, and the systems integrators and consultancies responsible for delivering these systems do not want to learn them (failure, in the software business, is lucrative).
Setting aside that cheery thought, let's move on to what lessons we can unilaterally extract from this disaster, in particular on what this tells us about the role of information content in software systems. This particular example is useful in this regard in part because I know a little too much about it and about the problem area to which it sought to bring order and savings. The Phoenix Pay System was named after the mythical bird that cyclically emerges from the ashes of its own immolation. The reason the phoenix moniker was taken is that this project emerged from a predecessor project called the "Public Service Compensation System" (PSCS) that was steered off the road in the mid-1990s. Upon leaving the military in 1991, where I had been managing a major pay and personnel software project, I was counselled by an uncommonly wise executive with a major software integrator to join the PSCS project. His words still ring in my ears: "Everyone should be part of a major software project disaster, and as early in their career as possible." Strangely, I took that advice and served on the PSCS project during its brief ascent and precipitous descent (we actually initiated the shutdown of the project well in advance of its failure in the public sphere - illustrating a lesson that was then lost on the Phoenix project). I was also then implicated for almost 10 years in the court case that came out of that project, where the federal government, successfully it turned out, sought damages from the system integrator behind the project. So I can claim to have some insight into this general problem area and into this particular example.
One of the tasks for which I was responsible on the ill-fated PSCS project was to establish a full-text database that contained all of the contracts and collective agreements that would bear upon the business rules that an integrated compensation system would need to apply. In effect, we set out to understand the content that the system would need to "understand" and effectively work upon. Complicating matters, PSCS, like Phoenix after it, sought to introduce a single system that would support literally hundreds of different departments, agencies, and agreements. In analyzing the contents of all these agreements, we were acquainted with the bizarre capriciousness of the sublunary world of human negotiations. We joked on the project that there was a unique rule that would only apply to left-handed lighthouse keepers who, in addition to the present, had worked between 1983 and 1989 and this rule would come into effect in circumstances where there was light rain forecast within 100 km of any one of the lighthouses they had worked at in the preceding 18 months. We were joking, sort of. The silliness of this scenario was in fact an expression of frustration that the business rules we were supposed to implement were just this crazy. How is software, even infinitely effective software, supposed to deal with this substratum of contractual content? The answer is that it can't. Only fools or crooks would pretend otherwise, and remembering bureaucrats and software integrators we see that we have struck gold in finding both.
The debacle of the Phoenix Pay System illustrates this point perfectly. We start with a customer determined to find a mythical outcome by acquiring a magical system that would allow them to retire hundreds of small, and wilfully quirky (in line with their quirky contractual contexts), pay systems, and to reduce the number of "pay specialists" needed to navigate that strange world of capricious business rules. We then add a major system integrator and a major consultancy who were most eager to design such a magical system and then to write the business case showcasing the massive savings to be had by its implementation. Hugs and handshakes ensued.
The software solution was hammered out, as usual in these types of projects, through the extensive customization of an "off-the-shelf" software product (ignoring the ill-effects of the Barnaclization of Systems), and the resulting system was thrown into production so that the reductions in pay specialists could be realized in line with the business case. But then reality reared its often ugly head and the shortcomings with the system, and indeed the entire concept, exploded into view. Hugs and handshakes ensured once again, but this time only on the side of the system integrators and consultancies - who would now be paid hundreds of millions of dollars more to first fix and then, in all likelihood, replace the phoenix system. On the customer side, the ill-consequences of childish naïveté begin to be felt as bureaucrats scramble to reassemble the pay specialists that they had only recently released and politicians spin tales about how the next tsunami of spending will make everything well. Rest assured that the cycle will repeat, making the "phoenix" name painfully appropriate.
So what does all this tell us about the content of systems? As I have elaborated upon elsewhere, content is the substratum of communication, the physical representation of what we are saying - including what we have negotiated, and what we are designing, planning, and justifying. Within complex software systems, the content that is really important is not so much the "end user documentation and training materials", important as they may be, but rather the exposition of how the system will work. This exposition will ideally be done with a precision, formality, and completeness that will allow the "system" to operate in a variety of ways - including through different software implementations and, most importantly, through a complex interaction between people (who will be knowledgeable and held genuinely accountable) and automation that can actually handle, and evolve to efficiently expedite, situations of unlimited capriciousness and complexity.
Where content is understood as the lingua franca underpinning systems, and not merely its public face, we can start to envision, design, implement, and evolve genuinely sustainable systems - ones that work with and build upon, instead of wishing away, institutional and individual responsibility. Needless to say this would be unpopular and unwelcome as it would limit the ability of customers to eschew their accountability and for integrators and consultancies to convert the dead weight of ignorance into financial gold. Maybe this helps to explain why content, and content technologies, have historically been such a hard sell - promoting as they do a grounded approach that tries to end, instead of fuelling, the destructive cycle of the phoenix.
You can follow this conversation by subscribing to the comment feed for this post.