Warranty on proprietary Software is a Lie

The other day I was talking to a mid-level IT manager, who was in the process of procuring a large amount of expensive software. He explained that he was buying a COTS (Commercial Off The Shelf) solution that he knew was not particularly good, but he had, as he said, no choice.

What do you mean, no choice? Why not using one of the several excellent free software alternatives?

Apparently, his company had a policy against using unsupported software, free software, or freeware. The policy was established by senior management, he said, because they needed a vendor standing behind a solution, so there was someone to accept liability in case the software did not perform as advertised, and they needed to hold the vendor legally accountable.

Stop reading for a second, look away from your screen, and think about that... holding the vendor legally accountable because of warranty.

So these large corporations are going to hold software vendors legally accountable? Have you ever heard of a case where a company has plowed a ton of money into a software deployment, and has utterly failed to get the software to work as advertised? Have you ever heard of a case where a company has bought a product and then discovered that, in order to work, it needed to spend an additional pile of money on system upgrades, or work-arounds, to get what it expected? Have you ever heard of software that needed an extra army of con$ultants to deploy it? Have you ever heard of a case where a company has spent a bunch of money on a product only to find out that a feature they were promised hadn't made it into the release? Have you ever heard of a company that bought some software and found out later that it was buggy garbage that required constant patching to keep it from being open season for hackers? Of course you have!

Have you ever heard of a vendor being taken to court because their crappy software did not work as advertised? NO!

So, if I understand the logic of my friend's senior management, they refuse to allow use of free/open source/unsupported software, so that they can have all the benefits of something that they never have the guts to take advantage of. My friend is not alone, either - far from it - the great corporations of the world take a look at the warranty that comes with their software, and roll over on their backs with their soft underbellies exposed. On one hand, they pretend that the vendors are scared of their legal muscle, but on the other hand, they do not have the guts to challenge a click-wrap license agreement.

If these same visionary senior executives had the collective wisdom of a swarm of gnats, they would realize that the customer is always right because the customer holds the purse-strings that the vendors depend upon. If even 50% of the FORTUNE 500 announced that they were going to stop buying software that came with (effectively) warranty of no warranty the vendors would scramble all over themselves to negotiate special back-room deals before word spread. Many vendors survive on the 10% to 20% annual maintenance fee that they collect. Many customers feel they have to pay that annual tax because it is the only way that they even have a prayer of eventually getting something that actually works. I know I will probably be burned in effigy for putting that in writing, but it is the truth: Have you ever heard of a customer keeping a product on maintenance because they hope that by installing the next version they might get something that actually works? Have you ever seen an IT professional praying that this one will not suck?

The vendors are not going to give their stupid customers an even break. They do not deserve it because they play with the non-existing but cosy warranty cloud in their heads... What was it that W.C. Fields used to say:

Never give a sucker an even break.
      — W.C. Fields

CIOs/CTOs: If you are not allowing the use of free software because you want the legal fallback — you would better take advantage of it or just keep taking it on the chin. Get smart, guys... You do not have to be a coward anymore to feed your kids...

Throwing good Money after Bad

Two years ago one of my consulting clients called me to discuss an online banking application. His employer (a major bank) had purchased an online banking application from a third party. Basically, it was one of those turnkey solutions (hahaha! talk to folks that are in the business for more than a year or so) where all you need to do is install it, change the .gif logos and a bit of HTML, point it at your mainframe, and you were ready to offer E-banking to your customer.

They paid $500,000 for this software. Once they had it more or less working, they were ready to roll it out, and decided it was time to look at security. You can probably guess what happened next... There was none!

Wait, I am being ungenerous. There was a security. They recommended that IIS (Internet Information Services) be configured to use SSL (Secure sockets layer) which provided the security part of secure online banking.

Again, stand up walk around for a bit but keep in mind — this is about online BANKING not your kids homepage! An IIS with SSL will do yes?

I got into a couple of conference calls with the provider of the software and started asking probing questions about the security architecture of the product. At first, they simply cancelled the conference call because I had not signed their non-disclosure. We got all of that cleared away and then I got a chance to learn that the software basically exposed a stock NT4 Windows/IIS web server to the Internet. Putting additional software on the NT4 box would be an unsupported configuration and the software vendor would not answer support calls in case there were any questions about anything at all.

The NT4 box talked over Microsoft's incredibly secure DCOM remote procedure calls to a back end system that translated the DCOM requests into SQL calls that were sent to a mainframe.

Things in this configuration that had security problems:

• Windows NT4
• Microsoft IIS
• Microsoft DCOM
• SQL to the mainframe

The software provider's supported configuration was that the customer could put a firewall in front of the NT4/IIS machine and then everything behind it will be protected. Apparently they had never heard of IIS bugs. Actually, they had apparently never heard of computer security at all as well.

So, then what happened? Did my client sue the software provider for misrepresenting their product? They might have had a case, right, since security is a property of secure online banking and the software provider billed their product as secure. This is, after all, why CIOs (Chief Information Officers) and CTOs (Chief Technology Officers) do not allow free software or freeware - it is so they can sue chumps like these and get their money back. Of course that is not what happened.

My client tried to design a security system for the software provider. Suddenly I found myself in lengthy phone calls trying to explain to the software provider things they could do to help shore up their pathetic architecture. After a few days of this I stopped even recording my time spent — I could not bear to charge my customer for my time — they were already haemmoraging money on the project. After a week of trying to explain things to the software provider's chief architect I gave up and submitted a written opinion to my client that they should delay the roll-out and find an alternative E-banking solution. Instead, my client bought a firewall. At least they were smart enough not to just buy a stupid turbo stateful whatever firewall — they bought web firewall that could be programmed to permit or block specific URLs and which could be used as a layer of input validation in front of the NT4/IIS box. As it turns out, they have survived long enough with that configuration (more or less) and a lot of hard work, and now senior management thinks their security people are alarmists. Their credibility is shot, and anyone with a business case can now steamroll over sensible security recommendations just by jumping over the security team's branch of the org chart.

What did all this wind up costing? $70,000 for a web firewall plus a consultant (me!) plus auditors plus additional testing plus pen testers - on top of the $500,000. Plus lost time. If whoever had picked the product in the first place had simply done the right thing and committed seppuku, a lot of trouble would have been saved. This, I call throwing good money after bad. Security, when added as an afterthought can cost 20% to 50% more than doing it right the first place. In this case, it added nearly $150,000 to a project that was already late and over its $1 million budget. And, if the software provider had a clue, they could have come up with an add-on security module that made their product not suck as much - for only $250,000.

Do not laugh. It works for Microsoft...

The entire computing industry has become addicted

to throwing good money after bad to the tune of billions of dollars. Take Microsoft Windows, for example: to be remotely tough enough to withstand Internet use, it needs automated patching, antivirus, a firewall, etc.

Of course Microsoft provides mediocre built-in capabilities to meet those requirements, but only a complete doofus would field a Windows box on the Internet for a mission-critical system without spending at least as much again as Windows cost - in order to make it usable. Sure, it is a $100 operating system - but it comes with a $100 hidden liability cost, and a patching and maintenance cost that will just keep on costing you forever. If that is not stupid, can you tell me what stupid is?

I absolutely do not understand how it is that companies convince themselves that they are saving money by buying crapware and then paying a king's ransom to make it work. At one company where I used to work, our sales VP bought Siebel's COTS customer relationship management tool for about $250,000. Then it needed Oracle. And of course then it needed customization which meant it needed consultants. By the time the plug was finally pulled (the VP got to keep his job, by the way) almost $1 million had been poured into this software pit and nothing had ever actually worked. In retrospect, I probably could have found a kid out of school to write something in PHP and mysql and it would have cost under $200,000 per year for him to sit around and drink coffee and mess with it - and I would have been able to fire him if it did not work, which is ever so much more satisfying than just writing the consultants a big check, telling them to leave, and looking at a Sun server, a pile of CDROM distributions, a rack of manuals - a $1 million write-off.

But the madness does not actually end if the software is actually cajoled into working. Suppose we had gotten Siebel working - why then we would have had to maintain the customizations when each new version came out. Remember, all this stuff is customized. What is off the shelf about that? Perhaps I should not even enter into the topic of US Government software boondoggles and how those are done. A measly $1 million write off is not even on the radar screen for those guys.

The Disaster with the FBI's VCF Project

Take, for example, the FBI's Virtual Case File (VCF) project, which is finally acknowledged to be a failure after having had $300 million on-the-record dollars sunk into it. On-the-record dollars are just the payout to the con$ultants and software companies and do not take into account the salaries of all the project managers who helped oversee the disaster, as well as related projects that were on separate budgets. Basically, VCF was a GOTS (Government Off the Shelf Software) project - an application that was going to be integrated atop COTS products (i.e.: Oracle, etc). VCF was probably a legitimately large project, and I am sure data conversion represented a large amount of the unsuccessfully spent money, but all this system was is a large database that can store graphics, manage and control who updated what, generate linkages and visualize relationships between files, and store text, video, images, OCR, etc.

This is not rocket science, we are talking about. My guess is that if the smart guys who built Google decided to build something like Google Virtual Case File it would be up and running in about a year, for under $10 million. As far as I can tell, nobody has committed seppuku over the VCF disaster. Why not? This is an investment in technology that would break or make most businessess, it is crucial to FBI's continued operations — yet it is a $300 million 10-year long clusterf*ck. Now these clowns want more money so they can try again.

Finally...

In the last 20 years, the fad in business has been to stick with core competencies and to not do anything in-house that can not be done for twice the price as a one-time charge (plus 20%/year maintenance). What that means is you pay several times what the product is worth and, at a 20% maintenance rate, you buy it all over again every 5 years.

Being smart about technology should be a core competence for a CTO, right? Maybe we need to outsource the CTOs to a managed technology visionary company or something. Think of the money we would all save!