Why Did My System Fail
by Dan Elam, eVisory

“Why did my system fail?” More than one manager has wondered what went with wrong: with the system, the paybacks, or even their career. While it isn’t uncommon for computer implementations to be unsuccessful, the failure of an imaging can have wide spread implications.

Imaging and document management’s promise comes embedded with inherent risk: the ability to share information with a wide range of people in a mission critical environment virtually ensures that failures are widely known and painstakingly difficult to resolve. Add the fact that imaging systems often are the catalyst for organizational change and we find that these wonderful systems can become reengineering time bombs.

This month we examine a few systems that failed and how these organizations have evolved. Since no one likes to be reminded of failures, the organizations are described anonymously.

Speciality Insurance Company
To some, insurance applications have been a no-brainer when it comes to imaging. FileNet, IBM, and Sigma (now Eastman Software) built their businesses around the great benefits that imaging could bring to insurance companies by lowering costs and improving customer service. Expensive systems yielded impressive cost justification scenarios.

In the late 1980’s, one insurance company decided that it was time for them to purchase an imaging system and close the competitive gap with other companies using the system. The initial application was for underwriting. In underwriting, documents are sent in to a group of experts to review. These “underwriters” decide whether the company should accept the risk and help determine the price of the premiums. The idea was to get the system installed in underwriting and roll out it to other departments - including the one with the biggest payback potential: claims.

What Happened
The company quickly selected an industry leading firm to install the system and develop the custom workflow scripts required. The implementation was relatively simple and the system was operational a few weeks later.

The underwriters refused to use the system.

Technology - no matter how effective - rarely delivers when the users won’t use the system. Since this company was a niche market speciality insurer, the underwriting function was vital to the company’s overall health and ability to accept new customers. The underwriters insisted that the system was too slow. In this case, the system had a negative cost justification. That is, the system cost more to operate than the old paper-based process. Eventually management caved in, took out the system, and went back to processing paper manually. The system for processing claims was never installed.

Where are They Now?
Since most competitors had already installed imaging systems, the company was forced to work higher costs for both underwriting and claims than their technology-enlightened competitors. Some in the IS group have proposed imaging since the first failure, but these ideas have been quickly shot down.

Since the system was installed, the company has been sold and purchased twice by larger companies with more leverage. The company is once again considering imaging in order to remain competitive.

Lessons Learned
Users have rights too. In this case, the MIS staff did the operational analysis and examined the cost justification, but failed to adequately to consider the role of the user. In this case, the user community had unusual power within the organization, but the problems should have been able to be avoided.

When users first complained of the system being “slow” the MIS department concentrated on how to wring a few sub-seconds of performance out of the system. Had they asked the asked the users why the system was perceived as slow, it may have been a different result. After all, the system could retrieve images far quicker than the old manual system could find paper.

A more thorough needs analysis that involved the end users would have eliminated the problem. Resistance to change is nothing new. Getting it right in the

State Agency
In the mid-1980’s, one progressive state agency decided to tackle their problem of storing massive amounts of paper. They funded research into critical image enhancement techniques that were needed to ensure that the system could operate. After a long procurement process, they awarded the multi-million dollar contract to install the system and perform the backfile conversion.

The backfile conversion was a massive effort in its own right. It included converting millions of pages of paper and microfilm-based documents. In order to meet the schedule, the systems integrator proposed converting the images while the software was being developed and installed.

The state agency waited a few months for their system to developed. When the vendor demonstrated the prototype, it didn’t work. The agency patiently waited for the vendor to develop the system and install it at the agency.

During the acceptance test period, the system failed to work properly. The vendor worked hard to solve the problem, but the system took many months before it was considered fully operational. Despite serious misgivings about deploying the system, the agency agreed to allow the vendor to load the data and begin using the system. If the system managed to meet specifications, the vendor would receive their remaining payments and the acceptance would be complete.

What Happened
Only minutes after the system went “live” the agency began to discover a critical mistake. The index information used to point to the images did not match the actual images. With millions of images “lost” in the system, it was impossible to find the document by searching the database. (Some industry veterans called this example “WORN” - Write Once - Read Never). The only solution would be to review every single image in the database and reindex the information.

The state agency refused to pay the contractor. Eventually, the vendor filed a lawsuit against the state while the state began legal efforts on their own. The lawsuit went to trial. With industry experts on both sides of the fence, the vendor legal team convinced the judge that “all” in reference to the images to be indexed did not mean that 100% of the information needed to be correct. Without a specific percentage requirement, the judge ruled, the issue of the indexing was irrelevant. The agency was ordered to pay the vendor.

Today the once successful vendor is out of the imaging marketplace. The state agency still does not have an operational system. The case is currently on appeal.

Lessons Learned
In this case the vendor and the user both made mistakes. Imaging was relatively new at the time the system was designed and neither side had personnel with real experience in managing a project of this complexity.

The state agency trusted the vendor too much. There was very little oversight and warning signs of the vendor problems did not receive sufficient attention. The agency also did not structure the progress payments in a way to retain sufficient leverage over the vendor.

On the vendor side, the project management team was inexperienced. Combined with a turnover of several project managers during the life of the project, the vendor had little hope for real continuity. As a result, the vendor did not know where the problems were until too late.

The vendor should have paid much more attention to properly managing the project. Demonstrations should have been conducted in house before they were shown to the customer. When the system was implemented demo failures were not unheard of. (Today if a vendor fails the system demo you should not award the contract.) The vendor should also have performed better monitoring of the backfile conversion process. Like most projects, the customer stuck with the vendor through many trials waiting for the vendor to get the project right. Only when the backfile conversion proved to be inaccurate did the customer finally give up on the vendor.

Financial Services Firm
A few years ago a small retirement company began looking at imaging technology to use for a variety of applications. Examples included HR, financial transactions, COLD, and document management.

The company carefully examined several vendors and selected a short list of vendors to participate in their procurement process. One vendor had recently demonstrated new technology which seemed ideal to fit the business issues and provided unique integration to a key legacy application.

Despite the seeming fit, the financial services firm had to modify the procurement rules in order to find a way for the vendor to be able to get enough resources to properly prepare the proposal. The vendor site visit had to be rescheduled because the implementation subcontractor originally bid had to be replaced with another vendor.

After some difficult contract negotiations, an award was made to the vendor. The financial services firm was to be only the second installation for the company’s new software.

What Happened
Imaging system failures are almost never caused by the hardware or software. Components can keep a system from delivering peak performance, but they are rarely the cause for a system to actually fail.

In this firm’s case, the problems were both technical and managerial. Despite being a multi-billion dollar firm, the vendor relied on a small critical subcontractor to perform the integration and system installation. They had limited resources and no experience with the new software. The COLD component refused to integrate and would not work properly. The initial desktop software would lock the machines, stranding transactions that users were working on.

Busy with other projects, the MIS department waited for the vendor to try new releases of the software in hopes of correcting the problems. They disrupted staff as they allowed the vendor to test new configurations in the “production” environment.

Finally, after a year of trying, both sides agreed to abandon the project. The vendor returned most of the money (only some common hardware like PCs was deducted).

In other cases, this would have meant the end of imaging for a few years. For this company, however, the lessons they had learned gave them confidence to attempt the installation themselves. Using a small team of internal and external resources, they built an imaging system using a component approach. It was a high risk attempt, but the system works fine today and is fully integrated with the mission critical legacy applications, although it does not deliver the same functionality envisioned by the vendor’s original product.

Lessons Learned
The users were seduced by the marketing demo and functional specifications. They would have been better to wait a few months and examine some other installations before attempting the installation themselves. Time and time again, companies on the leading edge become the bleeding edge. This case was no exception.

Resources were a major issue throughout this project. The vendor did not have sufficient resources to do the integration and installation. Neither did their first choice. The second choice was a desperate attempt to stay in the bidding process. The third and final firm simple did not have enough experience to successfully implement the project.

This project also demonstrates that a big company doesn’t necessarily have lots of people for a project. In the long run, the vendor was forced to rely on small firms which would have never been qualified to bid the contract on their own.

While the project was considered a failure, it also demonstrated the importance of a rigorous test plan and a tight contract. The test plan was provided to the vendor before the installation began. Instead of shooting for a moving target, the vendor always knew what they had to do to pass the system acceptance. Contract references to the test plan allowed the customer to negotiate from a position of strength when it came time to default the vendor. The vendor did not protest seriously since everything had been documented carefully before the installation began.

The more benefits an imaging system can bring, the bigger the risk is for failure. Most imaging systems are a success, but even a few failures can scare management and end users. Avoiding failure means careful planning, good oversight, and a little luck.