Kane Thompson Centre

For Studies in Information Systems and Information
Technology

Briefing Paper - 2

Author: Paul Beynon-Davies

Title: Information Systems Evaluation

Date of Publication: December
1998

Content

The importance of Information Systems (IS) evaluation to effective
IS management and development processes within organisations

The various types of IS evaluation

Relationship between the types of IS evaluation and organisational
learning in terms of IS and IT

Contact for further details:

Dr. Paul Beynon-Davies,
Reader,
School of
Computing,
University of Glamorgan,
Pontypridd
CF37 1DL

Tel: 01443 – 482243
Fax: 01443 – 482715

Email: [email-masked]

Not for Reproduction without the express permission of
the Author and the KTC

The Importance of IS Evaluation

In recent years a number of questions have been frequently
voiced by general management within organisations in relation to information
systems (IS) and information technology (IT):

Do we know how much is currently spent on IS and IT?

What value results from this spending?

How should IS/IT alternatives be justified/ prioritised/ financed?

Why do we continually experience cost and time overruns in relation to
IS/IT projects?

Why do IS budgets continue to rise while IT unit costs continue to fall?

How can we regain our belief in IS returns?

All such questions relate to the issue of evaluation. This
briefing paper focuses on the evaluation of information systems by organisations
and discusses the importance of this activity. We also highlight the different
forms of evaluation and indicate some of the problems in the way in which
organisations conduct evaluation.

A Model of IS Evaluation

Figure 1 presents a model of IS evaluation fitted to the
life-cycle of information systems development. This illustrates the importance
of evaluation to processes of organisational learning. Organisations need to
conduct evaluation to:

Assess and prioritise investment in IS and IT

Control IS costs

Determine the value arising from IS and IT

Determine changes needed to the organisation’s IS portfolio

Learn successful strategies for IS management and development

What are we evaluating

Conventional approaches to IS evaluation stress a product-based
focus - the evaluation of information systems or components. In terms of
information systems we can distinguish between three dimensions on which an IS
should ideally be assessed.

Functionality. Does the information system do what is required?

Usability. Is the information system useable by its intended
population?

Utility. Does the information deliver key business benefit for the
organisation?

Assessing the degree to which a system is functionally complete
and consistent is a classic concern of systems development.

Assessing the usability of systems has become important with
the continuing progress and use of graphical user interfaces and multi-media
interfaces.

Assessing the utility of IS is something which most
organisations conduct at the pre-implementation stage of a project but seldom
thereafter. However, this form of evaluation is becoming increasingly important
because of the greater pressure being placed on the IS function to account for
its activities.

Types of evaluation

Figure 1 makes a distinction between three types of IS
evaluation activity:

Strategic Evaluation. Sometimes referred to as pre-implementation
evaluation, this type of evaluation involves assessing or appraising an IS/IT
investment in terms of its potential for delivering benefit against estimated
costs.

Formative Evaluation. Formative evaluation involves assessing the
shape of an IS whilst in the development process itself. Formative evaluation
may be used to make crucial changes to the design of an information system or
to make critical decisions concerning the degree of project abandonment. Such
a form of evaluation is particularly relevant in the context of iterative
approaches to development such as rapid application development (RAD) (see KTC
briefing paper 1).

3. Summative Evaluation. This type of evaluation occurs
after an IS has been implemented. For this reason it is sometimes referred to as
post-implementation evaluation. Ideally, summative evaluation involves returning
to the costs and benefits established in strategic evaluation after a period of
use of the IS.

Costs and Benefits

IS evaluation may be defined as the attempt to assess the
success or failure of an information system and the associated process by which
the information system is developed and implemented. Although evidence suggests
that organisations do not conduct evaluation of their systems and projects very
effectively, the systematic evaluation of IS projects is critical to
improvements in both development and organisational success.

Evaluation is normally expressed in terms of costs and
benefits. IS costs concern the investment needed in an IS. IS benefits concern
the value that the organisation gains from having an IS. If benefits outweigh
costs then the project and system are regarded as successes. If costs outweigh
benefits then the system and project are classed as failures.

It is useful to make the distinction between two types of costs
and benefits associated with IS projects: tangible or visible costs/benefits and
intangible or invisible costs/benefits. Tangible costs or benefits are
frequently referred to as visible costs/benefits because they are reasonably
straightforward to measure. Intangible costs or benefits are frequently referred
to as invisible costs/benefits because most organisations experience difficulty
in assigning actual measurable quantities to such costs and benefits.

A typical list of cost-types associated with IS projects is
given below:

Hardware Costs: computers, printers, storage, accessories etc.

Software Costs: off-the-shelf packages, bespoke software,
development tools etc.

Installation Costs: data entry, data conversion etc.

Environmental costs: wiring, furniture, air-conditioning etc.

Running costs: electricity, communication costs etc.

Maintenance costs: on hardware and software.

Security costs: risk management and disaster recovery mechanisms.

Networking costs: network hardware, software and maintenance.

Training costs: frequently under-estimated.

Wider organisational costs: new salary structures, management etc.

The important point to make in relation to this list is that
the costs of an IS must be taken into account over its entire life, not solely
in terms of its development cost. Frequently organisation forget that the
initiation of an IS means a permanent commitment to costs in terms of continuing
resources being committed to the running and maintenance of the IS.

Below we list some of the classic benefits associated with
information systems:

Accuracy: increased accuracy of information

Quality: better quality information

Usability: more useful information

Flexibility: ability to use information more flexibly

User satisfaction: increased work satisfaction

Functionality: more effective working

Reliability: more reliable service

Utilisation: more information used

Relevance: more focused information

Productivity: increased levels of working

Security: more secure information

Profitability: more money

Speed: getting information quicker

Volume: more information

Information systems have frequently been initiated with the
objective of gaining tangible benefits such as reducing staff count or
increasing productivity. Intangible benefits are more generally associated with
issues of organisational efficiency. More recently people have started to argue
that intangible benefits such as increasing customer satisfaction or building
better links with suppliers have equal relevance in investment decisions.
Intangible benefits are more generally concerned with organisational
effectiveness.

The Model Explained

Most organisations conduct some form of strategic evaluation of
IS projects.

Strategic evaluation is an activity which attempts to establish
the balance of costs and benefits in terms of an intended IS project. It is
normally used to initiate a go/no-go decision in terms of a given project. It
may also be used to prioritise a number of IS investments. The most popular
techniques applied in this process are return on investment and
payback period. Such techniques are effective ways of evaluating tangible
costs against tangible benefits. One of the most popular frameworks which
includes facilities for an assessment of intangible costs and benefits is
Information Economics.

Formative evaluation should be an inherent part of the project
management process. Development projects should be continually assessed against
objectives and careful attention should be paid to this activity to avoid
project escalation. Project escalation is defined as the continued commitment to
an IS project in the face of continual negative information from formative
evaluation exercises. Major stakeholders in an IS project may be reluctant to
withdraw support because of heavy investment in personnel and other resources
devoted to a project.

At some point in this process a decision may be made to either
wholly or partially abandon a project. In either case, the organisation should
engage in another form of IS evaluation called here a Post-Mortem analysis. This
should attempt to determine the key reasons for such total or partial failure.
In a recent international survey, 60% of organisations internationally and 67%
of organisations in the UK had suffered at least one systems project that had
failed to deliver planned business benefits or had experienced significant cost
and time overruns.

The results of such an analysis are important in suggesting
ways in which the organisation may improve its development practice. It is for
this reason that the document produced from the Post-Mortem analysis needs to be
disseminated to senior management, project management and members of the project
team. This, of course, can only be done effectively if assurances of
non-recrimination are given to all project participants. Ideally, this analysis
should be conducted by a reputable senior executive that was not involved in any
way with the project under consideration. Alternatively, it should be undertaken
by an external body or consultant.

Wherever possible post-mortem information should be made
public. This is important in enabling the validation of IS development practice
and the effective progression of the profession of IS.

The figure also emphasises another important organisational
learning feedback loop. Even if a project reaches completion, it may fail in
some sense when it comes to be delivered. Therefore, at some suitable time after
a system has been delivered the organisation should engage in a summative
evaluation of the system and its project. One framework proposed for the
summative evaluation of IS is benefits management. Even at this point it
is possible that the system may be wholly or partially abandoned, in which case
it should also be the subject of a Post-Mortem analysis, as above.

It is important to emphasise that no system is ever complete. A
summative evaluation is likely to suggest a number of ways in which the system
may be modified or extended – normally both classed as systems maintenance. This
is the third feedback loop illustrated on the diagram. The conclusion is that
effective evaluation leads to effective management of maintenance.

For more information on ERP systems and implementation and other
topics relevant to IS managers and developers contact:

The
Kane-Thompson Centre

For
studies in Information Systems and Information Technology

Dr.
Paul Beynon-Davies,
Reader,
School of Computing,
University of
Glamorgan,
Pontypridd
CF37 1DL

Tel:
01443 - 482243
Fax: 01443 - 482715

Email:
mailto:[email-masked]