Of clouds and clocks: An approach to
information technology evaluation and personal freedom

Laurie McAulay*

How can our knowledge of evaluation have
more impact? If we believe that evaluation is a part of a deterministic
system, then it is appropriate that we try to make our evaluation
techniques more accurate. The alternative view is that information
technology (IT) and its evaluation evolve over time, as new solutions
are discovered, which in turn are rejected or accepted to provide new
states of knowledge. This indeterministic view implies alternative ways
of considering the impact of information technology evaluation. This
paper presents a range of views so as to encourage readers to surface
their assumptions about the impact of their own
research.

Introduction
The title of this paper is taken from a lecture given by
Karl Popper at Washington University in 1965. Ideas have no doubt
progressed in the intervening 35 years, but the framework offered by his
lecture is an intriguing one, and poses a challenging problem for
researchers that are interested in the evaluation of information
technology. Fundamentally, the framework invites us to position our
research in relation to our underlying beliefs about the nature of
information technology. The paper that follows poses fundamental questions
about what we expect information technology in general to offer, and how
evaluation might contribute to the evolution of information technology. No
answers are provided, but the intention is to inspire readers to surface
their own assumptions about the motivations behind their own
research.

The clocks of the title refer to systems that
are "regular, orderly, and highly predictable in their behaviour"; by
contrast, clouds are "highly irregular, disorderly, and more or less
unpredictable" (Popper 1972: 207). Might it be possible to position
information technology evaluation in relation to a continuum that
stretches from clocks at one extreme to clouds at the other? Popper (1972:
208) provides what he describes as a common-sense view that can guide us
in responding to this question:

There are lots of things, natural processes
and natural phenomena, which we may place between these two extremes -
the clouds on the left, and the clocks on the right. The changing
seasons are somewhat unreliable clocks, and may therefore be put
somewhere towards the right, though not too far. I suppose we shall
easily agree to put animals not too far from the clouds on the left, and
plants somewhat nearer to the clocks. Among the animals, a young puppy
will have to be placed further to the left than an old dog. Motor cars,
too, will find their place somewhere in our arrangement, according to
their reliability: a Cadillac, I suppose, is pretty far over to the
right, and even more so a Rolls Royce, which will be quite close to the
best of the clocks. Perhaps furthest to the right should be placed the
solar system.

Where might we position information technology
evaluation according to this typology? Differences of opinion are
possible. Walsham (1991: 93), for instance, concludes that "the
organizational metaphors of machines and organisms underlie much of the
current literature and practice of information systems". The machine
metaphor positions information technology evaluation towards the right, to
use Popper's imagery. However, the technical orientation that such a
stance implies was under attack as early as the 1950s, particularly
through the socio-technical approach associated with the Tavistock
Institute (Trist 1982). This leads to the alternative interpretation: that
information technology evaluation is organic in nature, and thus needs to
be positioned towards the left of Popper's continuum.

However, this common-sense approach goes
against the scientific advances of recent centuries, particularly the
Newtonian revolution, in which, as Popper (1972: 210) expresses it, "all
clouds are clocks". This deterministic view suggests that any attempt to
position systems between the two extremes testifies not so much to the
nature of systems per se but to our inability to model them adequately.
The concept of adequate modelling implies that the seemingly most
cloud-like systems behave with clockwork precision: a precision that it is
the role of researchers to discover. It is this view that begins the
discussion which is undertaken below.

The material for this discussion is taken from
an analysis of the existing literature and a protocol analysis study of a
small group of UK Top 500 financial directors as they responded to a short
information technology evaluation case study (King & McAulay 1997;
McAulay, Russell & Sims 1997). The method underpinning the paper is
inspired by deconstruction (Kilduff 1993) and discursive psychology
(Edwards 1997; Potter 1996). What follows is an excavation of alternative
views about the nature of information technology evaluation.

Clocks and information technology
evaluation
Introduction

If a clock implies regularity, order and
predictability, what does this suggest for information technology
evaluation? The answer that is to be found in the literature is
superficially benign and stems from fundamental definitions of the nature
of information technology evaluation. Just as clocks measure time, so
information technology evaluation measures the impact of information
technology. For instance, a recent definition has suggested that
information technology evaluation makes "explicit, quantitatively or
qualitatively, all the impacts of an IT project and the programme and
strategy of which it is a part" (Farbey, Land & Targett 1999: 190).
Impact, in turn, relates to value and usefulness: "the assessment or
appraisal of the value, worth or usefulness of an information system"
(Smithson & Hirschheim 1998: 160).

How might we model the impact of information
technology, and what assumptions are made about the relationship between
evaluation and that impact? Central to thinking on these matters is the
implication of Fitzgerald's (1998: 16) assertion that "organizations need
to be much more careful about which IT/IS projects they implement". The
implication of care in selection is surfaced through the proposition:
"Although IS are integral to the operation and success of most
organizations, organizations have limited resources and these resources
should be invested in anticipation of future gains" (Serafeimidis &
Smithson 2000: 93). In other words, information technology evaluation is a
necessary guide to the selection of appropriate applications. Information
technology evaluation is necessary to choice and justification, so that
"if a consistent justification process is not followed, the more
beneficial application may well be deferred, allowing those that are worth
less to proceed" (Ward 1990: 222). This process is not a passive one: it
is not sufficient to merely select the "more beneficial application";
"successful utilisation of information technology is by no means
automatic. Appraisal of costs and benefits is necessary if value for money
is to be obtained from existing information systems and plans made for
future applications" (Symons 1991: 205). In other words, knowledge is
necessary to constructive action; knowledge of costs and benefits is
necessary to their realisation (Farbey et al. 1999).
The
cognitive map in Figure 1 clarifies this analysis further and is drawn
from Smithson & Hirschheim (1998: 160-162). This analysis suggests
that information technology fulfils a number of purposes (see also
Ballantine, Galliers & Stray 1996: 130) and that these lead to changes
in the information system (IS) and thereby to impacts on performance,
which can be realised either through improvements in decision making or in
user productivity.

The diagram will be divided into two in order
to pursue the discussion further: the first slice through the available
literature will address the relationship between evaluation, the
justification of new technology, and information systems implementation,
while the second slice will address information systems implementation and
business performance. The first is represented by a substantial literature
that is central to evaluation and will be treated more comprehensively
than the second, which has its own body of literature, most notably
associated with the phrase, 'the productivity paradox'.

The evaluation-implementation
transformation

Although absolute determinism is avoided in
both the practitioner and academic rhetoric of the transformation of
evaluation into implementation, different forms of determinism are
nevertheless clearly evident. These take on different forms. Fitzgerald
(1998), for instance, adopts a "hindsight determinism" that is typified by
his account of a high street retailer's experience:

A high street retailer ... implemented a
system to link computers and tills in their stores to their head office
via an X25 private network. The system polled the shops every 10 seconds
and had a sophisticated system for locating and diagnosing faults. ...
When burglars cut the telephone lines in raids on shops the system
raised an alarm at head office and the police were notified. In one
weekend, out of 12 attempted break-ins at its branches, nine were foiled
by the system. The security benefits of the system were unanticipated
but substantial and could easily have been used in the evaluation and
justification process." (Fitzgerald 1998: 22, italics introduced by
the author).

p.8 figure 1. missing

With the benefit of hindsight, the high street
retailer's benefits were predictable. This is a point that Fitzgerald
generalises throughout his analysis, including the statement, for
instance, that "a large proportion of changes and enhancements made to
existing systems were predictable at the development stage"
(Fitzgerald 1998: 23, italics introduced by the author). In contrast,
Smithson & Hirschheim (1998) adopt an idealised form of determinism
that, consistent with Plato's concept of the ideal form, is both essential
but difficult to attain in practice. They analyse the case study of a
decision to outsource an IS department. An internal evaluation and an
external benchmarking exercise both supported the decision to reject
outsourcing, but, "although [the manager's] report was a realistic view of
the problems than can occur with outsourcing, senior management had
already made up its mind to outsource. ... In doing so, they pointedly
ignored an independent benchmarking report which portrayed [the company's]
IS function as close to 'best of breed' in their data capture operations
and above average in systems development" (p. 171). The conclusion to be
drawn from the case study is that "the failure to understand IS
evaluation, and adopt appropriate evaluation practices, can lead to
misconceived decisions with deleterious consequences for users and
developers alike". A UK Top 500 financial director provides an account of
how such an effective practice might be achieved. This adopts a form of
imposed determinism that is redolent with the information technology
evaluation literature in general:

What is very important at the beginning of
the project is that you agree a specification in detail: what's
required, who's going to do it, what will come out at the end ... what
are you trying to get out of this, how it will work, the costs and the
benefits. You have benchmarks ... along the way to make sure that what
you said you were going to achieve gets done. You don't allow the
project to go awry. ... It has to be continually monitored, including
the provisions in the original costing and the time-scale to train
(McAulay et al. 1997: 20).

Irrespective of the form of determinism that is
implied by these differing accounts, an underlying phenomenon is evident:
dissatisfaction and the avoidance of dissatisfaction. Weill & Olson
(1989) outline two themes in the dissatisfaction of practitioners and
their attitudes to evaluation. The first theme is that organisations are
dissatisfied with the available methods irrespective of the method chosen,
although return-on-investment (ROI) methods dominate. Ward, Taylor &
Bond (1996) found that only 36% of the respondents in their survey who had
an investment appraisal process were satisfied with their processes.
Common problems included the ability to account for the full range of
benefits, but also that benefits might be overstated to gain approval for
investment. The second theme is that many projects must be implemented
irrespective of the findings of evaluation. "A threshold investment is
often necessary to compete in the industry and is often not subject to ROI
or any other type of return calculation" (p. 11). This is similar to the
point made by Farbey et al. (1992) that projects might be implemented
because of top-down strategy (including the putative acts of faith),
top-down dictat, the need for incremental change or competitive
imperative. This analysis therefore raises two questions: the adequacy of
methods, particularly the quantification of costs and benefits (Powell
1992), and the relevance of methods - they might simply have no
impact.

Farbey, Targett & Land (1995: 250) seek to
overcome this dissatisfaction and the belief that "evaluation has too
narrow an interpretation in many organizations, being taken to refer to
the quantification of well-defined benefits with the sole objective of
getting the project over the organizational fences" of investment
appraisal. Their case study work suggests that evaluation produces new
information about projects and helps to clarify stakeholder perspectives.
This broadening provides a solution to the need for evaluation to do more
than provide simple efficiency metrics (Smithson & Hirschheim, 1998).
It serves to challenge managers to match evaluation with the purpose of
the investment. A number of authors have contributed to the debate as to
how evaluation characteristics and information technology might be
matched. The key point is that "it is very important to use the
appropriate basis for judgement of applications based on the role they are
expected to fulfil in the business" (Ward 1990: 224). Ward (1990) matches
three purposes for investment (substitutive - efficiency, complementary -
effectiveness, and innovative - competitive) and five evaluation
techniques (cost-benefit, value linking, value acceleration, value
restructuring, and innovation evaluation). Farbey et al. (1992)
alternatively suggest that the technique that is appropriate is determined
by the timing and level of investment, the environment, the nature of the
organisation, relation of the system to the organisation, organisational
characteristics and cause-and-effect relationships. Their framework was
later developed and presented as Farbey et al. (1994).

Cornford, Doukidis & Forster (1994)
illustrate the size of the challenge posed by contingency frameworks such
as these by showing that it is possible to create a substantial framework
for the evaluation of a single investment, an expert system. Evaluation
proceeds by means of a three-by-three framework, which considers
structure-process-outcome on one dimension and systems functions-human
perspectives-organisational context on a second dimension. The nine cells
that result prompt a systematic questioning of the impact of the system.
For instance, the structure-systems functions cell prompts questions such
as, "What are the real hardware and software requirements; is the software
architecture understandable and robust; does the full set of system
components work together in a technical sense" (Cornford et al. 499),
while the human perspective-structure cell adopts a stakeholder
perspective to ask a range of questions, such as, from the point of view
of the organisational administrator, "is the system a reasonable cost
effective alternative to existing tools or materials in use" (p. 501), or,
from a provider's perspective, "what are the changes to working conditions
and practices, in terms of the physical environment and skill
requirements?" (p. 500).

We know that, in practice, few investments are
evaluated with such rigour. Farbey et al. (1992) and Ward et al. (1996)
paint a consistent picture of formal procedures applying in roughly half
of all cases. Recent evidence suggests that this informality may in itself
be the source of practitioners' dissatisfaction with their information
systems. Sabherwal (1999) shows a high positive correlation between IS
planning sophistication and IS success. Nutt (1998) shows that analytical
and bargaining approaches to decision making outperformed individual
judgement based on prior experience or knowledge of the
situation.

These findings beg the question of the
relationship between the tools provided by formal evaluation and the
support that they give to the human decision maker (Powell 1992). Nixon
(1995: 277), for instance, shows that "strategic decisions reflect the
beliefs, assumptions and biases of top management" and that "decision
making was driven by the need to reconcile apparent inconsistencies among
different pieces of data or information, often from different sources:
between financial and non-financial information, between quantitative and
qualitative, between internal and external information, and between
'facts' and 'soft' data, including gossip, impressions, distilled
experience and intuition" (p. 280). The recursive nature of the
psychological process is modelled by Swan & Newell (1994). In their
model, knowledge is socially recognised, coded and embedded in
organisational systems, including evaluation, technical systems and
information technology. Individuals use this information, but do so
selectively, emphasising some elements rather than others. We can suspect
from existing case study research that soft data may have greater
psychological impact than hard facts, and this is supported by the
communications literature that considers how agendas are set. The result
of the sense-making process is that new solutions are adopted which, in
turn, shape knowledge. Deterministic models must cope with this recursion
and must model the psychology of the process; perhaps in ways similar to
that illustrated by Swan & Newell's (1994) work.

The psychological difficulty of gaining
acceptance for formal systems and hard facts is evident in the paradox of
benefits recognition. We know that measuring benefits is considered by
practitioners to be a significant problem (Galliers 1991; Ballantine et
al. 1996), and yet no other topic has yielded as much in terms of
consistency and comprehensiveness of findings as this area. Bacon (1992)
summarises criteria of assessment under six financial benefits, six
management benefits and three developmental benefits of IT implementation.
Ward et al. (1996) list eight benefits: cost reduction, management
information, process efficiency, enable change, competitive advantage,
business necessity, communications and service quality. Lederer &
Mirani (1995) draw upon the existing literature to create a survey that
includes 33 benefits. These are factored to give nine benefits: improved
information, strategic advantage, return on investment, reduced technology
cost, better applications development, reduced travel costs, reduced
workforce costs, business redesign and adherence to business redesign.
Although the words differ between these papers, there is so much in common
that it is not difficult to argue that we do understand the benefits of
information technology, despite proclamations that we do not. The more
interesting questions concern why our knowledge of these benefits is not
adopted by practice and whether the benefits change over time. There is
evidence from a study by Karahanna, Straub & Chervany (1999) that
perceptions of the benefits of adoption do change as implementation
proceeds, thus suggesting that deterministic models need to reflect
different kinds of benefits analysis at the different stages of the
project life cycle.

The preceding discussion problematises the neat
cause-effect relationships of Figure 1. There are many challenges for
those who wish to present a deterministic account of the transformation of
evaluation into implementation. Among these challenges, dynamism has been
highlighted as a central problem (Fitzgerald 1998; Smithson &
Hirschheim 1998). The changing world imposes limitations on the ability to
sustain hindsight, idealistic or imposed determinism, as explained by a
financial director:

The reason that you always have problems with
things that don't show returns sort of five or ten years afterwards is
that the world changes so rapidly that you can't be confident that it's
going to look like that when it actually happens (McAulay et al. 1997:
27).

A second financial director takes a more
radical position on the impossibility and unreliability of prediction
(Singer 1994: 223-224):

The challenge in all these things is that the
real world doesn't usually work out the way in which you put it into the
projections. If it does, you ask, you worry. It sounds very odd to say,
but ... things that come in exactly as they were supposed to ... you
tend to think that someone's manipulating the numbers ... because the
world isn't like that. Things change so often (McAulay et al. 1997:
39).

The implementation-performance
transformation

An obvious benefit of information technology
implementation is improved performance, and a growing literature is
producing insights into the relationships between implementation and
business performance. This literature centres on the well-known
productivity paradox and is extensive. This section will restrict its
attention to a small number of recent works that have shown positive
results.

The available analysis has proceeded at both
the national and firm level. Dewan & Kraemer (2000) show that IT
investment gives positive returns for developed countries, using IT
investment per gross domestic product (GDP) and GDP per worker as the
basis for measurement. They include technology, social and human capital,
infrastructure and policies as necessary preconditions to the achievement
of the potential of IT. Less developed countries do not show the same
returns because these preconditions are not always met. The theme of the
presence of additional variables to explain the IT investment-business
performance transformation is repeated in Rai, Patnayakuni &
Patnayakuni (1996) at the level of the firm. They test the model shown in
Figure 2 and show that the IS budget is not related to financial
performance as they define it, but is positively related to sales
performance.

p.10 figure 2. missing

Within a deterministic perspective on
evaluation, it is incumbent on us to show the relationships between the
benefits that we build into evaluation and business performance: the
relationships between the kinds of benefits represented by Rai et al.
(1996) as intermediate performance and performance at the aggregate level.
In this respect, knowledge is beginning to grow, but there continues to be
scope for important future research. Current knowledge drawn from outside
the information technology evaluation literature is surprisingly
supportive of the very evaluative measures that have always been
dismissed: those based on discounted cash flow (DCF). Brush, Bromley &
Margaretha (2000) show that free cash flow (the undistributed cash flow in
excess of that needed to create a positive net present value as shown by
DCF calculations) leads to increases in sales growth, which in turn leads
to performance improvements for certain kinds of firms, dependent upon
their governance structures. The argument is similar to that provided by
Dewan & Kraemer (2000): that there are preconditions for the
transformation of investment into performance. Likewise, Simerly & Li
(2000) show that debt can introduce a short-term orientation that can
limit strategic choices so that capital structure impacts performance
differently under stable and dynamic environments. The overall conclusion
that results from this literature, and which is well expressed by
KivijŠrvi & Saarinen (1995: 156), is that positive performance results
can be demonstrated but that the demonstration is complex: "investment in
information systems pays back, but only in the long term, probably as a
result of the long development and learning time required to take
advantage of them." The literature has come a long way in the five years
since Lubbe, Parker & Hoard (1995) could simply show that the most
profitable companies are likely to spend a significantly higher proportion
of the non-interest operating costs on information technology. However,
there is an opportunity to develop our understanding further through
improved models of the ways in which information technology investment can
be translated into improved business performance.

Clouds and information technology
evaluation
Popper (1972) somewhat surprisingly dismisses the
determinism implied by the metaphor of the clock and favours the cloud
metaphor as a way of understanding a range of systems, including those
involving computers. The reason for this is that determinism opposes
personal freedom: if determinism is taken to its extremes, human beings
are subject to laws that are determinable and that leave no room for
freewill, something that Popper regarded highly at both personal and
social levels. He would therefore have been opposed to the kind of
de-centring of the human being that is endemic to the analysis provided by
the previous section. However, he also dismissed the absolute uncertainty
implied by an extreme view of clouds, and he constructed his analysis on
the basis of the development of human beings, a process he likened to
"clouds controlling clouds". The basic argument is that the evolution of
humankind has proceeded exosomatically: by "developing new organs
outside our bodies or persons" (p. 238, italics introduced by the
author). These "organs" include both information technology and its
evaluation. Evolution proceeds by creating a "mulitiplicity of the
tentative solutions" and a "multiplicity of the trials", so that
development is a continual process of generating new possibilities, and
evaluating their appropriateness, so that knowledge is continuously
recreated and technologies are redefined. This is partly a process of
re-description: each successive trial may lead to new types of
utilisation, just as the telephone came to be used as a means by which one
person can speak to another, although its designer originally intended it
to be used to broadcast music (Markus 1994: 509).

There is no shortage of evaluation techniques
that hold the potential for re-description. Powell (1992) divides these
into objective and subjective and reviews the substantial literature that
deals with these. More recently, Van der Zee & de Jong (1999) suggest
that the Balanced Business Scorecard holds potential. Based on two case
studies, the Scorecard helped to improve integration, the speed of
strategy processes, team building motivations and awareness of the ability
of IT to solve business problems. This provides an example of a success
story in applying evaluation to information technology, and, moreover,
presenting further examples of success provides a challenging research
agenda for the future. However, belief in the efficacy of the Balanced
Scorecard might testify to nothing more than a further deepening of the
layers of clouds that interact in the implementation of information
technology. A literature that begins by offering information technology
evaluation as a solution to the problems of effective implementation of
information technology might be guilty of establishing the Balanced
Scorecard as a means of solving the problems of information technology
evaluation itself. Rather than an absolute solution to our problems, an
indeterminate view might see the Balanced Scorecard as yet another source
of potential success or dissatisfaction.

The number of clouds that may be relevant to
information technology evaluation continues to grow. Farbey et al. (1995),
for instance, suggest that five factors contribute to IT success: a
comprehensive search for benefits, post-implementation evaluation,
understanding management processes, good communication among stakeholders
and a champion to manage the acceptability of the project. These five
factors can each be considered as separate clouds that interact with one
other. By 1999, the analysis of Farbey et al. came to suggest some new
clouds to be added to their previous list: evaluation theory, stakeholder
theory, decision-making theory, project dynamics and learning theory. Each
factor or theory provides a set of knowledge that helps to advance our
knowledge of information technology evaluation, but each factor or theory
represents a body of knowledge which is itself being developed. The
interactions can themselves be tested both through the attempt to
systematise the knowledge and by testing against other forms of knowledge,
such as expert knowledge. So, for instance, Kathuria, Anadarajan &
Igbaria (1999) construct a knowledge-based system to select an IT
application for manufacturing based on matching both competitive
priorities and process structure to the available applications. This
fulfils both an evaluative and a selective function in relation to the
specific application.

If the aim of information technology evaluation
is exosomatic development, and the many techniques and influences that
impact upon information technology are seen in relation to a free agent,
we might seek ways in which evaluation can gain credibility, not solely in
its own right, but through the development of credibility in those who
advocate evaluation. Human beings and their tools develop interactively.
Sabherwal (1999: 139) expresses the point in relation to IS planning, of
which evaluation is a part, as follows: "Instead of focusing their
attention on trying to convince top management to take the actions
necessary to increase IS planning sophistication, IS managers should take
the actions necessary to build successful systems and thereby build a good
track record and credibility for the IS group." Clouds control clouds, but
clouds also impact upon other clouds.

Centring the individual in this kind of way
provides alternative views on the constructive development of information
technology evaluation. Singer (1994), for instance, proposes a radical way
of breaking out of the cycle of dissatisfaction with existing approaches,
not through the deployment of new techniques and an expansion of our views
about the many factors that might be relevant, but by re-describing
existing techniques with an emphasis on human capabilities. Singer begins
by establishing the inadequacy of existing accounting approaches to
evaluation as bluntly as any author in the information technology
evaluation field: "whilst many consider DCF to be almost useless, or even
counter productive, it is nonetheless very widely used." However, Singer
differs from other writers by taking an affirmative stance and by asking
how we might put DCF to constructive use. This relies on recognising
limitations, that "scientific knowledge tells us not to believe these
numbers", but by then asking "what are the strategic issues affecting or
impacting upon them?" (p. 228). The process explores future scenarios
through DCF techniques as an act of simulation. Decisions on information
technology implementation follow, not directly from the calculation, but
because "a particular re-description has compelling psychological force,
causing the suppression of other considerations" (p. 225). Evaluation
impacts not at the level of accurate descriptions of the future world,
which are likely to lead to dissatisfaction as events unfold in ways that
were not predicted, but at the level of re-descriptions: the new knowledge
that evaluation is known to be successful in generating.

Conclusion
Article after article throughout the nineties established
that evaluation was the key to ensuring information technology
effectiveness, and yet the same articles established that practitioners
are dissatisfied with the effectiveness of their evaluation practices. A
literature that promises to deliver impact fails at its very core: its
inability to impact practitioners in positive terms. Given the substantial
growth in knowledge in this area, we are left with one question: "How can
our knowledge of evaluation have more impact?" So, what can we learn about
impact from the framework suggested by Popper? If we believe that
evaluation can be likened to a clock, then the clockwork accuracy of the
crystal is the solution, and we must work as researchers to resolve the
gaps in our deterministic vision. We should console ourselves that
pendulums were once seen as the state of the art in the measurement of
time, and yet dissatisfaction led to change, which in turn led to our
current level of sophistication in time measurement. If, alternatively, we
believe that evaluation can be likened to a cloud, our knowledge of
information technology and its evaluation must evolve over time as a
process of generating new solutions, which in turn are rejected or
accepted to provide new states of knowledge. We should be open to the
multiple ways in which human beings and their technologies and evaluations
interact. We may learn not only from the dissatisfaction that is expressed
with regard to evaluation that falls short of the ideal, but may also
learn to understand the ways in which evaluation can come to be perceived
as successful. Whether we are determinists or indeterminists, information
technology evaluation continues to provide a fecund research
agenda.

References
Bacon, C.J. 1992. 'The use of decision criteria in
selecting information systems/technology investments', MIS
Quarterly, 16(3): 335-350.
Ballantine, J.A., Galliers, R.D. &
Stray, S.J. 1996. 'Information systems/technology evaluation practices:
Evidence from UK organizations', Journal of Information Technology,
11: 129-141.
Brush, T.H., Bromley, P. & Margaretha, H. 2000. 'The
free cash flow hypothesis for sales growth and firm performance',
Strategic Management Journal, 21: 455-472.
Cornford, T.,
Doukidis, G.I. & Forster, D. 1994. 'Experience with a structure,
process and outcome framework for evaluating an information system',
Omega, 22(5): 491-504.
Dewan, S. & Kraemer, K.L. 2000.
'Information technology and productivity: Evidence from country-level
data', Management Science, 46(4): 548-562.
Edwards, D. 1997.
Discourse and Cognition. London: Sage.
Farbey, B., Land, F.
& Targett, D. 1992. 'Evaluating investments in IT', Journal of
Information Technology, 7:109-122.
Farbey, B., Land, F. &
Targett, D. 1999. 'Moving IS evaluation forward: Learning themes and
research issues', Strategic Information Systems, 8:
189-207.
Farbey, B., Targett, D. & Land, F. 1994. 'Matching an IT
project with an appropriate method of evaluation: A research note on
"Evaluating investments in IT"', Journal of Information Technology,
9: 239-243.
Farbey, B., Targett, D. & Land, F. 1995. 'Evaluating
business information systems: Reflections on an empirical study',
Information Systems Journal, 5: 235-252.
Fitzgerald, G. 1998.
'Evaluating information systems projects: A multi-dimensional approach',
Journal of Information Technology, 13: 15-27.
Galliers, R.D.
1991. 'Strategic information systems planning: Myths, reality and
guidelines for successful implementation', European Journal of
Information Systems, 1(1): 55-64.
Karahanna, E., Straub, D.W. &
Chervany, N.L. 1999. 'Information technology adoption across time: A
cross-sectional comparison of pre-adoption and post-adoption beliefs',
MIS Quarterly, 23(2): 183-213.
Kathuria, R., Anadarajan, M.
& Igbaria, M. 1999. 'Selecting IT applications in manufacturing: A KBS
approach', Omega, 27: 605-616.
Kilduff, M. 1993. 'Deconstructing
organizations', Academy of Management Review, 18(1):
13-31.
King, M. & McAulay, L. 1997. 'Information technology
investment evaluation: Evidence and interpretations', Journal of
Information Technology, 12(2): 131-143.
KivijŠrvi, H. &
Saarinen, T. 1995, 'Investment in information systems and the financial
performance of the firm', Information and Management, 28:
143-163.
Lederer, A.L. & Mirani, R. 1995. 'Anticipating the
benefits of proposed information systems', Journal of Information
Technology, 10: 159-169.
Lubbe, S., Parker, G. & Hoard, A.
1995. 'The profit impact of IT investment', Journal of Information
Technology, 10: 44-51.
McAulay, L., Russell, G. & Sims, J.M.
1997. Inside Financial Management. London: Chartered Institute of
Management Accountants.
Markus, M.L. 1994. 'Electronic mail as the
medium of managerial choice', Organization Science, 5(4):
502-527.
Nixon, B. 1995. 'Technology investment and management
accounting practice', British Journal of Management, 6:
271-288.
Nutt, P.C. 1998. 'Evaluating alternatives to make strategic
choices', Omega, 26(3): 333-354.
Popper, K.R. 1972. Objective
Knowledge: An Evolutionary Approach. London: Oxford University
Press.
Potter, J. 1996. Representing Reality: Discourse, Rhetoric
and Social Construction. London: Sage.
Powell, P. 1992.
'Information technology evaluation: Is it different?' Journal of the
Operational Research Society, 43(1): 29-42.
Rai, A., Patnayakuni,
R. & Patnayakuni, N. 1996. 'Refocusing where and how IT value is
realized: An empirical investigation', Omega, 24(4):
399-412.
Sabherwal, R. 1999. 'The relationship between information
system planning sophistication and information system success: An
empirical assessment', Decision Sciences, 30(1):
137-167.
Serafeimidis, V. & Smithson, S. 2000. 'Information systems
evaluation in practice: A case study of organizational change', Journal
of Information Technology, 15: 93-105.
Simerly, R.L. & Li, M.
2000. 'Environmental dynamism, capital structure and performance: A
theoretical integration and an empirical test', Strategic Management
Journal, 21: 31-49.
Singer, A.E. 1994. 'DCF without forecasts',
Omega, 22(3): 221-235.
Smithson, S & Hirschheim, R. 1998.
'Analysing information systems evaluation: Another look at an old
problem', European Journal of Information Systems, 7(3):
158-174.
Swan, J. & Newell, S. 1994. 'Managerial cognitions in
decisions about technological innovation', EIASM Second International
Workshop on Managerial and Organisational Cognition, Brussels, 26-27
May.
Symons, V.J. 1991. 'A review of information systems evaluation:
Content, context and process', European Journal of Information
Systems, 1(3): 205-212.
Trist, E.L. 1982. 'The evolution of
sociotechnical systems as conceptual framework and as an action research
program', in A.H. van de Ven & W.F. Joyce (eds.), Perspectives on
Organization Design and Behaviour. New York: Wiley: 19-75.
Van der
Zee, J.T.M. & De Jong, B. 1999. 'Alignment is not enough: Integrating
business and technology management with the Balanced Business Scorecard',
Journal of Management Information Systems, 16(2):
137-156.
Walsham, G. 1991. 'Organizational metaphors and information
systems research', European Journal of Information Systems, 1(2):
83-94.
Ward, J., Taylor, P. & Bond, P. 1996. 'Evaluation and
realisation of IS/IT benefits: An empirical study of current practice',
European Journal of Information Systems, 4(4): 214-225.
Ward,
J.M. 1990. 'A portfolio approach to evaluating information systems and
setting priorities', Journal of Information Technology, 5(4):
222-231.
Weill, P. & Olson, M.H. 1989. 'Managing investment in
information technology: Mini case examples and implications', MIS
Quarterly, 13(1): 3-17.

Southern African Business Review
Special Issue on Information Technology 20004(2): 7-13
Of clouds and
clocks: An approach to information technology evaluation and personal
freedom
* Laurie McAulay is at the Loughborough University Business
School, Leicestershire, United Kingdom E-mail: [email-masked]