IBA â Image Based Authentication. MAUT â Multi .... addition, this paper presents a detailed list of decision models that have been used in IT security so far. .... research question. Evaluation of decision models is carried out with the help of
The report recommended partnering as a means of improving inter-firm relations. It is one ... client, consultants, contractor and sub-contractors of a construction project all have a role to play in delivering quality project. ... The client was diss
Oct 21, 2005 - Project No.: IP-516278. Project. Acronym: SAFEDOR. Project Title: Design, Operation and Regulation for Safety. Instrument: Integrated Project. Thematic. Priority: ... The information contained in this report is subject to change withou
John R. Meyer and Mahlon R. Straszheim, Techniques of Transport Planning, Volume One: Pricing and Project Evaluation (Washington, D.C.: Brookings Institution, ..... cars. This kind of investigation has proven tractable, as described in a later sectio
Throughout the novel, there are changes in the narrative perspective used to tell the story. As she is recounting her own life, the narrator speaks in the .... called The Beginning of JeanetteÂ´s Journey of Finding Her Gender Identity: The .... chapt
Project Management Capabilities: Key to Application Development Offshore. Outsourcing. James M. Erickson. Dept. of Information and Decision Sciences. University of Illinois â Chicago ... Capabilities have been used as theoretical bases in other out
According to (PMBOK ), "The preparation of a detailed project scope statement is critical to .... 5%. Table 1. âProject Successâ Criteria and Factors: The References. WSEAS TRANSACTIONS on BUSINESS and ECONOMICS. Morteza Shokri-Ghasabeh, Kamy
Jun 25, 2012 - synergies of team dynamics, can no longer rely on the social aspects of work life. With that in mind, a new approach ... from being a solution to the challenge of motivating a virtual project team. This study aims to explore the ......
COMPILED THREADED ANDROID. APPLICATIONS. A comparative study on time-efficiency. Bachelor Degree Project in Informatics. G2E, 22.5 credits, ECTS ... non-threaded Android-applications written in Java and C/C++. ..... applications are written by the de
Sep 22, 2009 - The Eighth International Symposium on Operations Research and Its Applications (ISORA'09). Zhangjiajie, China ... solution. Optimization techniques, such as integer programming used in  and genetic algorithm adopted in , have bee
Indeed,  states that the challenge for organizations is managing a potentially diverse range of projects while ensuring ... balancing. Portfolio performance management and review. Portfolio governance. Portfolio resource management. Portfolio comm
This paper reports on part of the research on evaluative language currently carried out within the CONTRANOT project,1 which aims at the creation and validation of contrastive functional descriptions through corpus analysis and annotation in. English
Key Concepts of the INCAS Multicomputer Project. JURGEN NEHMER, DIETER HABAN, FRIEDEMANN ... Intel's iPSC ), are now commercially available and can serve as advanced testbeds for research projects. .... unreliable communication via logical buses
I will discuss these issues in some detail as my aim of this paper is to resolve most of the empirical problems in order to .... To have a project evaluated, the inventor/entrepreneur fills out a questionnaire and pays a fee. The fee for an .... for
Kepner-Tregoe, Inc., a management consulting firm specializing in strategic and operational decision making and project management. He is also. Director of Marketing and. Product .... And, as good as the training materials .... Making the answers to
cedures such as corpus callosotomy, multiple subpial tran- section, and vagus nerve stimulation in pediatric epilepsy surgery. After completing the assigned task (Appendix 1), the members of the Subcommission continued to meet to identify other impor
http://eprints.nottingham.ac.uk/32243/2/MANUSCRIPT%20without%20author%20details.pdf. Copyright and reuse: ......  J.A. Adam, Virtual reality is for real, Spectrum, IEEE, 30 (10) (1993) 22-29.  A. RÃ¶bler ...  J.R. Li, L.P. Khoo, S.B. Tor, D
(answers in a 5-point Likert scale). Question 18. In general, it is easier to work with printed didactic material, than digital one and you prefer the printed material (answers in a 5-point Likert scale). The value of Web 2.0 Tools. Question 19. Blog
questionnaire,d provided by the General Coordination for Non-Communicable Diseases and Health ... a Instituto Brasileiro de Geografia e EstatÃstica (BR). IBGE Informativo para a imprensa: IBGE ... Rio de Janeiro; 2009 [cited 2010 Nov 10]. Available
towards Business Intelligence (BI) systems and projects dedicated to the management supporting of small and medium enterprises. (SME). To design the method, known existing commercial and no- .... supplemented with planning data, e.g., budgets in the
. Evaluation of RFID implementation benefits in supply chain - literature review of case studies (ex post evaluation) and methods for justification (ex ante).  Evaluation of ... Details of tactic module of ART (ART-T) were presented in . .
Structures (BDAS 2015), May 26-29 2015 in UstroÅ, Poland and published in the Communications in Computer and. Information Science ..... taken from actors in a studio. In the case of artificial utterance, it is im- portant to know whether or not the
www.textroad.com. *Corresponding Author: Davood Gharakhani, Department of Industrial Management, Qazvin branch, Islamic Azad University (IAU),. Qazvin ... KEYWORDS: Supply chain management, supplier selection criteria, Fuzzy set, DEMATEL. 1. INTRODUC
Key criteria in Project Evaluation A study of New Service Development
Master Thesis Autumn Semester 2008 Supervisor: Professor Tomas Blomquist Authors: Nguyen Cam Nhung Eriona Shtembari
We would like to express our sincere gratitude to our thesis supervisor: Professor Tomas Blomquist, for his guidance and support throughout the process of writing this thesis. We are indebted to him for the time he committed reviewing our work and for his thorough discussion along the way. We would also like to extend our special thanks to Professors Ralf Müller; Professor Kaye Remington and Markus Hällgren for their additional guidance by commenting on our research proposal; providing us relevant materials; and giving valuable advices for our theoretical study; as well as offering clarifications when doubts arise. We owe our appreciation to our respondents for their willingness and time to participate in our empirical work and contribute useful information leading to the answer of our research question. Moreover, we are grateful to all professors and administrative staffs in the Erasmus Mundus MSPME master program, for the academic knowledge and social experience that we have gained through the last sixteen months. To our fellow course colleagues, they are deserved our best respect and a ‘thank you’ for their peer support and their comments on our thesis providing us with helpful suggestions. Last but not least, we would like to express our deepest gratitude to our families and our loved ones for their love, support and encouragement throughout the period.
Umeå, January 2009
ABSTRACT Project evaluation is concerned with indicators setting and performance tracking along the life of a project. It plays an essential role to the success of any project and therefore demands special attention. At the heart of this process lies a system of criteria one has to take into account when performing the evaluation. Our thesis considers this problem in a particular context, namely New Service Development (NSD) projects. The topic is of our interest because innovation (hence NSD) has become an inherent aspect of service industry while the research dedicated to NSD project evaluation is rather limited. Our thesis aims at understanding the purposes, the process of evaluating NSD projects and pointing out specific criteria included during the evaluation of NSD projects. As a result, the research question pursuing is: ‘What are key Evaluation Criteria for New Service Development projects?’ From the literature review on project evaluation and new service development, our study reveals a list of eight important criteria of evaluation. This consists of three financial criteria: (1) profitability, (2) production cost, (3) return on investment; and five nonfinancial criteria: (4) strategic fit, (5) marketing criteria, (6) corporate social responsibilities, (7) information quality, and (8) facilitating factors. From empirical perspective, qualitative approach is applied to collect data through three case studies and a series of semi-structured interviews with seventeen respondents in Albania, Italy, Sweden and Vietnam, from companies offering various types of service. The case studies build comprehensive understanding on the process of new service development, of project evaluation for NSD whereas interviews check the transferability of the three cases and identify evaluation criteria employed in practice. The empirical results were analyzed in comparison with the arguments found from literature. Regarding the research question, the study found that the set of evaluation criteria collected from empirical study fits with the list of eight criteria proposed by literature. Among this set, two main criteria ‘strategic fit’ (4) and ‘customer satisfaction’ which is apart of ‘marketing criteria’ (5) are recommended as ‘must’ for the evaluation process on any type of NSD project. Findings of this research contribute to the existing knowledge provided by both academic and practitioners regarding both project management field and new service development area, by suggesting a set of key criteria that should be used as guidance in order to succeed with evaluation of NSD projects. Key words: project evaluation, project control, monitoring, criteria of evaluation, service innovation, service projects, new service development. ii
TABLE OF CONTENTS ACKNOWLEDGEMENT ..................................................................................................... i 1
INTRODUCTION ............................................................................................................1 1.1. Research background...................................................................................................1 1.2. Research purposes .......................................................................................................2 1.3. Organization of the thesis ............................................................................................3
LITERATURE REVIEW ..................................................................................................4 2.1. Project evaluation ........................................................................................................4 2.1.1
2.2. New service development (NSD) ..............................................................................16 2.2.1.
Definition of NSD ..........................................................................................16
Types of NSD.................................................................................................17
2.2.3. NSD process and activities .................................................................................19 2.2.4. The characteristics and implications ...................................................................23 2.3. Key findings and the gap ...........................................................................................25 3
RESEARCH METHODOLOGY .................................................................................27 3.1. Research Philosophy .................................................................................................28 3.2. Research Approach....................................................................................................29 3.3. Research Strategy ......................................................................................................30 3.4. Data collection methods ............................................................................................32 3.4.1 Semi- structure interview .....................................................................................32 3.4.2 Case study ............................................................................................................36
DATA ANALYSIS ......................................................................................................39 4.1. Data analysis from Case Studies ...............................................................................39 4.1.1. NSD process .......................................................................................................39 4.1.2. NSD project evaluation process ..........................................................................41 4.2. Data analysis from Semi- structured interviews ........................................................48
FINDINGS AND DISCUSSION .................................................................................60 iii
5.1. Understanding the purposes of project evaluation in respect to NSD projects .........60 5.2. Developing the process of evaluating NSD projects .................................................62 5.3. Identifying criteria that should be carefully evaluated to ensure the success of NSD projects .............................................................................................................................66 5.3.1. Challenges in evaluating NSD projects ..............................................................66 5.3.2. Proposition list ranked by respondents ...............................................................69 5.3.3. Discussion on each evaluation criteria ...............................................................70 6
LIST OF TABLES AND FIGURES Figure 2-1 Systemic steps in project evaluation process……………………………………6 Table 2-1 Comparison between academic and practical literature on the project evaluation process ……………………………………………………………………………………...6 Figure 2-1 Systemic step in project evaluation process ……………………………………6 Table 2-2 Classification of project evaluation based on project life cycle ……………….13 Table 2-3 Project evaluation criteria ……………………………………………………...15 Figure 2-2 Definition of new service development (NSD) ..................................................17 Table 2-4 Typology of NSD ………………………………………………………………19 Figure 2-3 Two models of NSD ...........................................................................................20 Figure 2-4 NSD process .......................................................................................................21 Table 2-5 Managing Key Activities ……………………………………………………....22 Table 2-6 Literature summary on important factors influencing NSD ……………….......25 Table 2-7 Proposition list of evaluation criteria in NSD projects ……………………...…26 Figure 3-1 A Framework for Research Design ...................................................................27 Table 3-1 Research Approaches ………………………………………………………….31 Table 3-2 List of interviewees …………………………………………………………....34 Figure 4-1 New Service Development process ....................................................................40 Figure 4-2 NSD project evaluation carried out by project sponsor .....................................42 Table 4-3 Strategic fit analyses framework …………………………………………….....43 Figure 4-4 NSD Project evaluation carried out by project manager ....................................44 Figure 4-5 Effect Management chart ..................................................................................45 Figure 5-1 Overall Process of NSD Project Evaluation .......................................................64 Table 5-1 Challenges in evaluating NSD project …………………………………………69 Figure 5-2 Key evaluation criteria in % (ranked by respondents)………………………...69 Table 5-2 Key evaluation criteria by no. of respondents ……………………………........70 Table 6-1 Findings on Evaluation criteria for NSD projects ...............................................79
1.1. Research background The project evaluation process has been emphasized by both researchers and practitioners as crucial for the success of projects. Gramham (2006) argues that it is impossible to set meaningful targets for profitable project outcomes, without appropriate measurement and evaluation systems in place. Reliable evaluation techniques and criteria are becoming more and more important to stakeholders who are interested either in a specific project or overall activity of the company (Akalu, 2003: 355; Oral et al, 1991). In addition, the projects success is not only determined on the basis of the three traditional perspectives which are time, cost and quality, but it should also consider the long term benefits, the continuous improvement and the sustainability of the projects’ outcomes. It happens that many projects fail to appeal to intended customers or fail to add value to the organisations’ business. Others have been considered as not efficient enough because they are not well evaluated before, during and after the project implementation (Nelson 2006, Örtengren, 2004). The project evaluation process is therefore, carefully undertaken during the project life cycle by organisations in order to ensure that the project is profitable, that it is on the right track with expected parameters, and that the goals of organisations would be achieved once the project is completed. Despite the huge effort on establishing suitable framework for projects evaluation, most of the work gives few hints on the evaluation criteria for service development projects (Johne and Storey, 1998). Therefore, a higher contribution from researchers is needed on the service industry. This is especially critical due to the increasing contribution of services to the global economy. As reported by Grönroos (2000) the service sector has for a long time counted for over 50% of gross national product or total employment in developed countries. According to the US industry statistics, ‘the non-good production industry accounts for approximately 70% of the total economic activity in United State’. Besides, at the meeting of the OECD Council at Ministerial level, the OECD (2005) reports that service industry gives an important contribution to the growth, productivity and innovation of OECD countries. Other developing countries are also moving towards the service industry instead of the manufacturing one, because of the demand from other countries as well as from their own citizen. The importance of service sector is further emphasized by researchers (such as Grönroos, 1998; Webster 1994) pointing out that it is the service elements that make the difference on the marketplace and not because of the product components in the manufacturing’s offering (Grönroos, 1998: 21). 1
Within the service sector, demanding customers and growing competition compels organisations to innovate and keep distinguishing themselves from others by providing clients with more value-added services. The development of new service is therefore becoming more and more significant to companies. However, the success of NSD projects is challenged by specific traits of service product (intangibility, heterogeneity and nonstorability), as well as by the novel ideas and the high risk of failure. This makes the evaluation process of NSD projects much more complex and requires special attention. In an extensive review of literature on NSD project, Johne and Storey (1998:221) suggest that ‘further research is required into procedure for choosing between NSD projects and for evaluating individual NSD project throughout their development period’. Surprisingly, academic research in this area, until recently, is still rather limited. The above discussion on project evaluation and NSD projects, together with the recommendation of Johne and Storey (1998), trigger an interest to investigate the key evaluation criteria that service companies should take into consideration in order to enhance the success of their innovation projects. Hence, the Research question is defined as below: What are key evaluation criteria for new service development projects?
1.2. Research purposes To reach the answer for the research question, we aim at the following three objectives: (1) Provide understanding on the purposes of the evaluation activity given its strong relationship to the success of the projects. Besides, we would like to clarify the concept and procedure of new service development (NSD). (2) Provide comprehensive understanding on the process of evaluating NSD projects by looking at the whole project life cycle and from different perspectives. (3) Identify the set of evaluation criteria that service companies consider during the process of evaluation for their NSD projects. To summarize, this research tries to give a valuable insight to the new service development process, the process to evaluate NSD projects and to provide practitioners with more comprehensive and practical guidance when doing evaluation for their NSD projects. This study is conducted with a hope to facilitate the evaluation process in launching new service and to add a small part to the current body of knowledge of project management from the view of a service company.
1.3. Organization of the thesis With the purpose of providing readers a clear and logical approach to the research topic, the thesis is divided into six chapters as below: Chapter 1: Introduction: This chapter sets the general research interest and background of the study. The research question and research objectives are provided following by the overall structure of the thesis. Chapter 2: Literature review: The theoretical background on project evaluation and NSD are reviewed and summarized in this chapter utilizing both academic and practical journals as well as books, online sources. The key findings and justification for our focus are presented in the last part. Chapter 3: Research Methodology: The research philosophy, approach and strategy applied by the thesis are identified. This chapter further explains the reason why a certain data collection method is chosen and describes how the data are collected. Chapter 4: Data analysis: This chapter provides the empirical data collected through semistructured interviews and case studies. Chapter 5: Findings and Discussion: This chapter discusses the empirical findings in connection to existing theory to address the three main purposes of the thesis that have been clarified in chapter 1. Besides, we made further inferences and comments on each section to develop the theory. Chapter 6: Conclusion: The key findings related to main purposes are summarised ending with the answer to the research question. Also, limitation and suggestions for further research are presented in this chapter.
2 LITERATURE REVIEW This chapter discusses two major parts of the research question from theoretical point of view. The first section provides specific and concrete review on the project evaluation field. The other one gives insights to the concept, process of new service development (NSD) projects and summaries important factors that influence NSD projects. At the end, a summary of key findings from both parts is presented with a justification of choosing the research topic.
2.1. Project evaluation This section is divided into three parts. The first part gives an overview of the project evaluation process, critical activities included in the process as well as a discussion on the purposes of project evaluation from various points of view. The second part classifies project evaluation into three categories based on different stages of the project life cycle, and summaries typical features of three types of project evaluation. The last part brings together common project evaluation criteria from the previous two sections. Project and program evaluation are used as synonyms in this paper. Both terms refer to the evaluation process of project having starting and ending points. 2.1.1
Project evaluation overview
a. The project evaluation process According to Steven et al. (1993), project evaluation is a combination of a number of activities ranging from setting indicators, developing model, defining measurable outcomes, identifying key stakeholders and their interests, selecting methodology for evaluation, collecting information, analyzing data and disseminating evaluation results for further learning. Two main chapters of his book are spent only for the description of how to collect data and how to analyze those data to support the evaluation process. The evaluation process therefore concerns very much on data gathering and information analysis. To support this view, McNamara (1994) stresses on the importance of sources of information, the reliability and validity of feedbacks that contribute to decision making in program evaluation. He says that the lack of good communication channel may lead to wrong evaluation results which dramatically affect the decision making procedure. On the other hand, Oral et al. (1991) consider setting criteria among the most important activities in the evaluation process, especially in projects that involve multiple stakeholders. They argue that it is necessary to ensure the ‘criteria used to evaluate projects are chosen with a maximum consensus among stakeholders subject to budgetary and other constraints’ (Oral 4
et al., 1991:872). Besides, they proposed a model that combines self evaluation and cross evaluation among the set of projects to enhance the possibility of the projects’ success. In the meantime, some authors develop suitable models or structures of evaluation for some specific types of project and generalize to other projects. For instance, Messner and Sanvido (2001) further contribute to previous studies by establishing ‘the organization based information architecture’ (OBIA) which provides a structure for the consistent identification of information needed to support the evaluation decision’ (p. 393). They put the project into the organizations’ context and consider five major information categories (organization, commitment, process, environment, and product) in order to have adequate information to decide whether to pursue or not a particular project opportunity. Although the study was focus on construction projects, they suggest this model to be applied to many other types of projects and in different contexts. Regarding the non- traditional projects, Fox and Baker (1985) recommend the use of simulation or experiment to evaluate innovation projects under the dynamic environment. They investigate the effect of different aspects of ‘ market conditions’ on project and based on that, suggest several criteria that companies should look at during the project evaluation process such as project cost, project impact, probability of project success, number of project generated and completion time, etc. Furthermore, Liang (2003) proposes a simple solution for evaluating Research and Development (R&D) projects illustrating through a case study in Taiwan. During the discussion, he emphasizes the critical role of management support taking into consideration the novelty of the project. The model also recommends project evaluators and decision makers to compare among projects four elements which are business advocacy, perceived investment size, expected contribution and innovativeness to decide whether to terminate or continue the project. However, Liang raised a typical problem in R&D projects as ‘too much technical focus and lack of essential information which should be collected and integrated before initiating the project’ (Liang, 2003:454). Another practical evaluation process focuses on how to establish indicators looking at both internal and external context of projects (Örtengren, 2004). The aim of this framework is to ensure the three main goals which are: Relevant, Feasibility and Sustainability. The sustainability of the project success was also of concerned by including risk analysis and assumption analysis during the evaluation process. Another advantage of this approach is that it uses the matrix structure to check the cause- effect relationship between goals, outcomes, immediate results, impacts and activities, indicators, means of verification, assumptions. However, there are some criticisms on this approach. Jackson (2000) says that the model is rigid and thus, stifles innovation and flexible management. The table below provides a brief comparison between academic and practical perspective on how project evaluation carried out.
Table 2-1 Comparison between academic and practical literature on the project evaluation process
The evaluation process
Academic literature Develop evaluation criteria in various aspects such as organization’s goals and strategies, the interest of stakeholders, resources available Select proper evaluation methodology Collect data and information Analyze data base on the evaluation indicators Provide lessons for further improvement (Adapted from Steven et al., 1993)
Practical literature Problem and situation analysis Stakeholder analysis Objective analysis Alternatives analysis Establish indicators/ measurement of objectives Risk analysis/ Risk management Analyze project assumptions Results verification (Adapted from Örtengren, 2004)
While the processes illustrated in Table 2.1 presents evaluation activities in a linear order, Bellamy et al. (2001) suggests an integrated evaluation process that takes into account the correlation among different activities to ensure the success of projects, and continuously generated knowledge for the next projects. Figure 2.1 provides clearer understanding on how project evaluation activities are integrated.
Figure 2-1 Systemic steps in project evaluation process (adapted from Bellamy et al., 2001)
To conclude, even though, there are some differences in the way of undertaking evaluation, literature on project evaluation emphasize the importance of stakeholder involvement, of information quality and of criteria establishment in the entire process. Moreover, the project evaluation is mentioned by both researchers and practitioners as an iterative process and go along all phases of the project life cycle. b. The purposes of Project Evaluation process It is found from literature that the evaluation process plays an important role in the success of projects. One of the major purposes of evaluation is to determine the worth or merit of projects, process or products under both internal and external constraints. Regarding to Ye and Tiong (2000), financial appraisal techniques in evaluating projects offers quantitative information to justify investment, particularly in high level of finance and political risk projects such as infrastructure. Whereas Chapman et al., (2005) argued that evaluation, as a risk management tool, allows project managers to reduce uncertainties when making decisions, especially before the project started. Moreover, they recognize that there is an iterative relationship between evaluation process and decision making and the subjective opinions of evaluators on the assumptions of project might lead to irrational decision. Hence, by combining both monetary and non monetary aspects, project managers aim at choosing the most appropriate projects and making determined decisions on the projects while doing evaluation. Besides, Frechtling (2002) mentions two reasons for conducting evaluation which are (1) ‘it provides information to help improve the project, information on whether the goal are being met and on how different aspects of a project are working are essential to the continuous improvement process; (2) provides new insights or new information that was not anticipated’ (p.11). He also stresses the role of evaluation process in facilitating information flows among stakeholders of organization. It enhances the feedback and feed forward mechanism through reports, questions that are delivered along the evaluation period. Agreed upon this point, Caulley (1993) suggests that evaluation could provide objective information, supply credible answers, and identify the reasons for success or failure. The insight of the project progress achieved through evaluation process allows management to take proper actions during the implementation of projects. Banwell et al. (2003: 79) further illustrates the role of project evaluation as a toolkit that ‘helps managers to guide and benchmark the development of organizations’ in the relation to the adoption of the project. In general, evaluation gives initial basis for monitor and control procedure of organizations. However, Brown and Remenyi (2002) comment that there is a growing consensus among academics that various ranges of issues should be addressed during evaluation. The traditional view of financially driven measurement-oriented evaluation should be replaced by a form of evaluation which is concerned with ongoing learning rather than simply 7
measuring. In addition, the International Development Research Centre (IDRC) argues that evaluations shift away from a tool for ‘control’ to a tool that can empower organizations and contribute to organizational learning. Evaluation is also considered as knowledge construction and capacity building by some authors (Vakola, 2000; Segone, 1998). The APM body of knowledge (2006) discusses that project evaluation review take place to check the likely or actual achievement of projects plan and to ensure the benefits of organization. Agreed with previous academic literature, it is also emphases that evaluation should be undertaken throughout the project life cycle. The following objectives of project evaluation review were identified (APM body of knowledge, 2006: 90):
Evaluate project management processes used Establish lessons learned and actions arising from them Raise any concern and agree on corrective actions Recognize individual and team performance
Validate the overall project against plan: schedule, budge, resource, quality Consider stakeholder relationship and perception Review the likely technical success of the projects
To support this opinion, Farbey et al. (1992) summarizes that 1) evaluation could be used as part of justification for a project, either an existing or a new project; 2) evaluation enables organizations to compare between a numbers of projects under constraint of resource available resulting in judicious decisions; 3) it provides a set of measures supporting the monitor and control system and 4) it determine the success or failure of projects base on initial benchmarks and provide lessons learned for the future (p.110). An empirical study on sixteen investments on IT project in various industries prop up the findings. Despite of touching different aspects of project evaluation, all the literatures concurred at the embedness of project evaluation on the decision making process along the project life cycle. It aims at problem solving and decision making (Sherwood-Smith, 1994 in Vakola, 2000; Wholey et al., 1994; Scriven, 1980). Evaluation gives criteria for the selection of project, assess current project status for the decision whether to continue with the project and finally, provides suggestion on taking further investment or not. 2.1.2
Classification of project evaluation
From the above discussion, it seems that the project life cycle influences pretty much the evaluation process, it is reasonable to classify project evaluation base on the life cycle. Steven et al. (1993) categories project evaluation into three phases which are planning phase, formative phase and summative phase. Later, McNamara (1997) expresses this consideration in another ways with three major types of project evaluation: goal-based,
process-based and outcome- based evaluation. The following categories are named as suggested by Steven et al. (1993). a. Planning project evaluation Planning project evaluation is taken at the very beginning of the project and prior to the project implementation. It gives justification to choose projects among many others. Many authors consider this process similar to the investment decision making as project actually is an investment. On one hand, the literature of project evaluation has placed a significant effort on providing various financial techniques that support the process. Some traditional methods used in project appraisal such as net present value (NPV), internal rate of return (IRR), Pay back period, discounted cash flow (DCF) were analyzed by many authors (Ballantine and Stray, 1998; Small, 1998; Müller, 2003). Small (1998) showed the role of these financial methods in evaluating projects, in particularly the carefully considered during the evaluation process. Small (1998) shows the role of these financial methods in evaluating projects, in particularly the cost and benefit analysis. The trade off between accrued cost and future benefits related to all stakeholders should be carefully considered during the evaluation process. Müller (2003) makes it clearer by explaining how those investment appraisal techniques supports accept or reject decision in relation to project through some simple examples. He also proposes some key investment factors that project managers should identify to ensure the financial gain of the projects. However, Akalu (2001) criticizes these methods on some of the problems below:
The dis-conformity in the measurement techniques before and after the projects. The dynamic characteristic of project is ignored as changes during project implementation are not accommodated. Might forego some good projects as do not consider the intangible long term benefits of projects Do not consider the strategic importance of projects
Akalu (2001) therefore proposes another method that takes into consideration the net stakeholders value by excluding the market value of debt used to finance the project from the amount contributed to stakeholders by the project. Yeo (2003) also recommends real option, another financial method as approach to investment evaluation, especially for R&D projects where value of management flexibility is crucial. However, ‘good analysis ties the details of strategy to financial implications’ (Barwise et al., 1989:85). On the other hand, Lopes and Flavell (1998:223) argue that the intense employment of financial and quantitative side of projects defined it ‘as myopic and misplaced’. The risk of failure would increase if other dimensions like the organizational and managerial aspects, political aspects, social acceptability, environmental problems, etc., depending on the 9
nature of the project are ignored. To contribute to this, Gardiner (2005:86) states that not only financial and economic aspects but also other factors such as environmental impacts, employment effects, etc should be taken into account to ‘assist organization to decide whether a project concept is worth turning into reality’. He also stresses that the decision at this early stage before major resources have been consumed is critical asking for careful evaluation and feasibility study is a good tool for evaluating proposals. Besides, AldelKader and Dugdale (1998) approach the evaluation process in a more pragmatic way. Referring back to theory on financial methods, combining with some non financial aspects and investigating into some large British companies, he emphasized the importance of intangible measurement criteria such as flexibility, delivery performance, quality improvement, reliability. Some strategic concerns during the evaluation process were identified by Alder (2000). The competitive advantage, value chain, cost driver, strategic fit, relationship with stakeholders, uncertainty are critical to the success of projects that need to be addressed during the evaluation process. In additional, Stokdale et al., (2006) suggest that the consideration of the integration between the content, context and process and the inter-relatedness aspect allows management to be more flexible in making decision. Greene (1988) mentions the participation of stakeholders as another criterion that should be considered in the evaluation project. Doloi and Faafari (2002) suggest that, in order to improve the successful of project, simulation method is useful in testing how the projects works and what are the possible problems if the project is launched. Furthermore, Mohamed and McCowan (1999) emphasize the need for a systematic evaluation of investment option, especially under the uncertain economy. b. Formative project evaluation The second phase of project evaluation also called as formative evaluation, regards the progress and implementation evaluation. The purpose of evaluating projects in such a phase is to enable the company to decide whether it is worth going ahead or is it better to kill the project. There are different methods used to evaluate this phase of the project. Most of the academic and practitioner literatures suggest the use of financial analyses such as cost-benefit analysis, ROI techniques as they provide important information to evaluate the ongoing progress of a project and take the right decision. Also APM Body of Knowledge (2006) suggests the use of investment appraisal techniques to provide a like-for-like comparison of options. According to it, the evaluation of the ongoing phase of the project is considered as addition to ongoing monitoring and control process. As discussed by International Development Research Centre (IDRC), this supports senior managers in decision making. Valeri and Rozenfeld (2004) also agree with this opinion. They consider evaluation process during product development as important to define the quality gate framework. It could be said that ongoing evaluation is a key issue calling the attention of all actors involved in the project. 10
According to Greene (1988) ongoing phase evaluation is important also to project stakeholders. Crawford and Bryce (2003) share the same opinion arguing that this type of evaluation, which externally focused, is stakeholder-driven and emphasises effectiveness of the project. While monitoring is considered as internally focussed, management-driven and emphasises on the efficiency of the project. However, Grabe (1983:13) emphasises that the principal objective of evaluation in ongoing phase is to ensure that implementation is on the ‘right track’. He argues that evaluation is seen as an opportunity for a direct contact with the project staff, dynamic situation in which activities are carried out as well as the possibility for curing malfunctions in project programming and implementation. Therefore ongoing and ad hoc evaluation of programs and project in the course of implementation are called as ‘evaluation in vivo’. Coutant and Cada (1985) and Steven et al. (1993) define evaluation of the ongoing phase of a project as the process evaluation, carried on during a project to monitor activities, to ensure that they will achieve objectives and to alter the direction, redefine new objectives, modify approaches, etc., if necessary. According to them monitoring of ongoing work generally takes three forms such as contacts between the agency project officer and the principal investigator, periodic progress or topical reports, and formal outside reviews. As argued by Farbey et al. (1992:110) ‘checks must be done to ensure that internal and external changes have not affected the feasibility of the project. At the same time progress on the project has to be assessed to ensure that the project is keeping within its budget’. They discuss that ROI techniques can be the natural choice. But there are projects such as service development which does not provide tangible benefits. Therefore, ROI is unable to capture many of the qualitative benefits which services such as IT bring (Farbey et al., 1992:109). As claimed by researcher (such as Parsons, 1983; Farbey et al., 1992; Hares and Royle, 1994; Remenyi, 1995; MIS, 1998 in Suwardy et al., 2003) there is an agreement in the literature that evaluations based on financial indicators are not appropriate to consider IT proposals and IT investment performance. As stated by Clemons (1991) it is necessary to take into consideration evaluation methods even though they can provide only information on the importance of alternative projects. But Danks (1997) criticises that it is vital for IT projects to take clear decisions through clear evaluation methodologies. Despite the method used during the evaluation of the ongoing phase of a project, information provided is very important to the company and project stakeholders. As mentioned by Grabe (1983:14) ‘evaluation becomes a tool and base-data supplier for a rolling planning where goals and objectives are gradually advanced’. In addition, Brown and Remenyi (2002) argue that ongoing evaluation produces information during the systems development process in order to help improve the product under development (. According to PMBOK the evaluation of the ongoing phase, considered also as monitoring 11
and control, provides feedback in order to undertake actions that can correct or prevent deviations from project management plan. c. Summative project evaluation The third phase of project evaluation is done after project completion. Even though it is important to evaluate projects after their completion, not much literature can be found. Therefore, the study on this phase is limited to few researchers. Summative phase is called as ‘ex-post’ evaluation by Grabe (1983:14) who argues that the main objective of such evaluation is ‘to determine a starting point for further activities in the same field, to explore the relative cost, effectiveness and impact of alternative approached, to identify common mistakes in comparable projects and to quantify such effects and impact patterns’. It could be said that this evaluation can be considered as learning tool for the company such as improvement in productivity or career patterns in employment, ‘snowballing’ effects, etc. This idea is supported also by APM and PMI bodies of knowledge. Referencing to Vakola, (2000) and Scriven (1967) it could be said that the evaluation after project completion makes an overall judgement about the effectiveness of a given project/programme. Uhl (2000) categorizes summative evaluation into two phases such as ‘testing phase’ aimed at confirming effectiveness of final version and ‘routine phase’ aimed at emphasizing quality assurance. Despite this sub-categorization, Crawford and Bryce (2003:363) claim that evaluation after the project/program is valuable ‘to mitigate poor project performance, demonstrate accountability and promote organizational learning for the benefit of future projects’. It ensures alignment of the performance measures with the project strategy. While Greene (1988) sees evaluation as program improvement. According to Coutant and Cada (1985: XII) evaluation in this phase is carried out to assess and interpret results at after the project is completed. They think that there are two important objectives in the last phase of project evaluation: ‘determining whether the contractor adequately carried out the goals and objectives of the work, as conceived in the proposal and contracted for in the work statement; determining whether the type of work done actually led to benefits to the resource commensurate with expectations and costs’. The purpose is to determine if the participants' needs were met, if the problem was solved, if the project was efficient, if recipients of results were satisfied, what directions new programs might take, etc. Chiesa and Masella (1996) also consider evaluation as important to measure the value of a project at the end. In contradiction to the opinion discussed above, Kumar (1990) suggests that in practice the primary reason for summative evaluations seems to be project closure and not project improvement. According to him the reason is that post implementation evaluations are being performed for the limited, short term reason of formalizing the end of the 12
development project and may not provide the more important long term, feedbackimprovement benefits of the evaluation process. To summarize, we are giving a brief overview of the evaluation process during the project life cycle as illustrated in the table below. Table 2-2 Classification of project evaluation based on project life cycle Type of project evaluation (PE) Project life cycle Methods to evaluate
Planning PE (Type 1)
Formative PE ( Type 2)
Summative PE (Type 3)
Close out phase
Financial: NPV, IRR, Payback period, DCF, real options Non Financial: Simulation, feasibility study
Provide measures/estimates to support investment decision making process Serve as baseline to set indicators for measuring success Mostly used for appraising project proposals or selecting projects in portfolio management
Narrow range of financial/economic impact
Before Gardiner (2005), Müller (2003), Akalu (2001), Abdel-Kader and Dugdale (1998), Alder (2000)
Improve overall performance of the project Provide objective information Provide measurements for control process
Wider range of human, organizational and economic impact
During project implementation Brown and Remenyi (2002); Grabe (1983); Caulley (1993); Banwell et al. ( 2003)
Financial techniques (i.e. ROI) Performance appraisal Effectiveness measurement End user satisfaction appraisal Make an overall judgement about the effectiveness of a given programme Provide measures/estimates to support decision making process Independent evaluation group is needed to avoid bias Narrow range of financial/economic impact After
Vakola (2000); Scriven (1967); Segone (1998)
Project evaluation criteria
The study in previous chapters shows that both researchers and practitioners suggest financial and non-financial methods and techniques to evaluate projects. In order to succeed it is important to make sure that the project is in alignment with the overall strategy of the company. Therefore criteria measuring the compatibility and consistency of a project with company’s strategy and long-range plan (Twiss, 1986; Chiesa and Masella, 1996) are seen as key criteria of project evaluation. According to Grabe (1983:15-16) there are several important factors which influence project evaluation. First, he claims that project and programme personnel are often ‘suspicious of the motives of evaluators’. Therefore it is crucial to establish conditions of trust between project staff and evaluation personnel. Second, ‘it is necessary to draw clear lines of distinction between monitoring and evaluation’ on the one hand, and ‘other management control functions such as auditing, periodical reviews and others’, on the other hand. He further argues that there are some criteria to be taken into consideration while evaluating the project. ‘First, project design and implementation must be relevant and remain relevant throughout the period of implementation. Second, action taken under the project should be effective. Third, it should have a significant impact, producing lasting change. Finally, it should be efficient which means that it should produce the desired results, with a minimum of undesirable side-effects, at lowest possible cost (a high cost/effectiveness ratio)’ (Grabe, 1983:13). As a result of the study introduced in this paper, key evaluation criteria are condensed as shown in Table 2.3
Criteria of PE Factors of success mostly mentioned in evaluation of project/program: - Commitment of all parties involved - Division of work/responsibilities - Clear and realistic objectives/goals - Specific links between project activities and objectives - Capacity of project group - Flexibility to adapt - Level of participation of end users in project evaluation Project evaluation scheme: - Results of project is potentially of great value to customers - Clear project scope - Alignment with organization’s strategies - Involvement of all stakeholders from the beginning till the end of project life cycle - Quality parameter is clear - Financial and technical control - Internal and external communication - Time - critical for NPD (introducing idea as soon as possible) - Cost, - Quality - critical for NSD - Customer focused (more in NSD) - Rework reduction - Technical specific focused - Balance score card in ICT projects - Financial perspectives - Customer perspectives - Internal perspectives - Innovation and learning perspectives - Decision environment (match the culture of the organization. Evaluation may have to conform to an existing corporate procedure or there may be no established practice.) - Organization characteristics: industry situation (stable or not) and leadership role (pioneer or follow) - Cause effect relationship. The degree to predict the impact of new service determines how to do evaluation - Economic contribution - Technological contribution - Social contribution - Probability of success - Resource requirement - Relationship between project staff and evaluation personnel - Draw clear lines of functions and responsibilities (in monitoring and evaluation other management control functions such as auditing, periodical reviews and others)
2.2. New service development (NSD) This section is divided into four parts starting from a discussion on the definition of NSD. The following part reviews some ways of classifying NSD. The third part focuses on describing NSD process and its key activities. The characteristic and implication together with a summary of important factors to NSD are presented in the last part. 2.2.1. Definition of NSD Although, New Service Development (NSD) activities are undertaken often in nowadays business life, it does not exist yet an explicit definition of NSD in literature. Some authors still refer to a broader term which is New Product Development (NPD). NPD, according to Business dictionary is ‘the process involved in getting a new product or service to market’. Kelly (2000) defines product development as ‘an overall process of strategy, organization, concept generation, product and marketing plan creation and evaluation and commercialization of new product or service’. This concept therefore combines both new product development (in a narrower concept to only manufacturing companies) and new service development. At the same time, many articles implicitly mention NPD or NSD as innovation in product or service, respectively (OECD, 2000; Menor et al., 2002, Van der Aa and Elfring, 2002). Chapman and Soosay (2003) claim that service innovation can be expressed in terms of new service launches. They also say that innovation in service sector could be both technical and non technical innovation while the latter plays a major role. All above literature agree that NSD offers organization’s clients not only the changes in product’s characteristics but also the changes in process and customer interfaces that their customers have never experienced before. However, Menor et al. (2002) view service innovation is less practical focus than new service development. Thus, the term is still a matter of controversy. More than that, there are two kinds of process that would be considered as new service development which is new product development and new offering development (NOD) in regard to service industry (Johne and Storey, 1998). Whereas new offering development is the development of additional service or support provided to customers together with the core products. This extra process allows organization to gain competitive advantage to its competitors (Storey and Easingwood, 1994). Two definitions of NSD have been found by researchers as most appropriate:
The development of service products which are new to the supplier ( Johne and Storey, 1998) The overall process of developing new service offerings ( Johnson et al., 2000)
Combining all of the above aspects, the following figure gives an overall view of NSD concept.
NPD- broad concept (Both service and manufacturing)
NSD/Innovation in services
NPD- narrow concept (Nonservice industrieFi gure 2-3s) NSD
Some industries that participated in service sector and doing NSD projects are summarized by De Brentani (1991) such as banking, insurance, other financial services, computer and systems, marketing and advertising, management consulting, accounting, communication, shipping and transportation. 2.2.2.
Types of NSD
Several ways of presenting NSD classification are discussed by literatures. Different authors look at NSD from different perspectives creating a variety of typology to this topic. Den Hertog (2000) classifies NSD based on various roles and levels of power of different stakeholders. Five types of NSD ranging from supplier- dominated innovation, innovation within service, client led innovation, to innovation through service and paradigmatic innovation are identified on the basic of three groups of actors and their influences on the projects. The involvement of customers is increasing from the first to the forth type while the last type requires the cooperation among all players. However, Jong et al. (2003) comment that the barrier between each type is not clearly identified because interest and power of each stakeholder are hard to well-understood. Approaching in a more pragmatic way, Gadrey et al. (1995) look at the differences in the nature of specific service areas, such as insurance, consulting and electronic information, in order to categorize service innovation. For example, ‘ad hoc innovation is specific for consulting activities’ (p. 9) as the new idea often come from clients and vary from time to time. Thus, consulting firms work more in collaboration with their customers in order to understand their needs and provide them with valuable solution which align with clients’ 17
strategies. The service innovation could be grouped into five types: product, process, organisational, market innovation and conquest of a new source of raw materials. However, different business areas classify NSD in various ways. From the broader view, Heany (1983) identifies four types of new product development in service industry (NSD): Product line extensions, product improvement, new products for the current market served, new products for an established market in which the business offering the innovation is not now recognized as a vendor. The functional impacts of each type was illustrated carefully while connecting with some managerial implications such as level of risk, the crucial role of expertise and experience staffs, or problem related to start up firms in developing new products. Lovelock (1984) further develops on previous study and come up with a category of six NSD types which ranging from smallest innovativeness to major innovations (table 2.4). Kelly (2000) defines three kinds of NSD on the same basic with Booz et al. (1982) cited in Johne and Storey (1998). While Booze et al. divide NSD into six groups (new to the world products, new product lines, additions to existing product lines, improvements and revisions to existing products, repositioning, cost reductions), she combines some types which have similar characteristics such as the core products that are new to the world or new to the company, or all activities or processes that improve or add value to current service are considered under one category. On the other hand, Johne and Storey (1998) broaden the development concept from the overall business development to specific process, offer and market development. The augmentation development which is part of offer development was stated as importance and affect organizations’ profit most. This show the needs from customers for supportive service to core product is increasing. By researching on some financial service clusters in Greece, Avlonitis (2001) presents six types of NSD depending on the degree of innovativeness that characterize each type. The range starts from the most extreme innovation of the new to market services to the least one of the Service re-positioning. He says that this way of categorizing NSD ‘lead to better developed new service and superior performance (p.335). According to Avlonitis, this is also the most popular way of classifying NSD. The table below summarizes the three most popular typologies of NSD.
Table 2-4 Typology of NSD
Stakeholder’s power 1) Supplier- dominated innovation 2) Innovation within service 3) Client led innovation 4) Innovation through service 5) Paradigmatic innovation Den Hertog (2000)
Typology of NSD in term of Nature of different sectors 1) Insurance sector 2) Consulting sector 3) Electronic information service 4) Etc.
Gadrey et al. (1995)
Extent of innovativeness 1) Major innovations 2) Start up businesses 3) New products for the current served market 4) Product line extensions 5) Product improvements 6) Style changes Lovelock (1984)
2.2.3. NSD process and activities a. Process According to Bowers (1985) claimed in Menor and Roth (2007: 828), an ‘NSD process comprises those activities, tasks, and information flows required by service firms to conceptualize, develop, evaluate, and prepare for market new intangible performances of value to customers’. In addition, Cooper et. al (1994) in Menor and Roth (2007) argues that new services are transformed from just an idea into launching stage through a process composed by different steps and evaluations such as project screening, market research, development, and test marketing. Researchers and practitioners base their arguments about NSD on product development. There is a lot of literature discussing on New Product Development (NPD). Many authors (for example Alam and Perry, 2002; Booz et. al, 1982) have developed theories and models of NDP process. They propose an eight linear stage model for tangible NPD. But it is not much literature regarding NSD process. Bowers (1987, 1989) in Alam and Perry (2002) developed a similar eight stage model for NSD process. Another model of 15 stages was developed by Scheuing and Johnson (1989). Even though this model is more comprehensive than the previous one, it does not address important issues such as cross-functional teams, parallel processing of development stages and cycle time reduction which have been highlighted in NPD models. Therefore it is said that there is no model of NSD that can match the model of NPD. Alam and Perry (2002) identify two models of 10 stages each as shown in the Figure 2.3. One model is linear while the other contains some concurrent stages.
Figure 2-3 Two models of NSD (Alam and Perry, 2002)
It is not clear which of this models fits best to NSD. Researchers debate among them as they have different opinions related to NSD models. Alam and Perry (2002) comment that some researchers (such as Scanbrough and Lannon, 1989) do not support the linear model, while others (such as Cooper and Edgett, 1996; de Brentani, 1989; Edvardsson and Olsson, 1996; Lievens et al., 1999) consider this model as one of the success factors for new services. The NSD process cycle represents a progression of planning, analysis and execution activities. The cyclic nature is meant to suggest the highly iterative and nonlinear processes typically employed in most NSD efforts. Indeed, services tended to use less formal NSD processes than those found in NPD (Johnson et al., 2000; Griffin 1997). The research undertaken by Alam and Perry (2002) points out the importance of customer needs. They suggest the adoption of a ‘customer-oriented model’ with the focus on 20
developing services that match customer needs. Another opinion is given by Shneider and Bowen (1984) related to the role of the front-line employees. According to them employees are in a prime position to identify customer needs in terms of service product, process and timing. Also contact staff can help stop process efficiency from overwhelming customer needs. Therefore they stress the importance to treat employees as highly valued customers. Jong and Vermeulen (2003) share the same view and they comment that frontline employees have an excellent view on unsatisfied customers needs and they are the first to recognize opportunities for innovation. Therefore their opinion should be taken into consideration while defining NSD process. However, Shostack (1984b) identifies four fundamental characteristics, such as objectivity, precision, fact-driven and methodologically based, in order to provide an effective development process for NSD. He also developed a ten-stage development process concluded by a post-introduction audit. According to him problems arise when stages are skipped or glossed over. Therefore, he stresses the importance of concluding all stages even in a good service design. Riel and Lievens (2003) mention that academic literature (such as Froehle et al., 2000; Storey and Easingwood, 1993; Tax and Stuart, 1997) consider the design of NSD process as internal success factor to provide innovation in a company, while others (Cooper and Kleinschmidt, 2000; Dutta et al., 1999; Storey and Easingwood, 1993, 1996, 1998) see NSD process, in terms of marketing, sales and distribution functions, as external factors related to innovation. In fact there should be no fear of failure. As mentioned by Dover, (1987) and Johne & Storey (1998), many service companies adopt a project team approach and employ product champions. These have been found to be important in pushing the project through the development process. Johne & Storey (1998: 199) propose an NSD process (as shown in Figure 2.4) which emerge six key themes which are: the corporate environment, the process itself, the people involved, analysis of opportunities, development and implementation.
Figure 2-4 NSD process by Johne and Storey (1998)
Newly implemented ideas are quickly imitated hence NSD is very important to companies. Consequently the process of service development is described as following the traditional stages of concept development, prototype development, prototype testing and launch (Voss et al., 1992 in Menor et al., 2002).
b. Key Activities Service industry is very competitive. Therefore companies try to be innovative and keep always updated with customer needs which are continuously changing fast, in order to be able to launch new ideas before their competitors. According to Storey and Kelly (2001), there is a growing body of knowledge on the new service development activities of service firms. Evaluating NSD activities in a service company enables the identification of actions necessary to improve the process. It is claimed that the service development process is a chain of activities which must occur for the service to function. It has been stressed that it is important to adequately reward development activities (Atuahene-Gima, 1996a; Scheuing and Johnson, 1989). Johne and Storey (1998: 209) consider NSD activities as supporting activities of two main tasks such as project task and program task. ‘The project task is concerned with getting a single new service product development right. The programme task is concerned with getting a series of service product developments right, usually over a longer time span’. They claim that one particular area that is stressed as being especially important is the link between the image of the new service and the image of the service organisation as a whole (Langeard and Eiglier, 1983; Thomas, 1978 in Easingwood, 1986). Edvardsson and Olsson (1996) argued in Johne and Storey (1998) that service development can be broken down into three activities:
service concept development service system development service process development
According to Jong and Vermeulen (2003), managing key activities enhances effective implementation of NSD. They organise key activities in two main groups as shown below: Table 2-5 Managing Key Activities (adopted from Jong and Vermeulen, 2003) People Involvement of frontline-employees Presence of product champions Management support
Funnel tools Multifunctional teams Availability of resources Pre-launch testing Market research and launch
The first group refers to people as they are widely considered in literature (i.e. Shneider and Bowen, 1984; van de Ven, 1986 in Jong and Vermeulen, 2003) as the heart of innovation. The second one refers to structure such as formal systems of work relationship among employees, providing rules and procedures to guide development process (Bower, 1989; Scheuing and Johnson, 1989; Avlonitis et al., 2001; Meyer and De Tore, 2001 in Jong and Vermeulen, 2003). 2.2.4. The characteristics and implications The specific features of NSD in regarding to those of NPD were discussed by many authors. Firstly, there was a consensus among literature on the intangibility trait of NSD. Thomas, (1978) stated in De Brentani (1991) that there is a risk on behalf of customers in buying such intangible new service which they could not fully examine the quality prior to purchase. On the one hand, intangibility allows new service or augmentations to be easily and quickly developed (Shostack, 1984a). This factor helps organization to be more flexible and response fast to customer’s needs adding value to their customers which turns out to bring revenue to the companies. On the other hand, some problems such as the quality of service design, the effectiveness of market launch or inadequate testing which lead to failure of new service were raised by Berry (1980). Another issue related to intangible feature of NSD is the difficulty in patterning or property right protection for the new service. Competitors find it very easy to imitate making organization hard to maintain sustainable profit and lessoning innovators’ incentive (Easingwood, 1986, Klivans, 1990). Secondary, the simultaneous production and consumption distinguish service and non service process (Dolfsma, 2004). In service sector, the interaction between suppliers and customers is substantial and therefore the barrier between producing, delivering and purchasing service is blurred (Grönroos, 1990). The relationship between companies and clients is very complex, crucial and lead to long term development of companies as stated by Jackson and Cooper (1988). According to Shostack (1984) in De Brentani (1995), this demands a high participation and performance of cross functional departments within the firms. In additional, De Brentani (1991) recommends that it is better for NSD projects to create more customized than standardized service as it helps organization to adjust quickly to customer’s special orders. The above literatures imply that both internal and external communication and interaction is highly crucial in this service industry than that in manufacturing sector. Moreover, Kelly (2000) realizes that the quality of service and the experience of customers on the service vary every time the service is purchased. This is because of the more human involvement in the transactions in service industry than that of manufacturing. In 23
contradict to De Brentani (1991)’s suggestion above, Lovelock (1983) argues that client might see the quality of service as inconsistence and poor and perceive the company as unreliable which lead to dramatic issues to the company’s development. The suggestions to companies are to constantly emphasize on training and practice to operation staffs who work with customer everyday to improve the standard of service. This is particularly critical for industries like courier, transportation (Maister, 1983 in De Brentani, 1991). Besides, Levitt (1976) advises that companies should also be flexible in dealing with different clients and might go further in applying technological systems to plan and reduce variability in service which in turns enhance customer’s satisfaction. Different from products, services cannot be produced in advanced and stocked somewhere before delivering to customers (Gallouj and Weinstein, 1997). By giving practical examples, they relate this feature with the customers’ demand which is always varied at different window of time and conclude the possible risks that companies might incurred. Berry (1980) also believes that the obvious consequences are the cost of idle facilities and human resource during the dip period, together with the loss of revenue when the demand is higher than supply. ‘The challenge for NSD is in designing alternate service level offerings (i.e. full versus limited service offerings for low and peak demand periods, respectively) and developing a countercyclical line of service’ (De Brentani, 1995:102). In supplement to the above four characteristics and implications on NSD, Cowell (1988) mentions ownership as the last factor that differentiate NSD from NPD. He argues that customers can only hire or access to service while having the full use of product. According to him, this is the basic difference between these two concepts. However, no practical or managerial implications have been suggested. In brief , base on the discussions on NSD characteristics and implication as well as process and key activities, some important factors have been identified as refer to the Table 2.6 below:
Table 2.6 Literature summary on important factors influencing NSD Source
Important Factors for NSD
Storey and Easingwood (1995) (factors depended on performance measure)
Sales: effective communication, overall company/product fit, distribution strength, market knowledge, product champion Profitability: staff skills and support, quality of service delivery, compatibility/importance, product/tangible quality, distribution strength, quick response Enhanced opportunity: compatibility/importance, product/tangible quality, market knowledge, product distinctiveness
Objectivity, precision, fact- driven, methodologically based
Atuahene Gima (1996b)
Market orientation Other factors: inter-functional teamwork, product-technology fit
Involvement of contact personnel and customers
Cooper et al. (1994)
De Brentani (1991)
Nature of service offerings Project synergy Market characteristics : Market competitiveness, product marker fit,
Edgett and Jones (1991)
Clear defined target market, Thorough and well organised development process, Effective performance of team members, Sufficient funding, Differentiate service, Top management support, Strong launch campaign
Shostack (1984b) Shneider (1984)
Johne and Storey (1998) NSD performance measurement
Financial performance (e.g. relative profits to sales; profitability level, and payback period) Window of opportunity (the degree to which the new product opened up new opportunities to the firm in terms of products and markets) Market impact (e.g. domestic and foreign market share)
2.3. Key findings and the gap Based on the literature review regarding project evaluation and New Service Development, the above study has come up with two findings: 1) Regarding project evaluation perspective, most of the contribution is given by researchers and practitioners who have specific knowledge and experience on project management field. The models of project evaluation established by literature are normally too technical and focus on financial aspects. The nonmonetary measurement criteria have been mentioned in very few articles while it is 25
suggested by some authors as critical, especially for innovation projects. Moreover, literature does not consider the different features of projects in different industries which results in an insufficient view on any particular sector. 2) Literature about New Service Development is mainly related to by marketing or economic research papers. A majority of studies discuss the features that differentiate service and manufacturing industries, the key factors that significantly affect the outcomes of NSD. The financial service sectors attracted much attention from many scholars. However, the project-alike characteristic of NSD was neglected. Not many literatures put NSD under the project context so the constraints and challenges related to project management could not be considered. Besides, the need for evaluating NSD projects has been raised by Johne and Storey since 1998 but not many studies have been done afterwards. The two key findings disclose a knowledge gap in the literature which is the inadequate of thorough and systematic study on evaluation criteria in NSD projects. In order to fill this gap the following steps are undertaken:
First, criteria considered as important for the evaluation process are identified based on general literature on Project Evaluation. A draft list is established (Table 2.3). Second, based on specific characteristics and important factors of NSD, it is created another list on the common factors recommended by literature that ensure the success of developing new service (Table 2.6). Finally, The above two lists are combined and then the common criteria are selected. The second list about NSD (table 2.6) provides specific aspect knowledge to the general list of project evaluation (table 2.3).
Table 2.7 below is the result of the above last step summarising all evaluation criteria in NSD projects. This is the proposition list which will be verified after our empirical study. Table 2.7 Proposition list of evaluation criteria in NSD projects Financial Criteria
Profitability criteria (revenue, number of customer, growth) Production criteria (promotion cost, salary) Financial parameter ( net present value, return on investment, cost of capital, payback period)
Strategic fit (objectives, strategy, policies, corporate values, company’s capacity) Marketing criteria (customer satisfaction, time to market) Environmental and CSR criteria Information quality criteria (accuracy, adequacy, appropriateness of feedbacks and feed forward) Facilitating criteria (stakeholder attitudes and participation)
3 RESEARCH METHODOLOGY The design of research methodology is considered very important and should be done prior of undertaking the research. Therefore the researcher should define the epistemology of research such as objectivism, subjectivism, etc, and philosophical stances such as positivism, interpretivism, etc. Furthermore, research strategy and research methods (procedures or techniques) should be identified, in order to obtain successful research results. By combining the elements of inquiry (knowledge claims, strategies and methods) research approaches are formed. In turn, the approaches are translated into processes in the design of research (Creswell, 2003). The figure below illustrates the process of research design. Elements of inquiry
Alternative knowledge Claims Approaches to research Design processes of research
Strategies of inquiry
Data collection Conceptualized by the researcher
Translated into practise
Data analyses Write-up
Validation Figure 3-1: A Framework for Research Design (Creswell, 2003)
In order to identify what are the important criteria for NSD project evaluation our research is based on:
Literature review from both researcher and practitioner’s points of view. It provides theoretical information about project evaluation and new service development and finalizes with a list of important criteria of evaluation in NSD projects. Seventeen semi-structured interviews with project managers, product managers and evaluation team from different companies in Albania, Italy, Sweden and Vietnam, dealing with new service development projects. 27
Case studies with three companies. Secondary data provided from companies documents.
In order to prevent error and bias as much as possible this paper refers to different sources of information when conducting the research. Many authors, such as Bryman and Bell (2003) and Saunders et al. (2003), support the research methodology based on the usage of multiple sources of information and they argue that it is the best way to confirm findings and limit errors. The following sections describe the underlying philosophy, approach, strategy and methods used to achieve our research objectives.
3.1. Research Philosophy According to Saunders et al. (2007), research philosophy relates to the development of knowledge and the nature of that knowledge. They further discuss that assumptions are done based on the philosophy adopted. These assumptions will underpin the research strategy and the methods that are chosen as part of the strategy and will give guidance to the whole research process. As suggested by some authors (Saunders et al., 2007; Bryman and Bell, 2003), Epistemology and Ontology are two kinds of thought about research philosophy. Moreover, all literatures agree that the above two considerations are related and support each other in the way one thinks about the development of knowledge resulting in the choice of research approach adopted during the research. From a broader view, Epistemology concerns ‘what constitutes acceptable knowledge in a field of study’ (Saunders et al., 2007:102). Under this way of thinking, there are two main principles which are positivism and interpretivism. Saunders et al., (2007) compare the positivists with ‘resource researchers’ while calling the interpretivists as ‘feeling researchers’ in order to distinguish the difference embedded in the two philosophies. Whereas the former is more focus on the facts and considers natural or physical science as the underpin doctrine, the latter looks inside the meaning of objects and try to understand the phenomenal from different points of view which is closer to social science. For positivists, reality is only represented by the objects, and the knowledge could be generated simply by observing the phenomena and describing them in a static way. On the other hand, interpretivists see the world in a more dynamic manner. They argue that the subject matter, especially in business and management field, is far different and complex than that of the natural science. The study of social world therefore emphasizes on the understanding of human’s actions and behaviours rather than just explaining them as in natural science (Bryman and Bell, 2003:16). The ‘social actors’ is in the centre of this philosophy.
More specifically, Ontology is concerned with nature of reality and raises questions of the assumptions done by researchers (Saunders et al., 2007). This consideration which takes in to account the validity of knowledge produced by researchers gives support to the contribution of Epistemology and ensures the quality of the study. Two aspects of ontology are objectivism and subjectivism whereas the former follows from the positivist position and the latter is more associated to the interpretivist’s. The objectivism portrays the position that social entities confront us as external facts that beyond our research influence (Bryman and Bell, 2003; Saunders et al., 2007). At the same time, subjectivism or constructionism holds a contradict thought. The subjectivists think that the only way to understand the phenomenal and make sense of the social behaviour is to involve in such activities. Hence, the researchers’ positions are different between the two types of ontology. While objectivists stand outside and observe the world, the subjectivists play a part of what is observed. Our position in this study is critical to positivist tradition and share the same opinion that social world of business and management is far too complex to lend itself to theorizing by definite ‘laws’. Therefore our research philosophy is oriented towards interpretivism which is based on ‘conducting research among people rather than objects’ (Saunders et al., 2007; Bryman and Bell, 2003). We see ‘the necessary of studying the details of situation to understand the reality or perhaps a reality working behind them’ as suggested by Remenyi et al. (1998) cited in Saunders et al., (2007:108). Also, we believe that it is impossible to generalize the theory by staying outside the organizations’ context as each project has its own unique characteristic and the circumstances, as well as the culture, are varied among companies and countries. Thus, it is important to understand the subjective meanings behind the process and the views of different organizations in evaluating their projects which finally allow us to answer our research question.
3.2. Research Approach As suggested by Saunders et al., (2007), research approach which deals with the way the study is going to be designed should be clearly defined at the beginning of the research project. It concerns the relationship between theory and research, whether the research should use the deductive or inductive approach (Bryman and Bell, 2003). Adopting the deductive approach, the researchers develop a theory and hypothesis and design a research strategy to test the hypothesis. The aim of this approach is to test the current theory. While using inductive approach means that researchers need to collect data and then, develop theory as a result of data analysis (Saunders et al., 2007). This approach, in contrast to the deductive one, is to develop a theory. Not only the process is different, the strategies, the data collection methods are not the same between the two approaches. The deduction emphasizes the quantity of the data in order to ensure the validity of the 29
generalize conclusion. It uses highly structured methodology to facilitate replication which is important to ensure reliability aspect of data collection (Gill and Johnson, 2002 in Saunders et al., 2007). At the same time, the induction focuses on the collection of qualitative data and the meanings of such data which contributes to the understanding of the new or unknown phenomena. This approach promotes ‘flexible structure to permit alternative explanations of what is going on’ (Saunders et al., 2007:119). The choice of research approach depends on the philosophy chosen by researchers. According to literature on research methodology, deductive approach is related to positivism while inductive approach is related to interpretivism. As our study is oriented towards interpretivism as mentioned above, then our research approach is concerned with inductive approach. Literatures, such as Saunders et al., (2007); Bryman and Bell, (2003), argue that inductive approach brings the preposition from the more specific observations to general theory which is contradict to that of deductive one. Our study therefore, starts from an existing preposition which is the list of criteria to evaluate NSD projects (Table 2.7) summarized from literature. We then collect data from project managers or project evaluators as well as from secondary sources to analyze their views, the process of project evaluation in practical world, particularly in new service development projects. This information set the basis for our interpretation and development of a list of key evaluation criteria for NSD projects that mostly applied in practice.
3.3. Research Strategy Research strategy is necessary to determine the method through which research data will be collected and analyzed. According to Creswell (2003), it is useful to consider the full range of possibilities for data collection in any study and to organize research strategies based on their degree of predetermined nature, their use of close-ended versus open-ended questioning, and their focus for numeric versus non-numeric data analyses. Moreover, Bryman and Bell (2003) argue that quantitative research can be constructed as a research strategy that emphasizes quantification in the collection and analysis of data. They further discuss that qualitative research is the other way around. By contrast, it can be constructed as a research strategy that emphasizes words rather than quantification in the collection and analysis of data. The following table shows distinctions that may be useful when choosing a strategy appropriate to the undertaken research.
Table 3-1 Research Approaches (Adapted from Creswell, 2003 and Bryman and Bell, 2003) Epistemological /Ontological orientation Method
Data collection type
Qualitative Constructivist/interpretivist/advo cacy/ Participatory knowledge claims Phenomenology, grounded theory, ethnography, case study and narrative Open-ended questions, Inductive, generation of theory
Quantitative Post positivist knowledge claims; objectivism
Mixed Pragmatic knowledge claims
Surveys and experiments
Sequential, concurrent and transformative
Instrument based questions Deductive, testing of theory predetermined approaches
Interview/document/observation /audiovisual data, text & image analysis Reach, deep data
Performance/observational/at titude/census data Statistical analysis Hard, reliable data
Both open and closeended questions Both predetermined and emerging approaches Multiple forms of data drawing on all possibilities Statistical and text analysis
Based on the research philosophy and approach as discussed in sections 3.1 and 3.2, the qualitative strategy has been chosen to fulfil our research objectives. In order to answer our research question we decide to collect qualitative data through methods such as semistructured interview and case study. The data collected are analyzed based on qualitative strategy which is focused on words rather than numbers. According to Strauss and Corbin (1990:8), ‘the importance of this methodology is that it provides a sense of vision, where it is, that the analyst wants to go with the research’. However, qualitative research is sometimes criticized by researchers as too impressionistic, subjective, difficult to replicate and restricted that make it difficult to generalize findings to other settings (Bryman and Bell, 2003). We therefore tried to improve such drawbacks and to ensure the validity and transferability of our data by using the form of triangulation as recommended by Stemler (2001) and Erlandson et al., (1993), which lends credibility to the findings by incorporating multiple sources of data, methods, investigators, or theories. In fact, it is the quality of theoretical inferences made out of qualitative data which are considered as crucial to the assessment of generalization.
3.4. Data collection methods This study is based on data collected through semi-structured interviews and case studies. The role of semi-structured interviews is to directly point out the criteria of evaluation for NSD projects, while the three case studies, with the support of both respondents’ words and the use of companies’ documents, are conducted to give a clear idea about the evaluation process. The purpose of combining two different data collection methods is to confirm the findings and to increase the transferability and reliability of the findings. 3.4.1 Semi- structure interview a. The reason Interview is among the most popular method in doing research. Although, there are three kinds of interview which are structured, un-structured and semi- structured interviews, the last one seems to be the most practical and appropriate for this paper thanks to its benefits giving to researchers. While structured interview is more suitable to quantitative strategy, the other two seem to be more useful to the qualitative strategy. Also, differ from structured interview, both unstructured and semi-structured interview are more flexible as they provide opportunity for interviewer to quickly follow the flow of the interview and make change according to interviewees’ interest and responses (Bryman and Bell, 2003). The advantage of these methods is to allow interviewee to answer the questions freely and express their views on particular issues that they think as important which turn out to benefit the interviewer. The researchers in this case therefore would be able to understand the topic more profoundly and explore all aspects of the topic from different points of view. This enhances the validity of the research. Considering the above perspectives, the semi-structured interview however, is chosen as the main data collection tool in this research due to some additional reasons. Firstly, it allows us to have a guideline along the interview process which helps us not to deviate from the initial track. By using this method, both interviewers and interviewees are concentrated on the research topic and discuss around that topic. The information gained from the interviews therefore specifically contributes to the purpose of this research and supports researchers to build up theory based on data collected. Secondly, as this ‘research is carried out by more than one person; the semi-structured is more preferred to ensure a modicum of comparability of interviewing style’ (Bryman and Bell, 2003:346). Thirdly, the more structure characteristic of semi-structured interview in comparison to unstructured ones enhances the possibility to compare multiple cases in different
circumstances. Since the research is taking into account the diversity of culture, geography and the variety of sector’s nature, the semi-structure interview is especially important. Our aim is to get insights into the interviewees’ experiences and what they see as relevant and important to the specified topic.
b. Selection of interviewees The selection of interviewees is carried out by looking at two elements:
Interviewees must have a role in the project evaluation process The projects that interviewees evaluate must be NSD
Many difficulties arisen prior to the interview process regarding the possibility to approach potential interviewees, especially project managers. Besides, the title ‘project manger’ or project evaluation team are not explicitly used in some non-project based companies. Under such challenges, the interviews were given to not only project mangers of NSD projects but also the evaluation team members as well as product managers and other managers who responsible to evaluate the projects of launching new services. Since the availability of respondents is crucial for the validity and success of the interview process, personal contacts were used to set up new contacts to further more relevant interviewees. This was suggested by Bryman and Bell (2003) as snowball technique which improves the credibility of data collected. Besides, a contact was collected and established through a career fare. Also, with the support of our supervisor on some local contacts, the list of interviewees was expanded. The sample included people from different countries such as Albania, Italy, Sweden and Vietnam. Although, they are working in the same service industry, their companies focus on different business area such as financial service, information, airlines, real estate, etc. The diversity of nationals and sectors allows researchers to look at many different perspectives of the topic and enhance the generalization process of the chosen inductive approach. The names of interviewees and some companies are not indicated in the list of interviewees as respect to their preference of anonymity. The position and experience of interviewees as well as the business areas and countries based of the companies were used instead.
Table 3-2 List of interviewees No.
Country Product Manager IT Manager
Investment Council Member Project evaluator
Energy/Consul ting service
Project Manager Senior Project Manager
1.5 years 13 years
Email; Telephone Email; Telephone
Product Manager Project evaluator
4.5 years 6 years
Banking Real estate
Email; Telephone Face- to face
35 mins 60 mins
Consulting manager Product Manager Steering Committee Member Expert of monitoring and controlling PM PM – Sponsor institution
Financial consulting Airlines Banking
Face- to face
Emails; Telephone Face to face, secondary data
50 mins 40 mins
Face to face
2 years 5 years
Face to face
Telecommunic ation Computing service Education service
3 years 2 years
c. Questionnaire formation The questions were formed based on the knowledge gained through literature review and followed closely the purpose of the research as well as the research question. Besides, the process to form questions for interview recommended by Bryman and Bell (2003) was taken into consideration. After the discussion among authors, the initial questions list was developed. In order to improve the feasibility and appropriateness of the questions, we sought for our supervisor’s advices. The available time of interviewees, their willingness to answer the questions and the proper time spending on each chapter of the interview was emphasized by our supervisor resulting in the revision on the first question list. Changes 34
have been made on the number of questions, the wording and excluding or including some questions to ensure the practical aspect of data collection process. The final questionnaire is attached in Appendix 1. The question guide was divided into small chapters to improve the clarity and communication among interviewees and interviewers. Four parts of the question are organized as below:
General information New service development (NSD) projects Project evaluation in NSD projects Final questions
The questions move from general to more specific area and end up by final questions which specifically support to the answering of the research question. Most questions are open-ended and provide respondents chances to express their views and experiences on the project evaluation process which turns out to improve researchers’ knowledge on the practical perspective of the topic and contribute to the current body of knowledge. The third part starts from some common questions and followed by others which would be varied depends on the specific phase focused by each company. By doing this, the researchers could see which part of project evaluation is more emphasized in practical world, especially in service industry. In the last chapter of the questions, the interviewees were asked for the difficulties during the project evaluation process which imply the important factors that influence the success of the process and the critical evaluation criteria. Besides, that the proposition criteria list were given to the interviewees and asked for their opinion ranging from the least to the most important factors helps to keep the interviews on track. Along the interview, the order of questions and the number of questions are flexible to react to the flow of the conversation. Some new questions would be raised by interviewers to clarify the answer as well as to explore the issue further. Also, changes would be made along the process together with the learning curve on how to frame, phrase and the order of the questions. d. Interview process The interview procedure followed the suggestion that the ‘dual approach’ should be applied in business and management research (Healey and Rawlinson, 1993 in Bryman and Bell, 2003). First, we made phone calls or sent emails to potential interviewees. Second, a letter of introduction about our research, the purpose and process has been sent to the respondents to follow up the process. Later, a telephone call or email was made a few days after the receipt of the letter to ask for their attendance on the interview process and set appointment. The interviews were carried out on the appropriate time to the interviewees. 35
The length of interview varied from an interviewee to another. However, it is restricted in roundly an hour as time is value to respondents. Many different tools of interview were applied according to interviewee’s preferences and time. Some interviews were undertaken in person while some others were carried out via phone or email. The interviews through telephone were mostly used due to budget constraint and the diversity of geographical locations of each interviewee. Only respondents from Albania were face-to- face interviewed as one of the researchers came back to collect data. All interviewees received emails with the interview guide prior to the telephone or direct interviews to facilitate the interview process and to save time. Interviewees hence, would have thought about the topic and have some draft idea of their answers that helps them response to the questions quicker. This action also allowed us to reduce misunderstanding between two parties. Some interviewees even asked to send the answers by emails first and only short interviews were conducted afterward to clarify any doubts. Understanding that taking note might distract our attention leading to the loss of information, voice recorder was employed during the interviews. Also, the recorder device helps us to minimize the human errors in memorizing and interpreting the answers. To ensure that there is no objection from the interviewees, we asked the respondents for their permission to use record device before starting to interview. The interviews were then transcribed right after the interviews for further data analysis. The summary of interviews was developed and analyze in chapter 4 of this paper. 3.4.2 Case study The case study method has been increasingly used as research tool. According to Bryman and Bell (2003), it provides deeper examination on the topic and gives more insights to the various aspects of the issue that mentioned by different interviewees in the same organization. Three cases were carried out by having more interviews with other person in the organizations in order to understand the evaluation process more thoroughly and get deep inside the companies from the different points of view. Besides, some sources of secondary data provided by interviewees, from the companies’ websites, their publications and press releases, etc, are employed in support to the analysis of the cases. The reason why we chose to study the case of these three companies is because of their good structure of project evaluation and the qualification and deep knowledge of persons dealing with project evaluation. They are big companies, well organized and with a strong project management. These companies have clear procedures of evaluating projects and they were available to provide us interviews and secondary data regarding project evaluation.
The three cases are:
CitiGroup Inc., United States:
CitiGroup is an international financial corporation. Its head quarter is in New York, United State and it has branches all over the world. The company employs 350,000 people and has 200 million customer accounts across six continents in more than 100 countries. Citi is organized into four major financial service segments – Consumer Banking, Global Cards, Institutional Clients Group, and Global Wealth Management. Every year, it spends millions of dollars for developing new services to their customers. It is famous as the best IT banks in the world providing customers with many electronic financial products, one of which is CitiDirect, a very convenient way of accessing and proceeding transactions online. The case study is carried out in the Vietnamese country branch. This countries branch has two offices in the two biggest cities in the country. It provides banking service only to corporate customers. The branch has a team to develop new service all the time to provide its customers with appropriate services. Although there are some legal and cultural differences, it follows the same procedure of the whole corporation in all of its activities. The project evaluation process for NSD projects is not an exception.
Luftfartsverket (LFV) Group, Sweden
LFV is a Swedish state-owned company organized into three different state enterprise divisions: Stockholm, Group Airports and Air Navigation Services. LFV is employing over 4,000 people, operating in 16 airports, serving approximately more than 2.7 millions passengers internationally and with annual revenue of approximately SEK 6 Billion. LFV operates and develops State-owned civil aviation airports; air navigation services for civilian and military aviation. Apart from the traditional services- air navigation, the group is investing in some new services such as mobile service where passengers receive messages about the detail of their fight prior to their departure. The new service they are offering, together with other new services, is aiming at facilitating the passengers’ travelling and giving them more convenient way of travel. The project is developed under a tight but flexible evaluation procedure to ensure the service is launched successfully, gains customers ‘satisfaction and increases business value of the company.
E.ON Group, Germany
E.ON Group’s Corporate Centre is in Düsseldorf, Germany. It is geographically and functionally in market units many countries in Europe and Russia but also in North America. The company has almost 88,000 employees and with annual sales of just under EUR69 billion. E.ON is the largest investor-owned energy service provider. They currently provide reliable power and gas solutions for around thirty million people in almost thirty countries around the world. They offer tailor-made energy services and energy-trading solutions to small and medium-sized companies and plants. Although the company is not working in the service industry, it is still considered within the scope of the thesis as the company is offering service solutions to the customers. This is called new offering development (NOD) which was mentioned in the definition of NSD in the chapter 2 as a type of NSD. The case study is undertaken in the Swedish branch. E.ON Sverige offers residential customers in Sweden electricity, natural gas and district heating. Like the case of Citigroup, E.ON Sverige applies the same project evaluation process for NSD with its head quarter.
4 DATA ANALYSIS In this chapter, we perform data analysis for our empirical work, while leaving main findings and discussions to the next chapter. Briefly, we conducted our empirical work using semi-structured interviews with fourteen companies and selected three of them to develop further into case studies. In the case studies, we used both primary data collected through semi-structured interviews with respondents from three companies and supplementary documents to illustrate in details the process of new service development and of NSD project evaluation. These processes are company-specific, but as we later found out, are also sample representative. In the vast semi-structured interview, we examine the variance between cases and other companies regarding the process of NSD as well as the purposes, the process and techniques of NSD project evaluation. Besides, the interviews also serve to point out some specific aspects including challenges faced during evaluation process and evaluation criteria used in all fourteen companies. The next chapter will base on this chapter, together with chapter 2 to generalize the findings and to provide implications.
4.1. Data analysis from Case Studies This part presents the findings from three case studies (Citigroup, LFV, E.ON) with the purpose of providing comprehensive understanding on the process of NSD and the evaluation process for NSD projects. In addition to the responses from interviewees, companies’ documents such as websites, evaluation procedures, project manager’s evaluation forms are referred to and analysed to enhance the understanding of project evaluation process for NSD projects. By summarizing both primary and secondary data collected from the three organizations, the section illustrates the following two processes:
NSD process NSD project evaluation process 4.1.1. NSD process
While investigating the three cases which are Citigroup, E.ON and LFV, we found a similar process of developing new service. Despite some minor differences, five main steps are followed by all the companies as illustrated in the figure 4.1 below. The product or project managers of these companies agree that the customers’ needs trigger their idea of the new service. Hence, there is a tight interaction between customers and the project team. In the case of Citigroup, each Relation Manager (RM) discuss individually with their customers; understand customers’ needs and propose the idea to the 39
country product manager (CPM) to develop new service. Since, E.ON also deals with only corporate customers, the NSD process is the same. The process of developing new service in LFV however is slightly different because the number of customers is large and the targeted customer was not identified since the beginning. Thus, market research and the target group of customers’ identification is the first step to be undertaken. The idea therefore, comes up through customers surveys. Other activities are alike in all of the cases. Once the idea is agreed to be developed, the project team is formed. The team members come from different departments and work toward the common goal which is the successful launch of the new service. At the same time they have to estimate costs and benefits related to the project as well as ask for fund which normally is given by the management board. In addition, one of the key activities in developing new service is testing the service fit with internal environment of the company on one hand as well as fit with external environment (such as customer’s requirements or company’s situation in the marketplace) on the other hand. In terms of internal fit, the new service needs to be tested to make sure that, for instance, the new system developed by Citigroup does not influence other branches’ systems. In terms of external fit, Figure 4-1 New service development process different methods of testing can be used, such as presentations given to clients as the case of E.ON; another method is organizing simulations as mentioned by LFV; or just communication with customer as stated by the respondent from Citigroup. In spite of having only five steps, this process covers most of activities mentioned in the model recommended by Alam and Perry (2002) such as idea generation, team foundation, analysis, testing . More than that, this model considers the communication plan as a key 40
step while Alam and Perry (2002) did not. There is a consensus among the companies about stakeholder involvement and the crucial role of end users during the development of new service which was also emphasized by previous researchers (Kelly, 2000; Grönroos, 1990; Dolfsma, 2004). 4.1.2. NSD project evaluation process a. Purposes Through the study of the three cases, we found that there are two groups of people (1) Project Sponsor and (2) Project Manager doing evaluation with different purposes. The senior project manager of LFV, the Country product manager of Citigroup and the Investment Council member of E.ON agreed that it is necessary to be sure that the new service they are launching is realistic, is exactly what their customers want, is worth with the fund that the project sponsors invested and finally, would enhance the organizations’ business value. ‘The wrong investment will not only cost us a lot of money but also negatively influence our brand’s value. Therefore, we need to evaluate the worthiness of the proposals all the time’, the respondent from Citigroup said. Further elaborate on this, the respondent from LFV mentioned that they evaluate project to prioritize the project portfolio and choose the best project to develop. Also in this case, we found documents showing that the management board and the Steering Committee look at the project managers’ performance during the implementation of project as a reason for their evaluation process. When being asked about the purposes of project evaluation for NSD, the project managers of LFV emphasized the need to evaluate projects during the implementation period in order to keep track the project with what stakeholders expected and to ensure the projects is going aligned with the companies’ mission. The evaluation procedure is undertaken all the time during the implementation of project so that project managers would be able to control the direction of project as well as to foresee the possible issues and take quick response. Moreover, evaluation allows company to understand what customer wants and adjust the new service to fit well with their demand. At the same time, as the service is new to the organizations, it is of special importance to continuously evaluate the project to ensure that the project is not going away from the expected results. For E.ON’ project manager, evaluation is the initial step to monitor, to manage risk and to prepare for corrective actions. While the respondents from Citigroup mentioned the purpose of their company in evaluating NSD projects is to see if the new launch would not influence the system and the procedure of other branches. Also in Citigroup, since the project team members are not full time working for the projects, the level of motivation is different. Then, evaluation is to see their commitment to the projects and provide incentives for their productivity. 41
In all cases, project evaluation is carried out by both groups for a long term purpose – organizational learning. The knowledge transfer is what the companies are looking at while evaluating NSD projects even after the implementation. b. Process Based on the purposes of different stakeholders in evaluating NSD projects, this section is divided into two parts elaborating the evaluation process from the point of view of two groups: Project sponsor and Project Manager. While the former group performs evaluation process more formally, the latter is more flexible in doing evaluation. This is due to the nature of each group’s targets. I.
NSD project evaluation- A perspective from Project Sponsor
Project Sponsor and/or Management Board are involved in project evaluation from the aspect of decision making. Their view is more at the strategic level and their objective in project evaluation is to look at the long term benefit and development of the whole organization. They carry out the project evaluation prior to the implementation (Phase 1) and during project implementation as well as after the project completion (Phase 2).
Prior to project implementation Purposes: Priorities projects portfolio; Make decision whether to develop a project
During project implementation and after project completion Purposes: Assess project manager’s performance; Verify project performance
Figure 4-1 NSD project evaluation carried out by project sponsor
Phase 1: Investment appraisal The investment appraisal is undertaken only by the Project Sponsor. In this phase, the Management Secretariat performs a preliminary assessment of all the proposed projects, and come up with a list of priority projects for the Committee consideration. Several meetings among the Steering Committee and the project owners are held. After the presentations of the initiatives, a group of senior analysts and the heads of all departments are gathered and start doing the evaluation for the remaining projects. They first determine the current target of the organization, then analyze if the new service fit well with the corporate goals. The figure shows different levels that the group will look at while analyse the strategic fit of the project. Two questions that are always asked: (1) Why is this to be done? And (2) How is this to be done? If the group find the specific answers for these two questions along the analysis base on the proposals, then the proposals passed the first evaluation level.
The next step is to study the feasibility and see if the new service is realistic and suitable with customer’s expectation. The market impact and stakeholder analysis are carried out at the same time to understand the level of interest of different stakeholders to the new service. Other analysis such as financial, capacity, potential issues, assumption analyses are taken on in order to be sure that Figure 4-2 Strategic fit analysis framework the company is able to implement the project under both internal and external constraints. During the evaluation process, the group use SWOT analysis, determine also the negative and positive forces that might influence the achievement of the project objectives to choose the potential projects. There are some questions being asked all the time to enhance the decision making process: -
What are the overall and specific objectives that the project intended to achieve? Is the new service realistic and can we win the market? Who will be benefit from this?
How long will the project take? How can the benefits guaranteed? Which risks should be taken into consideration? Do we have enough fund and resource for it?
Base on the results from all of the above evaluations, the Management Secretariat justifies the exclusion of project proposals from the list for the Committee consideration. The Committee will discuss and approve projects based on their suitability for the organisation’s strategies, their relative importance, their impact to the market and to the firm’s development and having in mind the need for a balanced distribution of projects across the action lines. The soft results like the image of the company, competitiveness and customer satisfaction, etc. are also considered but it is difficult to estimate such affects in this phase. Phase 2: Performance Appraisal During the implementation of the project and afterwards, the Project Sponsor do not do major evaluation. In this period, their role is to set milestones to the project and to evaluate the project manager’s performance and the impacts of the new service on the company’s strategies and competitiveness. The main evaluation criteria are defined as well as the sources and means of verification are determined at the beginning of NSD projects. The Project Manager has the responsibility to keep the Project Sponsor updated on the progress of the project. The 43
Steering Committee analyzes the weekly or monthly reports (depending on the dimension of projects) from the Project Manager. The reports need to show at least the three key criteria of the project which are time, cost and quality of the project. All the reports must be cleared, must have supporting documents and could always be double checked by internal or external audit. In order to ensure the validity of the data, The Reference Group is assigned to go along with the Project Manager as an external party during the development of new service. On one hand, they evaluate the progress of project according to the initial agreed indicators and criticize the results of each work package along the projects. On the other hand, the Reference Group will work more with other stakeholders of projects, communicate with clients, with suppliers and sponsor or regulatory parties and get comments from them regarding the new service. Therefore, this group also acts as facilitating factor by giving recommendations or solutions when needed. The Steering Committee and the Reference Group will evaluate the projects in relation to the overall project portfolio and decide to continue with the project or not. As mentioned by the respondent from E.ON, if the project is running out of budget and becomes more expensive than the estimated number around 5 to 10%, it would be stopped or waits for a new decision to be approved. In most of the cases, the Project Manager’s performance is assessed during this period of time as well. The sponsor will evaluate the project manager’s performance by looking at not only the soft skills such as the relationship with project team, the ability to solve problems, the visionary view, but also the hard skills like the planning, organizing or monitoring ability, etc. The evaluation tool of the Project Sponsor during these two phases is mainly through the communication channel and the report mechanism. II.
NSD project evaluation- A perspective from Project Manager
The Project Manager is responsible to evaluate the project during the implementation of NSD and afterwards. The process is divided into two phases: Progress Evaluation (Phase 1) and Project Review (Phase 2) as following:
During Project implementation Purposes: Set basis for monitor mechanism; Keep track with goals; Take quick response; Improve commitment to staffs
After project completion Purposes: Improvement and Learning
Figure 4-3 NSD Project evaluation carried out by project manager
In order to facilitate the flow of the research, we found as appropriate to use the model (illustrated in Figure 4.5) adapted from a case study to discuss the evaluation process in conjunction with the process of NSD projects. The justification for this choice is that we found from all the cases, project or product managers are doing evaluation in flexible ways due to the changing environment and customers’ requirements. In the three companies, there is no separate evaluation team and the project/ product manager has the responsibility to evaluate the project all the time. Hence, it is very difficult to distinguish the project implementation and the evaluation’s process. Although, some different evaluation criteria and tools are utilized by different companies, the general process is alike in all of the cases. The chart in Figure 4.5 shows four major evaluation steps going along four groups of activities that are undertaken throughout the NSD projects. The activities illustrated in the circle in the middle of the graph presents the key evaluation steps while the four squares include four main groups of activities carried out before and after the launch of new service. There is a cause and effect relationship between each group of activities and between different evaluation steps which turn out to help the project manager to ensure the progress of their NSD projects. The circle in the middle of the graph shows the iterative cycle of project evaluation promoting knowledge transfer.
Figure 4-4 Effect Management chart (Adapted from LFV)
Phase 1: Progress evaluation 1) Preparation There are three main development activities in this preparation step which are (1) analysis customers’ demands; (2) identify organization’s benefits if the service is launched; (3) match the customer’s needs and organization’s goals. The process starts with the common purpose of the project, and then the project team decides the target customer groups; determine the goals of these target groups (what they want and their expectation). At the same time, the evaluation activities begin from the last development activity – effect mapping. This could be done using demonstration, workshop, interview and observation to filter and fit the two objectives of customers and of the organization. Once the purpose is 45
determined, the project team and the project manager discuss and agree on the specific objectives of the projects. Later on, the project manager identifies evaluating criteria and selects indicators for further steps, based on the specific objectives and the project context. 2) Quality Assurance This is a special important step in the NSD process where the new service is being designed and created. The evaluation process is therefore mainly from the customers’ point of view. The evaluation of end users is crucial during this phase which allows the project team to develop the service according to customers’ expectation. In order to ensure the success of the new service, all the respondents agree that they will go out and talk to their customers and assess the idea of the service through questionnaires or survey, with their customers. . This activity aims at estimating the level of customer’s expectation on the new service. This called for a communication and interaction plan. The project manager of LFV said that: ‘We do not decide this is exactly what we want. Instead, we use two week iteration evaluation process… Every two weeks, we go out, give presentations, demonstrate to customers about what the team have been found and ask customers about their feeling... customers give feedback and comments; the team come back and work on the new requirement and keep doing like that…and hopefully at the end we would offer our customers what they really want’ Such cycle of customer’ requirement, design, evaluation and design helps project team to assure the quality of new service and to avoid the time and money wasted for the amending activities. Besides, the system checking is necessary to ensure the news service is running well. In this step, the Product Owner creates a ‘To do list’ and priorities the work package according to the market demand and customer’s change overtime. This allows flexibility which is critical when developing new service. The timelines and information channel is also set for further evaluation activities. 3) Implementation The evaluation process in this step is to provide basic information for the control mechanism of the project. Three major evaluation activities are undertaken during this period. The first and second activities are carried out in parallel. Firstly, daily meeting between Project manager and project team are held to keep the project manager updated on the team member’s performance and progress. The meetings are normally last for less than one hour. Three questions are given to each team member: (1) What have you done since the last meetings? (2) What will you do between now and the next meeting? 46
(3) Is there anything preventing you from doing what you have planned? The first two questions allow the project manager to have a detailed idea of how the project progressed and to maintain focus on the purpose. The last question raises current and potential problems that might negatively affect the movement of the project. The project manager then will evaluate the issues, discuss with related people to find the solutions. If the problem is over the capacity of the project manager, it will be escalated to the Reference Group and Steering Committee for recommendation and higher level of decision. Secondly, the project manager has his/ her own data to validate the information and keeps update the data according to change and on the basic of cumulative progress. He/ she also uses the project software, progress graph, financial data to assess budget, the time remaining and the resource available for the remaining work of the project. He/ she looks at the effect of each work package on the market and compare with the initial indicator as well, to see if the project is on the right track and if change need to be made. Thirdly, after each milestone, there is a big evaluation meeting with the team to assess the team‘s performance, level of knowledge and give motivation for the next milestone. Among three evaluation activities in this phase, two of them are based on the ability of the project manager to interact and get feedback from the team. This is to say, communication is a very efficient tool of evaluation and it helps project managers to avoid risk and prepare for corrective action. Phase 2: Project review The remaining work after the launch is to measure the project results in comparison to the initial evaluation criteria. There are two types of results that should be evaluated at the end of project to see how the project went. •
Hard result: Revenue, Cost, Volume, Return on investment. In order to assess such results, Project Managers base on financial reports at the end of the first launch to see the difference between the actual and the estimated results, how difference it is, and which side the differences is (positive or negative). The evaluation period for this kind of results could be short.
Soft result: Customer satisfaction, the competitiveness and the image of the company. Unlike the hard result, project managers find soft result a real challenge for their evaluation job. The customer satisfaction evaluation could be achieved through workshops, customer surveys as doing by LFV or by communicating directly with customer as done E.ON or corporate banks like Citigroup. Sometimes the industry indicators and the company’s share price are used to see the 47
commercial impact of the new service on the market and on the position of company. However, it always takes longer time to evaluate this type of result. In the mean time, the internal review is undertaken with the presence of the project sponsor, the team, the reference group to get recommendations and generate lessons for the next project. The project manager can also talk to heads of other departments to see the project in relation to the overall procedure and system of the organization.
4.2. Data analysis from Semi- structured interviews This section presents data collected through semi-structured interview method. From the interview guide, we choose six most important questions that contribute to achieve our research objectives. These questions illustrate data related to purposes of evaluation; evaluation process; and criteria of evaluation for NSD projects. Questions (1) to (4) analyse the information obtained from eleven interviewees, excluding those from the three cases. This is because the same interview questions were given to respondents from cases which serve as primary data for the analysis of previous section, section 4.1. The results from these four questions help to cross check the credibility and transferability of data collected from cases regarding the first two purposes of our thesis: understanding the purposes of project evaluation for NSD and developing the process of NSD project evaluation. This choice also helps to avoid repetition along the chapter. While questions ((5) and (6) present the information obtained from the interviews conducted with all respondents (both cases and 11 other respondents). These questions allow us to achieve the last objective of our thesis which is: pointing out the evaluation criteria applied by companies. 1) Could you please tell us about the process of developing new service in your company? This question was given only to the product manager of the project who is directly involved in the development of new service. Thus, the answers are only from nine interviewees instead of eleven. While asking about the process of developing new service, six out of nine respondents mention similar activities. A respondent from an airlines company operating in Vietnam said that they start with the market research to identify customers’ needs. Another respondent added ‘The board of management will evaluate the idea and once it is approved, the team is formed. The team members are from different departments’. However, two consulting managers of consulting firms in Sweden and in Albania mentioned that the initiation of a new service often comes from their client, the client will approach the company and talk about the idea. The consulting managers will evaluate the 48
credibility of the client and the capacity of the firms and arrange a group to develop the service. Only the project manager from the secondary school ‘Gjon Buzuku’, in Albania said that their responsibility is just to develop the idea from their sponsor. The remaining steps which are estimation of cost, resource arrangement, interaction with customers and testing the new service are undertaken by all companies that the respondents are working at. The project manager of an academic institution in Italy said that they have to estimate the cost generated from the project and see if the budget is available. The project managers of Kaupthing bank, the IT Company in Italy and AMC – a telecommunication company in Albania, mentioned that the testing activity is especially important for NSD projects. In the telecommunication company in Albania, the team works closely with the technical division to resolve problems. While Kaupthing bank’s business development manager communicates with her customers to ensure the quality of new service offered. In general, we saw the consensus among the respondents about the crucial role of end users during the NSD projects, especially during the design of new service. One respondent from Center of Agriculture Technology Transform in Albania stated that ‘It is necessary to get the end users to try our service and to give comments all the time... their involvement during the development of service would definitely enhance the success of the project.’ Based on the discussion of respondents, there are five key activities during the NSD projects (1) Identify customer needs; (2) Form the team; (3) Analysis; (4) Interaction plan; (5) Testing. However, due to the nature of the business area, the activities are slightly different. 2) What are the purposes of the evaluation process in your company? Project evaluation is undertaken in service companies for various reasons. However, there are some commonalities among the respondents. All eleven respondents mentioned that they do evaluation to align the quality of new service with customers’ expectation from time to time. This is considered one of their primary goals in NSD project evaluation. For example, the project manager of the secondary school ‘Gjon Buzuku’ in Albania stated that they evaluate the project all the time to assess what could be improved in term of educational or social service’s quality to meet students’ future hope. Also, the project manager of the academic institution in Italy said that they try to add more practical aspects in the content of the new program to enable their students to get the job they want after graduation. The respondent from an IT company in Italy added: ‘yes, in each business area, the common need of customers is predictable...But their level of expectation is not easy to understand and unfortunately, it changed overtime’. From other business area point of view, the product manager of the 49
airlines company in Vietnam also share the same opinion that it is better to ‘keep always updated with and be sensitive to the trend of customer’s needs’. Further concern is expressed by the consulting manager of a financial service in Sweden that it is necessary to evaluate project in order to keep track with initial goals and to set basic indicators for the monitoring system. Most of the interviewees do not want to terminate the projects for many reasons such as the amount of money invested (the project manager of a sponsor institution for education service in Albania), their team and their own effort and time (Project manager of Center of Agriculture Technology Transforms) together with their relationship and their credibility with the project sponsors (Project manager of ‘Gjon Buzuku’ secondary school and the academic institution in Italy). ‘It is also the problem of the company’s image and competitiveness in the market’, said the Project manager of Collier International. Therefore they see the evaluation process as a way to reduce such possibility and to save the project. The respondent from a financial service firm in Sweden said that: ‘Because it is not a small investment, it needs to be evaluated to make sure that its actual outcome is what we want. Once the project is started, we would try not to kill it even things changed, we evaluate the deviation and make adjustments accordingly’. The same purpose is mentioned by the respondent from a banking association in Albania who deals also with big investments. They find deviation a better solution when something happens during project implementation. In additional, all nine respondents have the same opinion in doing end project review to see if the project is success or not and to gain lessons. For instance, the project manager of the academic institution in Italy mentioned that they are doing the review to see if the education program that they are providing has any problems, so that they could improve for the next time. The project manager of AMC, Albania concerns also the impediment and wasting time of the project. She said that ‘the purpose is to find the best shortcut which will shorten the procedure and make possible to have a quick service launch in order to be competitive in the market.’ The project manager of an IT company in Italy is a supporter of this opinion. The time to market is however, not a main target of other companies while evaluating NSD projects. On the other hand, four respondents from Collier International, Kaupthing bank, an academic institution in Italy and a financial consulting firm in Sweden mentioned that the project evaluation is undertaken to ensure that the companies chose the right customers and that the companies have enough capacity enabling them to give valuable services to their clients. Although nine out of eleven respondents, due to their positions, did not discuss much about the investment appraisal, they all advocated the previous literature (Day, 2007) that it is necessary to see if the project is worth to do given the novelty of service. However, the other two respondents who hold the position of sponsor or who come from the Steering 50
Committee emphasized very much on this purpose. They stated that their objective of evaluating NSD projects is to choose from their portfolio the most profitable and the most strategic fit project under the resource constraints. One of them said that: ‘Our budget and resource are limited… We need to evaluate project to ensure that our investment is right. This is especially important to new service projects as they are more risky but they generate larger impact and give bigger chance for the company to be more competitive…However, we do not want to hire people and pay more for the project without seeing any feasible outcomes’. In concluding this question, five purposes of project evaluation for NSD are agreed by the majority of respondents: (1) Align service quality with customer’s expectation overtime, (2) Keep track with initial indicators and set basic indicators for monitoring system, (3) Save project, (4) Gain lessons; (5) See if the project is worth doing. 3) Could you please explain the process of evaluation for NSD projects applied by your company? When summarizing the answers of this question, we found that different respondents focus on different parts of the evaluation process. Two interviewees said that they evaluate the NSD projects as a business case before the project start. The project manager of the Albanian Association of banks described that the Steering Committee will read carefully the proposals, identify the aims and the customer group that the project target. They will then, discuss the benefit and the opportunity cost of taking the projects from strategic point of view. Discussing about this process, he continued: ‘it is strictly important to assess the level of alignment between project specific objectives and the overall corporate missions. We can use either the balance score card or tree analysis to ensure the development of new service is not out of our goals’. In the meetings among steering committee, they determine both positive and negative impacts of the projects on various aspects. The project manager of a sponsor institution in education in Albania stated that there is always a group of senior analysts assessing and analyzing the proposals carefully from different perspectives such as cost, benefits, feasibility, market impact, stakeholder interests, etc. and reports the results to the committee. Once again, he emphasized the communication among the committee’s members is crucial leading to the decision. The majority of respondents (nine out of eleven) discussed more about the evaluation process during the implementation of NSD projects. Some of them expressed the serious concern on setting evaluation criteria. The project manager of a consulting firm in Sweden stated that ‘we need to establish a set of criteria at the beginning of the project to have a basis for further evaluation steps’. To support this, the product manager of an airline company in Vietnam said ‘we compare actual results with estimated parameters to see how 51
we are doing’. However, other activities are fairly different among the respondents. Three of them raised an important fact when evaluating NSD projects that their evaluation process is carrying out together with the development of NSD and in very simple way. The respondents from Kaupthing Bank said that they are just talking to customers to get feedback from them and come back to develop the service base on their requests. The project managers of a consulting firm in Sweden and Collier International, the IT company in Italy share the same opinion, one of them mentioned ‘our service is mainly man power, the only thing we have to evaluate is our staffs’ performance and this evaluation comes from our clients. However, as a project manager, I have to keep my own data to double check the validity of the feedback from customers and assess our people base on the contracts’ terms and our internal check lists’. The other two schools from Albania and Italy also look at students and staffs’ feedbacks to evaluate if the project is delivered as expected or not. Some industry indicators are used to appraise the quality of project. However, they are all agreed that customer analysis is important during the evaluation process as it helps them to ensure the quality of project match with customer’s requirement. Besides, ‘internal discussion gives useful information for the evaluation process’ as stated by a respondent from the Italian IT company. Other respondents also support this. On the other hand, the process is more complicated in the case of AMC; the Center of Agriculture Technology Transforms and the airlines company in Vietnam. Project managers of these companies assess the deliverables and time remaining for the project implementation using project software. Beside, the Vietnamese product manager said that ‘I record the cost of the project everyday base on the reports from our staffs and from the financial controller and compare it with the estimated numbers to keep track with the budget of the project’. The AMC project manager give supplementary opinion by saying that ‘ I meet the IT managers everyday to ensure that the system of the new service is running well, otherwise we will have to find solution to fix it right away’. All interviewees said that they review the project in terms of the initial indicators to see if the project is success or not. For instance, the project manager of AMC looks at the revenue actually generated from the new service after the first launch. Or the product manager of the airline company in Vietnam determines the volume of customers using the service in the next month. While the University in Italy use survey to see how many students get job right after the course. Among the interviewees, one of them said that they do not do also the investment appraisal as their responsibilities is only to implement the project while their project sponsor which is another institution will do investment appraisal as well as project review. In brief, the common evaluation activities mentioned by the majority of respondents are: setting indicators, strategic analysis, customer analysis, discussion, actual versus estimated comparison. However, the data collected from the interviews regarding the NSD evaluation process show that there are different ways of doing evaluation along the life 52
cycle of project. Depending on the purposes, the evaluation process is carried out differently by different people. 4) What are the techniques that you are using for evaluating NSD projects? In response to this question, most of interviewees highlighted that it depends on the objectives of the project to use different tools and techniques. Amongst the most common tools mentioned by respondents are positive net present values, greater internal rate of return in compare with weighted average cost of capital, and the payback time. Talking about these financial tools, the respondent from the Italian University said that ‘Of course, these financial numbers are measurable and they give us a clear idea of how big the profit could be’. Besides, the real options analysis is recommended by the respondent from AMC as particularly crucial for innovation projects because it answers the question of how to measure the value of flexibility which very much related to any NSD projects. However, none of the other eight respondents mention this technique during the interviews. Regarding the non- financial techniques, the product manager of the airlines company operating in Vietnam uses SWOT analysis to evaluate NSD projects from four basic aspects: strength, weaknesses, opportunities and threats. Moreover, the Collier International project manager emphasised that they do not use financial tools to evaluate their NSD projects. Instead, they use non financial techniques to evaluate their projects such as communication, survey, workshops. The respondents from another consulting firm in Sweden, AMC, the IT Company in Italy and Kaupthing Bank also brought up the need to use these non-financial tools to evaluate the intangible results. For the sponsor institutions like the Albania association of bank (AAB) and the institution for educational service in Albania, the logical framework for project evaluation is used. The respondents from AAB said that: ‘By using this framework, we could evaluate the overall impacts of the new service, not only to the company but also to the society’. The other project manager put in another concern when applying this framework which is the sustainable success of the project. Besides, the AAB’s project manager also uses the balance scorecard to evaluate the NSD project. However, we saw a consensus among interviewees to combine various tools in doing evaluation for NSD. Besides, it is raised by the respondents that specific techniques could be applied by one project but may not suitable to others. 5) What are the difficulties and challenges in the evaluation process? While asked this question respondents mentioned several challenges and problems that affect the project evaluation process. 53
During our interviews one of the respondents from an airline company in Vietnam said that ‘communication channel is crucial for us when we evaluate projects in our company’. Also the respondent from AMC said that gathering information is a big challenge for them during the evaluation process. She stated that ‘one of the implications of our projects evaluation is information reliability. We pay much attention on the feedback and feed forward process and we care a lot about the reliability of the information gathered’. The respondents that we interviewed from Kaupthing bank in Sweden, the sponsor institution for education service or the Albanian Association of Banks, stated that the major challenge they face when evaluating the performance of Project Manager is to get the honest comments from the project team. However, in one of the case, the project team members are very expert and powerful and therefore, they are very straightforward in evaluating their manager. . Another difficulty added from a company in Vietnam offering banking services, which is a multinational company operating in many cities and countries, is the fact that company structure and procedures complicate the evaluation process. The respondent from this company stated: ‘Most of our new services are related to IT…When a project is developed, we have to ensure that new system would not have negative impact to other countries’ procedure’. However this fact is not noticed by other big companies such as the Swedish airline and EON. The respondents from EON (as also mentioned when analyzing this case study in section 3.4.2) stated that even though they operate in five different regions, they normally have the same investment council doing evaluation for projects in different countries. Nature of the project were mentioned and emphasized as a problem by some of the respondents from companies such as EON, AMC and the airline company operating in Sweden. One of the respondents from EON stated: ‘we find no unique elements to measure in NSD project, most of the criteria we take into consideration when we evaluate our NSD projects depend very much on the nature of the service offered’. The respondent from AMC added that projects are launched from different departments regarding the nature of the new service offered and different departments use different criteria of evaluation. Some respondents stressed that evaluating soft result is not easy, more challenging and there are no standard way to evaluate such result. The respondents from both the airport company in Sweden and the airline company in Vietnam expressed their concern on this issue. One of them stated that ‘it is difficult in new service projects to measure soft results such as prestige, competitiveness and social effects’. The project manager and project sponsor of a secondary school in Albania supported this when saying that the social impact
is important for their project, however they find it not easy to measure such impact and normally it takes quite long time to see the real impact. Intangibility as the main characteristic of services was added to the list of challenges mentioned by respondents. One of the respondents from an airport company in Sweden stated that ‘it is difficult to evaluate NSD projects because of sentimentalism, subjectivity and feelings or mental and spiritual state of customers in the specific moment when asked about the service offered’. The same problem was emphasized also by the interviewee from the banking company in Vietnam who stated that ‘it is hard to measure the invisible value in NSD projects’. While the respondent from a Swedish airline stated: ‘Our projects are mainly related to IT services and because of their fragile nature we face difficulties during the evaluation process’. Customer involvement was stressed as the biggest difficulty by all the respondents. All their opinions about this challenge can be represented by the statement made from the respondent of Kapthing Bank who said that ‘it is really difficult to get customer involved and most of all get their honest evaluation’. The challenge of risk assessment was brought into attention during the interview with the Investment Steering Committee member from EON. He stated that they have to involve a company that offers risk assessment services. He further added ‘our projects are big and last log and they are associated with a high risk. Risk assessment is difficult for us as it includes a lot of costs and is related to many unpredictable factors such as the exchange rate tendency for long periods of time’. This difficulty was emphasized also from the respondent of the banking sector in Vietnam, as risk is always a big concern to banks. However, the respondent from Collier International operating in real estate’s sector said that the company is not facing any considerable challenge during project evaluation. She further stated that ‘usually it is an easy process for the company to evaluate and select the appropriate projects’.
6) Do you recommend a set of evaluation criteria that you use for your work? Interviewed companies have similar criteria of evaluation which they consider as ‘set of criteria’ in project evaluation, but the importance they give to each criteria vary from project to project based on the nature of service offered. This question generated the following information regarding criteria of evaluation for NSD projects. Profitability is considered as important criteria of evaluation from the majority of respondents. A respondent from a bank in Vietnam said that ‘We only accept the project that brings profit to us. There is no reason to do business without having profit’. To 55
support this, Collier International respondent in Albania states that ‘the less risky and the more profitable the project, the more chances the client has to be selected’. Furthermore, the respondent from EON said that they look at incremental growth from year to year in an investment in order to measure the profitability the company will have from a certain project’. Moreover the respondent from a company in Sweden dealing with airport services said that they are ‘selecting projects benefiting from scale effect already now or in the future’. In addition to profitability criteria, Production cost is mentioned to be a relevant criterion that is taken into consideration during project evaluation. The respondent from the academic institute in Italy claims that ‘we can select also projects that are not much profitable just because they are asked from important clients for us, but at least the project revenues should cover the costs’. One of the respondents from EON stated that ‘we get in principle three criteria: revenues, production costs and investments’. The respondent further stated ‘These operational criteria drive us in a financial analyses by accounting complete profit and loss, preparing the balance sheets and cash flow analyses for each years of the project life cycle. Moreover, key figures are calculated and the most important ones are presented in Summary outputs report including graphs as well’. Another criteria mentioned by respondents is the Return on Investment. Mainly the rate of return on investment is calculated by companies by using methods such as pay-back period, present value after initial investment, Nominal Internal Rate of Return post tax, etc. All respondents from fourteen companies agree on these above financial indicators as a tool to evaluate project. In addition to these methods, the respondent from EON said that they use also indicators such as Weighted Average Cost of Capital (WACC) or the Return On Capital Employed (ROCE). Although, one of them said that ‘it depends on the specific goal of each project, the main investment criteria are chosen’, another respondent stated that ‘it is return on investment shows the real profit of the project to shareholders...it is necessary to consider this aspect’. Some other respondents also mentioned this point. Strategic fit is one of the most important non financial criteria which were mentioned from all the respondents. The respondent from the academic institute in Italy stated that ‘we cannot undertake all projects offered, it should strategically fit to our strategy, objectives and goals as well as the activity of our institution’. In addition the respondent from the Centre of Agriculture Technology Transform claimed ‘we have to make sure that our projects are in alignment with our strategy and they fulfil also the international monitoring standards our country adopted ’. Furthermore, scope is emphasized by the respondent from Collier International in Albania that states ‘the project should be within the scope of work of the company as well as should not have conflict of interest with other departments’. Resource management is added as part of the strategic fit criteria. The respondent of the academic institute in Italy claims that ‘it is very important to control and keep within the budget. Never ask for extra budget to the project sponsor’. He continued to 56
stress the importance of resource management by saying that ‘we cannot undertake a project if they do not have available staff or enough qualified staff’. Moreover, respondents from the interviewed companies argued that management skills play the main role and project management quality is taken into consideration on the evaluation of projects during their whole life cycle as a strategic issue. As for its importance, one of the respondents from the secondary school ‘Gjon Buzuku’ in Albania said that they are considering to involve in the project an external consultant who would help with advices on developing and re-enforcing internal management. Respondents stress the importance of an effective project management related to resource management, staff appraisal, project activities, quality of project and risk significance. In addition, the respondent from Collier International stated ‘we are careful on managing our human resources. We use man power as main indicator to evaluate their involvement in project’. Other criterion of evaluation is customer satisfaction and involvement which was mentioned explicitly by twelve and implicitly by the remaining four respondents. In fact, all of them came up to the same position regarding this criterion. One of the respondents from EON said: ‘We put our customers first and by doing so we provide excellent service and value. We currently provide reliable power and gas solutions for around thirty million people in almost thirty countries around the world’. The project manager of an airport company in Sweden emphasized all the time during the interview that this factor is crucial in both the development of NSD project and in the evaluation of NSD. In the IT Company in Italy, the involvement of customers into the testing period is necessary to see if the new service fit well with the requirement of customers. Environmental concerns and Corporate social responsibilities (CSR) is another criteria mentioned by some respondents such as EON, FBSH, Centre of Agriculture Technology Transform. They commented on this criterion as a concern related to environmental issues, job creation and governmental regulations. Environmental concerns were especially noticed during the whole interview with the respondents from EON and Centre of Agriculture Technology Transform. EON mentioned ‘the company is very much concerned about climate protection and always keeps in mind to be ‘environmental friendly’ while selecting in which project to invest’. Also the respondent from the Centre of Agriculture Technology Transform said ‘we are primarily concerned with environmental protection’. Regarding job creation EON stated ‘when we undertake our projects we also think about our responsibility to create jobs for the community’ and the sponsor of education service in Albania said ‘we measure our project based on how many contracted members it was able to involve’. Regarding governmental regulations the respondent from the Centre of Agriculture Technology Transform stated as mentioned also before ‘…we have to fulfil also the international monitoring standards our country adopted’. Information quality was mentioned from five companies such as EON, the Swedish Airport Company, AMC, the airline company in Vietnam and IT Company in Italy, as an 57
important criterion that they take into consideration while evaluating projects. The respondent from the airline in Vietnam claimed ‘I would like to emphasize that we are very careful about the information obtained during the feedback process from our customers and we pay attention on how we transmit this information to different levels on the organization.’ While the respondent from IT Company in Italy stated ‘we select projects based on customer needs. Therefore we are very much concerned with accuracy and relevance of the information we gather from the market research’. Stakeholders’ involvement and their attitude towards the project is considered as a facilitating factors from ten respondents from both case studies and semi-structured interview. However, the respondent from the Centre of Agriculture and Technology Transform stressed ‘we have very strong relationship with our stakeholders and they help us continuously with new regulations issued from government or agreements regarding international standards of monitoring’. Moreover, relationships with stakeholders are mentioned as important criteria by almost all the respondents but some respondents from companies such as the IT Company and academic institution in Italy, the banking company in Vietnam, and the Centre of Agriculture Technology Transform in Albania, mentioned particularly the importance of Improvement of Relationship with stakeholders. The respondent from the IT Company in Italy said ‘Well, we do have cases that we select projects because our clients need additional services even though we don’t profit much from that particular project. However, we see our benefit in this case is long term, related to customer relationship improvement’. In addition the respondent from the banking company in Vietnam stated ‘…by launching new services we try not only to gain new clients but also improving our relationship with the existing one’. According to the respondent from the academic institution in Italy ‘effective project management should take care in solving problems promptly with stakeholders for better relationships in the future’. Also the respondent from the ‘Centre of Agriculture Technology Transform’ commented that the relationship with stakeholders, especially project sponsor, is a determinant factor. Image is also considered by most of the respondents an important criterion of evaluation. The respondent from academic institution in Italy stated ‘a good image of the project will increase the number of applications which in our case means that the project and the service offered through it is good and attractive for our new clients’. In addition the respondent from the Asian airline company stated ‘we take care of our prestige when we select projects’ and the senior manager from the airport company in Sweden added ‘we are very much concerned with soft results of the project and we want to achieve a higher image of our company from one project to the next one’. The respondent from AMC further commented ‘from the brain storming with all employees we chose only new ideas that will be transformed in projects which will improve the image of the company’.
Credibility of customer is strongly taken into consideration by companies such as, Collier International, a company offering banking services in Vietnam, a company operating on consultancy service sector in Sweden and another IT company in Italy. One of the respondents from EON stated: ‘we are doing big investments of long duration to a specific client and it is crucial to evaluate how much credible the client is. We have to make sure that the client is not going to be bankrupted soon’. On Time Project Delivering was mentioned during the interview with the respondent from an IT company in Italy who stated ‘we always keep in mind that we should deliver on time the new software to our client. But we also evaluate if the time spent to deliver a service to our client is worth for the profitability the project will generate’. But he further added that the criterion of time is also related to the fact how urgent a need is and the time available to the company to fulfil the specific need of the customer.
5 FINDINGS AND DISCUSSION The structure of this chapter is developed based on the three main purposes of the study mentioned in chapter 1.
To understand the purposes of project evaluation in respect to NSD projects To develop the process of evaluating NSD projects To identify the set of criteria that service companies consider while doing evaluation for their NSD projects
This chapter discusses the findings collected from both case studies and interviews while referring back to the theory in order to give implications and find the answer for the research question which is: ‘What are the key evaluation criteria for New Service Development (NSD) projects?’
5.1. Understanding the purposes of project evaluation in respect to NSD projects Through the study of the three cases and the interviews, we found that there are two ways of looking at the purposes of project evaluation for NSD. The first one is to look at the project from the sponsor’s point of view. This group conduct evaluation in order to pursuit the following three purposes: (1) To appraise the worthiness of the investment or to prioritise project portfolio. (2) To assess the performance of the project manager and to ensure the project is on the right track. (3) To enhance the organizational learning. All the data collected from both cases and interviews agree on the above three purposes. Besides, the previous literature also pointed out these three objectives of organizations when doing evaluation for NSD projects (Ye and Tiong, 2000; Mohamed and McCowan, 1999; Chapman, 2005). Besides, Wholey et al. (1994) emphasized the main aim of project evaluation is to support the decision making procedure which was implicitly mentioned by all the respondents who are the representative of project sponsor. The activity goes after evaluation is deciding whether to take the project or not, to continue with the project or not, to take the next part of the previous project or not. Hence, the implication of this is in both practice and academic world, evaluation is used by project sponsor as a tool to set basic for decision making process, to ensure and protect their investment throughout the 60
whole life cycle of NSD projects. Besides, this shows the consensus in the purposes of project sponsors while doing evaluation, regardless the nature of project. Secondly, by looking at project from the project manager view, there are five purposes that the project managers take on while doing evaluation for NSD projects. The first four purposes are associated with the implementation of project while the last one link to the end project evaluation process: (1) Keep track with the common goals (2) Improve the ability to respond quickly to the market (3) Align service quality with customers’ expectation overtime (4) Enhance team member’s commitment and productivity to project (5) Gain lessons for the next generation of projects Three of the above purposes (1) keep track with the common goals, (3) align service quality with customers’ expectation overtime, (5) gain lessons for the next generation of projects were mentioned explicitly in both cases and interviews. Among the three purposes, keeping track with the common goals and gaining lessons are the same with purposes of project sponsor discussed above. This means that organizations concern very much on keeping control the project implementation while promoting organization learning as well. These purposes are consistent with the recommendations of IDRC and APM (2006) and emphasise the trend of evaluation moving further to ongoing learning rather than just measuring in today’s market (Brown and Remenyi, 2002). Knowledge transfer and learning is the basic for innovation and thus, become even more critical to NSD projects. Moreover, our respondents expressed their serious concern to the alignment between service quality and customer’s expectation during the evaluation. As discussed by Kelly (2000), the quality of service and the experience of customers on the service vary every time the service is purchased, it is necessary to verify their expectation all the time to ensure the satisfaction of customer on the service. The fact, however, was not emphasised much by researchers in project evaluation. This shows the specific difference between the aim of evaluating NSD and that of NPD. In addition, the three cases emphasised on (2) the ability to quick respond to the market and (4) the commitment and productivity of the project team. Although, the interviewees from other eleven companies did not mention the same terms, they discussed their concern of saving the projects. For us, in order to save the projects, there are two conditions that companies should take care of: internally, the productivity and commitment of the member toward the project goals; and externally, the possible issue coming from the market. Thus, it could be implied that the same concerns are given to all respondents. Regarding the ability to respond to the market, Frechtling (2002) somehow mentioned this by saying ‘evaluation provides new insights or information that was not anticipated’. However, the 61
researcher stopped at the information collected while practitioners go beyond that by using the information to react to the environment. This shows the proactive attitude of project managers in corporate world toward the uncertainty economy. At the same time, enhancing team members’ commitment and productivity to project was mentioned by most of previous researchers in project evaluation such as Ye and Tiong (2000), Mohamed and McCowan (1999), Chapman (2005), Vakola, (2000), Caulley, (1993), Vakola (2000), Segone (1998), Fabey et al. (1992). Although literature do not distinguish the purposes of conducting evaluation for NSD projects from two different considerations (project sponsor and project manager), the findings reveal that there is not much difference between theory and practices in the purposes of doing evaluation for NSD. This is to say, companies doing NSD projects, academics and project evaluators are all looking at similar objectives while conducting evaluation for NSD projects. In addition, there is no significant difference in the purposes of doing evaluation in NSD projects with those in any other type of project, apart from the aim to align service quality with customer’s expectation to provide value- added service to them. This implies that in NSD projects, customer’s involvement is more critical to the success of project than that in NPD projects.
5.2. Developing the process of evaluating NSD projects In consistence with the previous studies of McNamara (1994); Steven et al. (1993); Farbey et al. (1992), the results from both case studies and interviews proved that all companies carry out project evaluation throughout the life cycle of projects. This shows the importance of project evaluation at anytime and in any type of project in both academic and practice. There are three phases of project evaluation for NSD, pursuing different purposes as discussed in section 5.1. They are: Phase 1: Investment Appraisal Phase 2: Performance Evaluation Phase 3: End Project Review Although three phases are named differently from the phases discussed earlier in the literature review in chapter 2, the meanings are similar. Referring back to the work of Steven et al. (1993), the Investment Appraisal phase is carried out at the same period with the Planning project evaluation; Performance evaluation includes similar activities with the Formative project evaluation; and the End Project Review is taken after the project completes as it is with Summative project evaluation. However, this way of classifying project evaluation process is better in the sense that it explicitly states the key activity and purpose of doing evaluation in each phase. 62
In addition, we found that there are two main groups of people doing the project evaluation and a Reference group. These two groups are: Group 1: Project Sponsor including: The Funding Group that could be either external institution or the company’s Management Board and the representatives of project sponsor such as: the Investment Council, The Steering Committee. Group 2: Project Manager. The involvement of reference group in the evaluation process is to facilitate the implementation of NSD projects given the novelty of service to the organizations. This group could come from different departments of organization, from suppliers and also from customers to give relevant advices to the project team during the development of projects. Although, the two main groups are mentioned by both cases and interviewees, not all companies are having the Reference Group. However, we see that the presence of reference group is essential to the evaluation process in NSD projects because it was raised by Alam and Perry (2002), Shneider and Bowen (1984); Jong and Vermeulen (2003) that opinions of customers and internal staffs are key factors that influence the opportunity of innovation as well as the possibility for NSD projects’ success. Hence, the participation of the reference group in the evaluation process for NSD projects is of importance. Besides, it reveals from the above empirical data that each group is involved in different phases of project evaluation process and with different purposes. As refer to the previous section (5.1) on the purpose of project evaluation for NSD, group 1 evaluates the project as a business opportunity while group 2 evaluates the performance and progress of the project. Therefore, the project sponsor starts evaluating projects from the first phase until the last phase whereas, the project manager only starts assessing the project from the second phase and together with their sponsor look back to gain lessons in the last phase of project evaluation. While the project sponsors’ role in the first phase is crucial, they act as minor player in the next two phases. As seen from the three cases (section 4.1), the investment appraisal is carried out, mostly by the Project Sponsor group, prior to the project implementation to justify the decision to develop the new service. Once the project is approved, the project manager is assigned and leads the project team to develop new service. During that period, the project manager keeps on evaluating the progress and performance of the project team and report to the Steering Committee on the weekly or monthly basic. After the official launch of the new service, the Project Manager and the Steering Committee holds a bigger evaluation to sees what went wrong, what went right and documents for the next project improvement as well
as for organizational knowledge transfer. This process was again mentioned in similar way by the interviewees from other companies (section 4.2). The following figure described by both case studies and interview illustrates the overall process of NSD project evaluation:
• • • • • •
Prior to Project implementation Carry out by Project Sponsor Assess objectives and benefits of projects. Feasibility study and options analysis Analyze impacts of projects on various aspects Discuss the suitability of projects to strategies, context, capacity
Priorities and decide which project to follow
During the implementation of project The Steering Committee set evaluation indicators and decide methods of evaluation; sources and means of verification The Project Manager meet project team to assess the progress of project, foresee potential issues and prepare for corrective actions; use financial and non financial tools to keep update and verify the data; discuss with customers and related departments to make sure the quality of service The Steering Committee assess Project manager's performance by talking with team members, customers to get feedbacks
After the implementation of project Project Manager and Project Sponsor compare the actual data with initial indicators
Develop Customers survey to see the market impact of the service
Use financial reports for a period of time to see the sustainable impact of service to the market and how much the service add to business value
Document for organization’s learning
Reference group provides relevant recommendations
Figure 5-1 Overall Process of NSD Project Evaluation
From the figure above, we see that the NSD project evaluation process applied by corporate world is quite similar to the evaluation process recommended by Bellamy et al., (2001). Some key activities are mentioned by both the researcher and practitioners including project objectives and potential influential context assessment, indicators establishment, progress analysis, and ‘actual versus estimated parameters’ comparison. Among those activities, respondents see setting indicators one of the crucial activities of evaluation. Oral et al., (1991) also emphasised this action. They argued that because project evaluator has to refer back to the initial criteria to keep the project on the right track and to move the project toward the common goal all the time, a judicious evaluation criteria set at the beginning enhances the success possibility of the evaluation process. 64
Beside, to confirm the previous studies of Steven et al. (1993); McNamara (1994), most of the respondents from the interviews and cases mentioned collecting information through communication as a big part of evaluation process. The feedback and feed forward channel is therefore of special concern by both academic and practice during the evaluation process. In general, the model collected from practices show more detail on each activities of evaluation process which turn out to give a clearer idea of what should be done, who should do and when should do the evaluation. In addition, the new model goes through the whole life cycle of the projects providing a sufficient understanding of the evaluation process. However, in order to have more detail perception on each evaluation phase, we would like to recommend the model illustrated in section 4.1.2. As suggested by Gardiner (2005), Lopes and Flavell (1998), Abdel-Kader and Dugdale (1998), Stokdale et al. (2006), the following analyses are undertaken throughout the project evaluation process for NSD projects: Strategies analysis Risk Assessment Stakeholder analysis Capacity analysis Market Impact analysis Feasibility study and options analysis Financial analysis Assumption analysis The findings from our empirical study have positive indication to this. In addition, respondents from both cases and interviews expressed their emphasis on strategic analysis and customer analysis as the most common and critical parts of their evaluation process (see chapter 4). Although previous studies on project evaluation mentioned stakeholder analysis as one of the key activities (Steven et al., 1993; Örtengren, 2004), practitioners of NSD evaluation pay more attention to one group of stakeholder: customer. Despite of the critical role of customers in the success of any business, the issue is more sensitive to service industry as it is customers who set the standard for service rather than any physical indicators. This fact, once again, confirms the discussions of NSD researchers like Kelly (2000), De Brentani (1991) and Lovelock (1983) about the variance of customers’ experience and demand to services, leading to a need to analyse this group all the time. Also, customer analysis is considered as part of market impact analysis by most of our respondents. In addition, we found that, capacity, assumption, feasibility analysis are not mentioned much by respondents. This fact, however, does not mean that those aspects are discounted by them but they are considered as part of strategic analysis. One respondent stated that: ‘To us, capacity, feasibility, or assumption analysis are included in strategic analysis and should be assess prior to the decision of developing the service...We would never develop an idea if it is out of our scope or capacity’. Furthermore, according to respondents, an early analysis and evaluation influence positively to the success of the launch as the time to market is more critical and shorter in service industry in comparison to that in manufacturing industry.
Regarding the tools and techniques, the responses from cases and interviews show that the evaluation tools are used variously in each evaluation phase as well as by different among evaluators’ groups. Small (1998), Ballantine and Stray ( 1998) and Müller (2003) suggested some financial measurements from which project evaluators could look at to assess the projects such as Net present value (NPV), return on investment (ROI), Discounted cash flow (DCF), Internal rate of return (IRR) or Pay back period. Most of respondents confirmed that those basic financial indicators are used all the time when they do evaluation for NSD projects. On the other hand, the use of non-financial techniques is also emphasized as necessary to evaluate the market impact of the new service. According to Doloi and Jaafari (2002), simulation is a helpful method to test how the project works and what are the possible problems of the project. Also, Fox and Baker (1985) particularly recommend simulation or experiment to the evaluation of innovation projects. The respondents from our empirical study are not only agree with this suggestion but also emphasize that simulation, survey or direct communication are even more important tools to customize their service. This reveals the consensus among practices and theory on the evaluation methods while indicating the specific needs of more non-financial method for evaluating NSD projects given the intangibility trait of service products. The implication of this finding is that although the tools and techniques applied are similar between academic and practices, the process of project evaluation for NSD is rather complex due to the specific characteristic of NSD project. Particularly, it requires more customer involvement and more discussion among different groups. Also, the project evaluation process for NSD is simultaneously taken with the development of new service which complicated the process. Hence, project evaluators for NSD projects should always keep in mind the critical role of customers when performing evaluation. More than that, in order to be success, practitioners of NSD project evaluation ought to improve communication skill as it is needed to get appropriate information from different stakeholders.
5.3. Identifying criteria that should be carefully evaluated to ensure the success of NSD projects 5.3.1. Challenges in evaluating NSD projects The data collected from case studies and interviews reveal several problems in evaluating new service development projects. Some of them are related to the special characteristic of new service while some are associated to the project evaluation in general. Most of the challenges are in consistence with what have been found by previous studies on both project evaluation and new service development. The main difference between literature and practice is related to the tactical concern of academic versus the more dynamic attitude of practitioners. 66
An airline company operating in Vietnam mentioned (as previously illustrated in section 4.2) communication channel as a challenge in their evaluation process whereas the Albanian telecommunication company AMC added that they are having difficulties to gather reliable information. This confirmed the argument of McNamara (1994) that sources of information are very important and lack of good communication channel may lead to a wrong evaluation results which dramatically affect the decision making procedure. The problem calls for more attention and effort on how to enhance the feedback and feed forward system within companies as well as with external stakeholders in practice. Moreover, there should be a clear verification process to validate the information collected through the evaluation period which improves the quality of the evaluation process. The willingness of people to participate and provide value comments allows the evaluation results to be more reliable which turn out to ensure the success of NSD projects. Although companies are trying hard to improve communication mechanism by flatter the company’s structure, some multinational or large companies are still considering company’s hierarchy and procedure as a factor that complicates the evaluation process of NSD projects. By analyzing our data collected through interviews with several companies in different countries, it came out that in some companies, (such as the banking company in Vietnam), where the decision making is centralized in one regional office, it is not easy to come up with common evaluation indicators. Some evaluation criteria are not agreed among the regional office and local branch resulting in slower the evaluation process and influence the quality of the evaluation results. This problem however was not notified by previous studies. Another issue leading to the difficulty of setting evaluation indicator which is the nature of the projects was raise by our respondents. Gadrey et al. (1995) comment about the differences in the nature of different types of new service development projects. According to him, the choice of evaluation criteria varies among sectors due to the particular feature of each. The consensus between research and practice on this issue implies that although, setting indicators is a key activity in the evaluation process as recommended by Bellamy et al. (2001), it is more complex and need to be applied in a more flexible way in practical world. It appears from both cases and interviews that soft results lead towards evaluation difficulties or problems. The respondent from the airline company operating in Vietnam brought into attention of our data analyses the difficulties associated with measuring intangible outcomes generated from the new service. He stated (as mentioned in section 4.2) that they are facing difficulties to evaluate prestige or social effects of the new service to the company. This fact explained why there was a lot of effort in the academic work (Müller, 2003; Ballantine and Stray, 1998; Turner, 1995; Akalu, 2001) regarding the measurement tools in evaluation process such as NPV, IRR, payback period, discounted cash flow, etc. However, the non- financial aspect of projects could not be evaluated 67
simply by those numbers. The long term impacts of the new service to the organization’s development, the window opportunity which the new service opened up to the firms are of more concerned by companies, particularly from a strategic point of view. The measurement issue becomes more serious in new service development projects due to the Intangibility characteristic of service. Therefore the evaluation of NSD projects is more complex than the evaluation of NPD projects. The opinion of the respondents from the Swedish airport company and a Vietnamese company (refer to section 4.2) about the difficulties of project evaluation was that in the case of NSD evaluation there are not many specific or standard criteria to assess because of their intangible value. Even though, all the interviews came into the same position that it is very important to involve customers into project evaluation, the challenge associated to the novelty feature of NSD projects is how to get customers involved especially to make them try the service during the development of service. This was also raised by Thomas (1978) that customers find it risky to try the new service as they could not fully examine the quality of service prior to purchase. Therefore companies face difficulties to measure the usability of service which is a crucial criterion of project impact evaluation. Another problem related to customer involvement could be inferred that the comments are changed from time to time due to the human trait of customers, the evaluation process may take longer time than expected to get correct responses. The implication of this is that customers play a crucial role in securing the quality of new service development project evaluation’s result. The above characteristics of new service lead to difficulty in Risk assessment. The respondents from EON in Sweden mentioned that it is difficult to measure risk which is very high in service projects. In order to have accurate risk assessment and analyses they refer to a risk analyst company. The following table summarizes the problems faced from the companies. It illustrates the consequences in evaluation process and the impact on evaluation criteria.
Table 5-1 Challenges in evaluating NSD projects No. of respondents 10 3
Source of problem Communication channel
Company structure and procedures Nature of the project
Wrong results of evaluation Low stakeholder involvement in evaluation process Difficulties on determining indicators Complicated evaluation procedure -
Difficulties on measurement of prestige, competitiveness, social effects. - Subjective evaluation - Subjective evaluation results - Time consuming Difficulties on measurement of project profitability
5.3.2. Proposition list ranked by respondents All the respondents were kindly asked to rank a list of main criteria considered from literature as crucial. The criteria are ranked by assigning 1 to the less important and 8 to the most important criteria. The list was ranked by sixteen respondents instead of seventeen as two respondents coming from a company said that they are applying a common set of criteria for their evaluation process. Some respondents have given the same grade to different criteria as they consider these criteria to be equally important. Although the respondents come from different service area, they assigned grade to same list of each criteria based on their experience on project evaluation. Therefore it could be said that the results for the prioritization of each criteria should be accepted with a confidence level. A bar chart with two columns is chosen to present the data (refer to Figure 5.2).
Figure 5-2 Key Evaluation Criteria in % (ranked by respondnets)
In order to give a common evaluation rate to each criterion ranked by the respondents, the graph is established to describe two main points: The level of importance given to each criteria (percentage) The percentage of respondents (over 16 respondents) ranked a particular criterion as very important with grade 7 and 8. It is obvious from the graph that ‘marketing criteria’ is considered as the most important evaluation criteria of NSD projects with the level of 16.2% ranked by all respondents. Among them, 68.8% agreed that this is the most important criteria in the evaluation process. The second important criteria is ‘strategic fit’ with just 1.2% and 2.5% less than the first one regarding the level of importance and the number of respondents judged it as important or extremely important, respectively. It is followed by ‘profitability’, ‘Information quality’, ‘facilitating criteria’, and ‘return on investment’. The least important criteria that the companies take into consideration when evaluating NSD projects, came out to be ‘ Production cost’ and ‘environmental and CRS criteria’ which was ranked with only 10.6% and 6.7% level of importance and only 18.8% and 37.5% respondents considered these two criteria as importance, respectively. 5.3.3. Discussion on each evaluation criteria This section discusses each evaluation criteria based on the empirical data analysed in chapter 4 and the information presented on the figure 5.2 and the table 5.2 below. Some personal comments are presented to facilitate the discussion. Table 5-2 Key Evaluation Criteria (by no. of respondents) Criteria Grade equal to Profitability (revenue, number of customer, growth) Production cost (promotion cost, salary) Return on Investment ( NPV, IRR, cost of capital, payback period) Strategic fit (objectives, strategy, policies, corporate values, company’s capacity) Marketing criteria (customer satisfaction, time to market) Environmental and CSR criteria Information quality (accuracy, adequacy, appropriateness of feedbacks and feed forward) Facilitating factors (stakeholder attitudes and participation)
Regarding the table, each cell typically shows number of respondents who assign the grade labelled in the column to the criteria given in the corresponding row. For example, the value corresponding to row ‘Production cost’ and column ‘7’ equals to 3 means that 3 out of 16 respondents ranked this criteria with grade 7. The same explanation is applied to all other criteria. 70
Financial Criteria Criteria such as Profitability, Production cost and Return on investment are grouped together based on their financial nature. These criteria were investigated and recommended by many authors such as Johne and Storey (1998), Ballantine and Stray (1998), Small (1998) or Müller, (2003). As mentioned by them, the financial aspect of a project is very essential to evaluate projects since it provides numerical idea to project sponsor or evaluators of how much the project will cost, how much the company will win and for how long the revenue could cover the amount of money invested or how much shareholders gain from the project. These are visible information that project evaluators could at least measure and compare over time. However, these numbers have to be based on realistic data and use suitable projection standard to avoid over or under estimation. Along the project implementation, the data have to be kept updated to see if the project is running on budget or how much changes influence shareholders’ value. The findings from case studies and interviews provide the same financial criteria with the proposition list suggested in chapter 2 and are discussed in details as below. Profitability Revenue and volume of customers purchasing the new service or the incremental growth from year to year are grouped as profitability criteria in the evaluation process. It is obvious that the first indicator the management board or project sponsor looks at is the possible tangible or intangible benefit that the project could bring to the firm and its shareholders. Normally, due to the business trait of corporations, no firm would choose the project without seeing profits. According to the analyses of case studies and semi structure interviews, profitability is evaluated during the whole life cycle of the project. Companies select projects based on the potential revenue the project will generate. Respondents comment that profitability is not only important to select profitable projects since the early stage of decision making but also during the on-going phase of the project and help assessing the success of the project in monetary terms at the end of project. For example, the Swedish company offering air navigation services looks at the number of passenger using the new service to see if the service is profitable and success as expected. Another company, EON, is also taking care of the incremental growth of the revenue over time to ensure the sustainable growth opportunities for the company. Half of respondents refer to this criterion as extremely important, five of them ranked this as fairly important while only three pay less attention on this criterion shows the importance of the criteria in the evaluation process (table 5-2).
Production cost The production cost in service companies is normally related to staffs’ salary, promotion cost, etc. It therefore emphasizes the concern of organizations on internal capacity and used to checked whether the company able to pay for the cost generated by the development of new service. This criterion is mentioned by some of the respondents as important criteria which should be compared with profitability criteria in order to evaluate the chance of a certain project in being selected or not form the company. According to the respondent from EON, this operation criterion is used to evaluate and monitor their projects. They use financial techniques to calculate the value of these operational criteria for a number of years ahead. By doing so the company is able to decide if the project is profitable or not and if the project is selected, then these records about operational criteria are used to monitor and evaluate how the project is going during its implementation phase. Based on the judgement of respondents, this criterion is however not much as important as the other two financial criteria as judged by respondents. The explanation for this is due to the difference between service and product development projects. In manufacturing sector, production cost is rather large whereas that in service companies is not significantly a huge investment. Return on Investment Return on investment takes into consideration the amount of money the company or project sponsor put in the project. It looks at the net value that the project shareholders receive after a period of time. It shows the concern of shareholders to the projects while the profitability provides company’s overall benefit. The main difference between this criteria and profitability is that it refers to financial aspect of project while the profitability criteria refer to accounting numbers. The data collected from our empirical study verify the importance given to this criterion by academics and researchers as one of the key evaluation criteria to ensure the net value that stakeholder are going to get. Almost all the companies interviewed use the same traditional financial methods such as pay-off period, present value after initial investment, Nominal Internal Rate of Return post tax, Weighted Average Cost of Capital (WACC) or the Return On Capital Employed (ROCE) to see if the project is meeting the required level. Before deciding which project to undertake the company is analyzing the rate of return the project is making on the total capital employed in that project. Non Financial Criteria According to Lopes and Flavell (1998), there are other aspects rather than just financial numbers that companies should take into consideration during the evaluation of project such as organizational, managerial, political, social and environmental dimensions. Lack of those criteria could lead to failure of the projects. Results from our empirical study found
some non financial criteria which are almost similar to the non financial criteria summarized in the proposition list in chapter 2. Strategic fit As mentioned by Alder (2000), non financial criteria related to strategic issues such as ‘strategies fit, relationship with stakeholders, competitive advantage, value chain, and cost drive’ are very important criteria used to support the decision of selecting or keeping a project from the planning phase until the end of project. In consistence with this, respondents from all the companies involved in this study mentioned important issues related to strategic fit such as strategy alignment, internal capacity and scope. Although each of them mentioned those criteria in different ways and with various concerns, the results from case studies and interviews support previous study and show that strategic fit is considered among the most important criteria of evaluation. Strategies Alignment: Respondents discussed the need to fit the objectives of projects to the overall corporate goals as well as to the current conditions of the company prior and during the project implementation. For instance, even though the first selection of the projects is done based on requirements of stakeholders, the respondent from the academic institute in Italy mentioned that in his institution the decision to take part in a project is mainly based on strategic fit of the project with the overall strategy and objectives of the institution. By stating this, he sustains strongly the importance of strategic fit from the perspective of strategic alignment to the project objectives which was supported also by some authors such as Twiss (1986); Chiesa and Masella (1996). The model of project evaluation for NSD described in the analysis of the three cases (figure 4.3) is an advocate to this position as well. Improvement of Relationship with stakeholders: Building and maintaining good relationships with stakeholders is considered as crucial criteria of success by literature (Greene, 1988; Crawford and Bryce, 2003). In addition, what is considered as important from case studies and semi-structured interviews analyses in not only to build good relationship with stakeholders but also to further improve these relationship. Most of the companies/institutions interviewed have formal and informal consultation with their stakeholders and take into consideration while evaluating, the fact that how much the project will improve relationships with existing stakeholders. In some of the cases, the target of the company is not to gain short term profit but to satisfy a party’s needs which turn out to bring long term effect to the company. Moreover, image and brand values, as important characteristics of an organization discussed by Farbey et al. (1992), are often mentioned by our respondents from case studies and semi-structured interviews. They commented that these soft results are taken into consideration when deciding to undertake a project or after the completion of the project to see the impact of the new service to the reputation of the companies. This 73
criterion presents the competitiveness and position of the company in the market. Management or project sponsor are always consider this criterion as it contribute to the long term development of organizations. Hence, it is more than important to take it into account while evaluating projects. Scope: In addition, the respondent mentioned one of the three main criteria that have been strongly recommended by the triangle PMI model which is scope. As suggested by Andersen et al. (2002), because scope may vary from project to project depending on the type of service project, it is important to have a clear scope in order to go on the right track with what have been expected. The scope of project also shows the feasibility of the service putting in the context of the organisation as well as of the current market (Stokdale et al., 2006). Clear and realistic goals, rights and responsibilities can only be achieved with a clear scope. The internal capacity of the organisations is another strategic issue that should be taken in during the evaluation process as refer to Örtengren (2004). In service industry, people are the key resource. The results from the empirical study show that companies concern about the capacity available, the suitability of current quality and skills of staffs while making decision on selecting projects. This criterion allows project evaluator to make correct decision in choosing or replacing the project. The concern is not limited in the quality of staffs but extend to the professional ability of project managers. Some companies (see case studies and interviews analysis section 4) have criteria to evaluate the performance of project managers during the project implementation to ensure that the project’s result would contribute to the corporate value. Credibility of customer: Although this criterion was not included in the proposition list in chapter 2, it is implicitly associated to the strategic fit criteria because it is related to the level of risk that the company could accept while undertaking the project. As founded from the two case studies (EON and Citigroup) and semi-structured interviews with several other companies, customers’ credibility is considered an important criterion of project evaluation. In big investment projects, the quality and the reputation of clients secure the companies from liabilities and risks. Thus, company should assess the customers’ ability to pay for the new service offered before deciding to undertake the project. However, this criterion is applied only to companies offering specific service to specific customers. With companies serving a large number of customers and have no opportunity to choose customers, this indicator is not applicable. The number of respondents ranked strategic fit as extremely important (7 and 8) is eleven out of sixteen while nobody thinks it is not important ( refer to table 5.2). This is to say, academic, practical companies and project professionals are using the same criteria in evaluating NSD projects. In spite of the specific feature of NSD, strategic fit is always a must in the project evaluation process. 74
Marketing Criteria Customer satisfaction is considered very important by all respondents from both case studies and semi-structured interviews. This criterion should be assessed during the whole life cycle of the project to ensure the new service adds value to customers and sustain the profitability to the organizations (Shneider and Bowen; 1984). Since customer is the user of the service, their satisfaction is considered one of the best measurements of the quality of service offered through the project. Whereas, quality, which is more sensitive to NSD than to NPD (Tukel and Walter; 2001), defines how successful a project is in terms of market impact. Thus, base on the level of customer satisfaction to the new service, companies could check how much the project fulfilled its objectives and how much impact the service put on the targeted market. Furthermore, the finding shows that unlike in other types of projects, customer satisfaction is used as key evaluation criteria in not only the project review phase but also during the implementation of NSD projects. Project mangers evaluate the level of customer’s satisfaction by giving presentations or simulation all the time in order to ensure the final version of service offering would meet customer’s expectation. The quality of service is therefore could be secured and a big part of the project success is achieved. To support this, a respondent argued that ‘The more satisfy the customers give to the new service during the development of service, the less time needed to redesign and test the new service...It reduces time to market of the service and the duration of the project’. This links to another aspect of marketing criteria- Timing. Although, this criterion was not much mentioned explicitly by respondents in both cases and interviews, it was implicitly referred to as one of the evaluation criteria in NSD projects. The time could be time to finish project which turn out to be the time to market. The criterion of time is also related to the fact how urgent a need is and the time available to the company to fulfil the specific need of the customer. As shown in table 5.2 the marketing criteria was ranked as most important evaluation indicator in NSD projects with eleven people consider it as extremely important (7 and 8) and the remaining five people refers to it as fairly important. Nobody is under- rating this criterion. This criterion is even rated higher than strategic fits with 1.2% more in the level of importance ranked by all respondents and with 12.5% more in the number of people rated it as the most important criterion. However, the emphasis was given to customer satisfaction aspect. This is consistence with literature such as Den Hertog (2000), Grönroos, (1990) in service that customers’ assessment is essential in the development of new service. The results present the critical position of this criterion, specifically to new service development projects. Environmental concerns and Corporate social responsibilities (CSR) This criterion refers to the attention of corporate entities to the society such as environment, government regulations and social impacts. In support to the recommendation of Farbey et al. (1992), Case studies and semi-structured interviews point 75
out that project evaluation should take into consideration how much the project match the culture of the company in terms of environmental and CSR issues. Some respondents pay special attention to environment protection responsibility such as EON, LFV, the Center of Agriculture Technology Transforms. This is due to the nature of the business area they are working in like airlines or energy. However, some large companies refer to CSR as one of their key evaluation criteria because this responsibility closely influences the image of the company to the business world. Before undertaking a project each company has to check if they fulfil all regulations issued by the government of the country where the project will be carried or how much the project could support the employment issue of the country. According to EON case study the company is concerned about creating jobs through the projects they select to undertake. Most of respondents agree that company should pay more attention to CSR and consider it as an evaluation criteria as advocate to the trend toward CSR in academic world. However, only one respondent ranked this criterion as most important criteria while eight interviewees consider this as least important. The only respondent ranked CSR as their primary evaluation criteria because the company is working in the agricultural field where environmental issue is of strategic concern. Also, the company is state-owned company so the government regulations and employment issues are one of the biggest objectives of stakeholders. The result shows that companies are still more concerned on the firms’ benefits than to the societies’. Facilitating factors The involvement of stakeholders and their attitude toward the new service is considered as facilitating factors (Andersen et al., 2002). This criterion was also mentioned by both three case studies and interviews. It could be the attitude of external stakeholders or of internal staffs to the project. For example, the management support in the case of Citigroup, the customer involvement during the service design period in LFV, the sponsor‘s positive attitude to the project in the Centre of Agriculture Technology Transforms or the secondary school ‘Gjon Buzuku’. The facilitating factor could also influence the project implementation due to the changes in government’s regulations or the negative movement of customers’ requirements. If the level of support given to the new service is low, then the project might not be developed or it might not be able to reach the targeted objectives. In addition, from the internal point of view, the commitment of the team members toward the project is another facilitating factor as it affects directly to the success of projects. The support from other departments’ staff also provides a better condition for the new service to be developed. However, different from Andersen et al., (2002), NSD practices pay more attention to the involvement of customers to the projects. This could be implied that customers play a crucial role in the success of new service development projects and their participation in the development of new service contribute to the quality and results of projects. The rating from respondents ( table 5.2 and figure 5.2) toward this criterion is 13% level of importance and 37.5% (six respondents ) people think that this criterion is importance or 76
extremely important. The level of importance judged by all respondents from both cases and interviews show that this criteria is fairly importance in the evaluation process with just 3.2% less than the most important criteria – Marketing criteria. Hence, this is one of the indicators that companies should look at when evaluating projects. Information quality The information quality includes the appropriateness, the accuracy and promptness of feedbacks given by stakeholders regarding the new service development projects. This related closely to the facilitating factors as if the stakeholders have positive attitude toward the projects, they will give more correct and value response to the project which allows the evaluation results to be more correct. Data collected from both case study and semistructured interviews support the findings of Tukel and Walter (2001) and Steven et al. (1993) that companies should pay attention to the communication channel and the quality of information gathered as it will affect the quality of evaluation process. We noticed that the respondents from EON and Asian airline as companies with many levels of organization are very much concerned with accuracy and adequacy of the information provided from the feedback and feed forward in the communication channel. While the respondent from AMC and IT company in Italy are mainly focused on the appropriateness and quality of information provided from customers’ feedback. The quality of feedback from customers is again, especially important to the project as stressed by most of the project managers interviewed. This is to say, in service industry, customers play a major role and it is always recommended to evaluate this group of stakeholders along the project life cycle. The rating for this criterion is 13.4% with 25% of people (four respondents) agreed that this is extremely important to be considered in the evaluation of NSD projects (table 5.2 and figure 5.2).
6 CONCLUSION The analysis and discussion in chapter 4 and 5, together with the literature review in chapter 2 contributed to achieve the three purposes of our thesis which led us to answer the research question ‘What are key evaluation criteria for New Service Development projects?’ The consensus on various reasons of conducting evaluation among respondents from three case studies and seventeen semi-structured interviews as well as from literature raises the critical role of evaluation process to the success of NSD projects. Besides, the findings offer a sufficient understanding on the process and the key activities of developing new service, the overall process of evaluation for NSD projects during the whole life cycle of projects and from different perspectives. The distinction between the evaluation process for NSD and that for NPD is mainly on the intangibility and the human involvement characteristics of service which complicates the evaluation process. This provides challenges for practitioners in measuring soft results generated by the development of the new service. It is also hard to get customers involved and obtain their valid comments as their answers are very emotional and inconsistent over the time. Therefore the evaluation of NSD projects is usually considered as more subjective. Furthermore, we noticed from the interviews conducted with people of different relations and interests in the project that project manager and project sponsor see differently the evaluation process and take into consideration different criteria of evaluation for NSD projects along the project life cycle perspective. This finding provides a practical insight into the theoretical framework recommended by Bellamy et al (2001) and add richer detail to each phase of the evaluation process offering an adequate view of the whole process. The project evaluation in the planning phase is based on input indicators which represent resources that should be employed to run the project. The most used criteria for this phase is Strategic fit as well as other financial criteria such as revenue, cost and payback period. Since the main asset of service companies is people, the resource availability criterion is among the critical ones. Also, due to the novelty of new service to the company itself, the feasibility criterion is of importance during the planning phase of evaluation process. In some cases, the environment and CSR is taken into consideration depending on the current target of the company and the nature of project. If the project evaluation is doing well in this phase, the project holds a higher possibility of success since the planning phase of project evaluation is carried out prior to project. Criteria of evaluation on the formative phase are concerned with output (service offered) indicators. These criteria are related to project activities. As it also mentioned above by one 78
of the respondents, usability of the service is crucial. The marketing criteria, quality of information, and facilitating factors are what project evaluators looking at in this phase in order to ensure the success of the new service. While it is obvious the marketing criteria targets at the goal of the project, the quality of information and facilitating factors show that the communication channel contributes to significantly to the results of the evaluation of NSD projects. In summative phase of NSD project evaluation, criteria of evaluation are considered as impact indicators which express the actual differences on the level of service offered after project implementation and the degree expected. Project managers that we interviewed take into consideration both financial and non financial criteria, but the main criteria is profitability and customer satisfaction. At the same time, project sponsors are interested in evaluating the development impact in a specific service industry, as well as in satisfying the customer after the project completion. To conclude, from our research through case studies and semi structured interviews with several companies in Albania, Italy, Sweden and Vietnam a list of criteria is finalized as illustrated in the Table 6.1 below: Table 6-1Findings on Evaluation criteria for NSD projects
Financial Criteria 1. Profitability 2. Production cost 3. Return on Investment
6. 7. 8.
Non Financial Criteria Strategic fit Strategies alignment Credibility of customer Time, cost, scope Corporate Image Stakeholders enhancement Marketing criteria Timing Customer satisfaction Corporate social responsibilities (CSR) Information quality Facilitating factors
By comparing data found from the empirical study with the contribution of researchers discussed in the literature review (chapter 2) we found that the above list of criteria (Table 6.1) fits completely with the proposition list (table 2.7) in chapter 2. Therefore we can suggest this list of criteria as most important to take into consideration during evaluation process for new service development projects. However, different criteria have different importance on the evaluation process, depending on the nature and type of service projects. For that reason, this study comes up with a conclusion that some of the criteria are more 79
important than the others but there exist criteria that must be always taken into consideration when evaluating NSD projects. In terms of the research question which is ‘ What are key evaluation criteria for New Service Development projects’, the findings from this study reveal three groups of evaluation criteria as follow:
There are six criteria which were mentioned as more important by respondents: 1) Strategic fit 2) Marketing criteria ( Customer satisfaction) 3) Information quality 4) Profitability 5) Return on Investment 6) Facilitating factors
The results of the empirical study confirm the list of criteria proposed from researchers (in chapter 2) as crucial to project evaluation. For this reason, the study strongly recommends that the above list of criteria should be taken into consideration in NSD project evaluation.
The remaining two criteria are: 1) Environmental concerns and CSR 2) Production costs
Although these two criteria are not ranked as important as the other six indicators, the percentage of the production costs is slightly less than that of other more important criteria. The implication for this is that in service industry, the cost for producing new service is not as high as that in manufacturing sector. Not much investment in machinery or plant but mainly in human resource which do not raise heavy investment is taken into consideration when developing new service. The respondents who consider environmental and CSR as important is however far less than those who consider it as important. This could be explained by the fact that not many companies pay attention on the responsibilities to society while doing business, especially companies in developing countries. However, the trend is moving upward and more and more companies are considering CSR and environment as one of their corporate goals.
Criteria that must always be considered on project evaluation 1) Marketing criteria (Customer satisfaction) 2) Strategic fit
These two criteria are ranked with the highest points from all the respondents from both case studies and semi-structured interviews. Hence, we conclude that these criteria are the most important and indispensable to the NSD project evaluation. Although criteria based 80
on the service nature vary from project to project, the marketing criteria and strategic fit must always be included to the set of project evaluation criteria for NSD projects. In the marketing criteria, the customer satisfaction is especially important. The justification is that service companies are becoming more and more customer focused and the reason why they develop new services is mainly to fulfil and satisfy customer needs. This finding adds up to the current project evaluation research. Besides, in order to succeed with NSD projects it is necessary to ensure a strategic fit of the project. In addition, this study calls the attention regarding challenges that interviewed companies are facing during NSD project evaluation. The biggest difficulty they have to deal with is customer involvement. As revealed from theory, customer involvement is a very important criterion for NSD projects unlike NPD projects. Therefore it influences strongly the results of project evaluation. Limitation of study The main limitation of this study is related to the method applied. As mentioned in the methodology chapter, one of the drawbacks of the qualitative strategy which is chosen by this study is too subjective, impressionistic making it difficult to generalize findings to other settings (Bryman and Bell, 2003). Besides, since the sample of the study is small with only three case studies and seventeen interviews makes it difficult to us to generalize some of the evaluation criteria stated by some respondents. A quantitative method with larger sample size would definitely improve the results of the research. During the study, we also faced some obstacles when collecting empirical data. Some contacts showed interest to participate in the interviews at the beginning, kept promising that they would be available for the interviews but finally, they did not show up or cancelled the appointment. The reason for this is the industry that we are conducting the study is service sector; most of them are from financial area. The last three months are the very peak time of the year resulting the last minute of refusals by the potential interviewees. Besides, the time and budget constraints reduce the ability to get more data from the corporate world.
Suggestion for further study Since the study is among the initial efforts to connect project management with service development field, further studies could be approached from either project evaluation perspective or on NSD projects. Regarding project evaluation, it is noticed by both practice and theory that even though, the end project review is crucial for the organizational learning, there are not many studies on that phase of project evaluation. Thus, a deeper investigation on that part of project evaluation would be possible. From the aspect of NSD projects, this study reveals that there is a concern of project professionals and literature on the management of stakeholders. Since the involvement of customers is of crucial importance during the development of NSD project, it could be interesting to conduct a study on stakeholder management for NSD project. In addition, the challenges rose through the empirical study of this thesis in respect to how to measure soft results such as image, social impact that could induce a new area for further research. Particularly of interest could be a further research regarding solutions on how to get customers more involved in project evaluation. Moreover, since there are needs for more effort on finding a new method to measure non- financial indicators, each of the above criteria found by this study could trigger a research topic. Otherwise, the same research topic would be further investigated applying restriction to geographical location or to business area.
APPENDIX Appendix 1: Interview question guide Definition of New service development (NSD) projects: projects to develop service products that are new to the supplier. The project has the new idea generation as starting point and ends on the date of launching new service. The evaluation process could be applied prior to the projects, during or after projects depending on your company’s practice and purposes. General Information: -
Could you please briefly describe your company?
What are your experience and/or roles with project evaluation, especially with NSD project?
How does your company form project/evaluation teams? What is the relationship between them?
How much are stakeholders (end users, vendors, sponsor, etc.) involved/influence the evaluation process?
New Service Development Projects -
Could you please tell us about the process of developing new service in your company?
What are key activities involved in NSD process?
Project Evaluation in NSD projects -
What are the purposes of the evaluation process in your company?
Could you please tell us about the process of evaluation for NSD projects applied by your company?
What are the techniques that you use for evaluating NSD projects?
How do you decide to invest in the new service?
How do you ensure the NSD project is on the right track?
Base on what can you say this NSD project is success?
Are there any specific criteria related to NSD evaluation which differs from NPD (New product development) evaluation, given the differences between service and non-service industry? What are they?
Final Questions What are the difficulties and challenges in the evaluation process?
Do you recommend a set of evaluation criteria that you use for your work?
Could you please based on your opinion rank the list of evaluation criteria for NSD projects that we found from the literature? (From 1: least important ; to 8: most important)
Profitability criteria (revenue, number of customer, growth) Production criteria salary)
Financial parameter ( net present value, return on investment, cost of capital, payback period)
Strategic fit (objectives, strategy, policies, corporate values, company’s capacity) Marketing criteria (customer satisfaction, time to market) Environmental and CSR criteria Information quality criteria (accuracy, adequacy, appropriateness of feedbacks and feed forward) Facilitating criteria (stakeholder attitudes and participation) -
Does your company have any official framework for evaluating NSD projects? Is it possible for us to access?
Would you prefer to keep the above information anonymous or can we make the name of the company public? Thank you very much for your kind attendance! 84
Appendix 2: Proposition list ranked by respondents Criteria /Respondent Profitability (revenue, number of customer, growth) Production cost (promotion cost, salary) Return on Investment ( net present value, return on investment, cost of capital, payback period) Strategic fit (objectives, strategy, policies, corporate values, company’s capacity) Marketing criteria (customer satisfaction, time to market) Environmental and CSR criteria Information quality (accuracy, adequacy, appropriateness of feedbacks and feed forward) Facilitating factors (stakeholder attitudes and participation)
2 3 4 5 6 7
8 9 10 11 12 13 14 15 16
2 4 5 6 6 1
7 5 7
5 3 7 5 4 1
2 7 7
3 2 7 4 7 1
5 7 8
8 6 4 8 7 5
8 4 8
7 5 8 7 5 8
6 8 7
1 1 3 2 8 7
4 1 1
4 8 6 1 6 4
8 6 5
6 7 8 3 3 6
3 8 6
REFERENCES Abdel-Kader M.G., & Dugdale D. Investment in advanced manufacturing technology: A study of practice in large U.K. Companies (1998) Management Accounting Research, 9 (3), 261-284. Adler, R. W. (2000). Strategic Investment Decision Appraisal Techniques: The Old and the New. Business Horizons, 43 (6), 15-22. Akalu, M. M. (2001). Re- examining project appraisal and control: developing a focus on wealth creation. International journal of project management, 19 (7), 375-383. Akalu, M. M. (2003). The process of investment appraisal: the experience of 10 large British and Dutch companies. International Journal of Project Management, 21 (5), 355-362. Alam, I. & Perry C. (2002). A customer-oriented new service development process. Journal of Service Marketing, 16 (6), 515-534. Andersen, E.S., Dyrhaug, Q.X., & Jessen, S.A., (2002). Evaluation of Chinese projects and comparison with Norwegian projects. International Journal of Project Management, 20 (8), 601–609. APM. (2006). APM Body of Knowledge. Buckinghamshire: Ingmar Folkmans. Atuahen-Gima, K. (1996a). Differential potency of factors affecting innovation performance in manufacturing and services firms in Australia. Journal of Product innovation management, 13 (1), 35-52. Atuahene-Gima, K. (1996b). Market orientation and innovation. Journal of Business, 35(2), 93-103. Avlonitis, G. J., Papastathopoulou, P. G., & Gounaris, S. P. (2001). An empirically-based typology of product innovativeness for new financial services: Success and failure scenarios. The Journal of Product Innovation Management, 18 (5), 324-342. Ballantine, J., & Stray, S. (1998). Financial appraisal and the IS/IT investment decision making process. Journal of Information Technology, 13 (1), 3- 14. Banwell, L., Ray, K., Coulson, G., & Proud, D. (2003). Evaluation, Impact and Outcomes the Jubilee project. Performance Measurement and Metrics, 4 (2), 79-86.
Barwise, P., Marsh, P. R., & Wensley, R. (1989). Mush Finance and Strategy Clash? Harvard Business Review, 67 (5), 85-90. Bellamy, J. A., Walker, D. H., McDonald, G. T., & Syme, G. J. (2001). A systems approach to the evaluation of natural resource management initiatives. Journal of Environmental Management, 63 (4), 407–423. Berry, L. (1980). Services Marketing Is Different. The Journal of Business, 30 (3), 24-29. Bowers, M. (1989). Developing new services: improving the process makes it better. Journal of services Marketing, 3 (1), 15-20. Brown, A., & Remenyi, D., (2002) ‘Ninth European Conference on Information Technology Evaluation’. Reading, UK: MCIL Bryman, A., & Bell, E., (2003). Business Research Methods. New York: Oxford University Press Inc. Caulley D., (1993). Evaluation: Does it make a difference?.Evaluation Journal of Australia 5 (2), 3-15. Chapman, C., Ward, S., & Harwood, I. (2006). Minimising the effects of dysfunctional corporate culture in estimation and evaluation processes: A constructively simple approach. International Journal of Project Management, 24 (2), 106-115. Chapman, R.L., & Soosay, C., (2003). Innovation in logistic services and the new business model A conceptual framework. International Journal of Physical Distribution & Logistics Management, 33 (7), 630-650. Chiesa V. & Masella C. (1996). Searching for an effective measure of R&D performance. Management Decision, 34 (7), 49–57. Citigroup website: http://www.citigroup.com Clemons E. K. (1991). Making the investment decision: evaluating strategic opportunities in information technology. Communications of the ACM, 34 (1), 22-36. Coutant C. C. & Cada G. F. (1985). Analysis and development of a project evaluation process. U.S. Department of Energy, Bonneville Power Administration, Division of Fish and Wildlife, Portland, Oregon. Cooper, R., Easingwood, C., Edgett, S., Kleinschmidt, E., & Storey, E.C. (1994). What distinguishes the top performing new products in financial services. Journal of product innovation management, 11 (4), 281-299.
Cowell, D. W. (1988). New Service Development. Journal of marketing management, 3 (3), 296-312. Crawford, P., & Bryce, P. (2003). Project monitoring and evaluation: a method for enhancing the efficiency and effectiveness of aid project implementation. International Journal of Project Management, 21(5), 363-373. Creswell J.W (2003). Research Design: Qualitative, Quantitative, and Mixed Method Approaches, second edition. Sage Publications. Inc. Danks, D. (1997). When big means ugly. MIS, July, 56-60. Day, G. S. (2007). Is It Real? Can We Win? Is It Worth Doing? Harvard Business Review, 85 (12), 110-120. De Brentani, U. D. (1995). New Industrial Service Development:Scenarios for Success and Failure. Journal of Business Research, 32 (2), 93-103. De Brentani, U. D. (1991). Success factors in developing new business services. European Journal of Marketing 25 (2) , 33-57. Dolfsma, W. (2004). The Process of New Service Development - Issues of formalitization and appropriability. Erasmus research institute of manageement, 1-20. Doloi, H., & Jaafari, A. (2002). Conceptual Simulation model for strategic decision evaluation in project management. Logistics Information Management, 15 ( 2), 88104. Dover, P.A. (1987). Innovation in banking: the in-home computerised banking example. International Journal of Bank Marketing, 5 (1), 39-54. Easingwood, C. J. (1986). New Product Development for Service Companies. Journal of Product Innovation Management, 3 (4), 264-275. Edgett, S., & Jones, S. (1991). New product development in the financial service industry: a case study. Journal of marketing management, 7 (3), 271-284. EON website: http://www.eon.com Erlandson, D.A., Harris, E.L., Skipper, B.L., & Allen, S.D. (1993). Doing Naturalistic Inquiry: A Guide to Methods. Newbury Park, CA: Sage Publications. Farbey B., Land F. & Targett D. (1992) Evaluating investments in IT. Journal of Information Technology, 7 (2), 109 -122.
Fox, G. E., & Baker, N.R (1985). Project selection decision making linked to a dynamic environment. Management Science, 31 (10), 1272- 1285 Frechtling, J. (2002). The 2002 user friendly handbook for project evaluation. Arlington: National Science Foundation. Gardiner, P., 2005. Project Management: A Strategic Planning Approach. New York: Palgrave MacMillan. Gadrey, J., Gallouj, F., & Weinstein, O. (1995). New modes of innovation How services benefit industry. International Journal of Service Industry Management, 6 (3), 4-16. Gallouj, F., & Weinstein, O. (1997). Innovation in services. Research policy, 26 (4-5), 537-556. Gifford, S., & Wilson, C. (1995). A model of project evaluation with limited attention. Economic Theory, 5(1), 67-78. Grabe S. (1983) ‘Evaluation Manual’. UNESCO, Imprimerie de la Manutention, Mayenne, France. Graham, D. (2006). Evaluation Lessons: Green groups must measure gains. Regeneration & Renewal, Haymarket Business Publications Ltd, London. Greene, J. G. (1988). Stakeholder Participation and Utilization in Program Evaluation. Evaluation Review, April, 12 (2), 91-116. Griffin, A., (1997). PDMA research on new product development practices: updating trends and benchmarking best practices. Journal of Product Innovation Management, 14 (6), 429–458. Grönroos, C. (1990). Service Management and Marketing: Managing the Moments of Truth in Service Competition. Lexington: Lexington Books, Lexington MA. Grönroos R. (2000). Service Management and Marketing. A customer relationship management approach. Second edition. John Wiley and Sons, ltd, England Grönroos R. (1998). Service marketing theory: back to basics. Working paper. CERS, Finland. Hares, J. & Royle, D. (1994). Measuring the Value of Information Technology. John-Wiley & Sons, Chichester. Heany, D. F. (1983). Degrees of product innovation. The Journal of Business Strategy, 3 (4), 3.
Hertog, P. (2000). Knowledge-Intensive Business Services as Co-Producers of Innovation. International Journal of Innovation Management, December, 4(4), 491. International Development Research Centre (IDRC), Canada. http://www.idrc.ca Jackson, B. (2000). Designing Projects and Project Evaluations Using The Logical Framework Approach. IUCN Monitoring and Evaluation (M&E) Initiative , 1-11. Accessed online at 29October 2008. Available at: http://www.infra.kth.se/courses/1H1146/Files/logicalframeworkapproach.pdf Jackson. R. W,. & Cooper. P. D., (1988). Unique Aspects of Marketing Industrial Services. Industrial Marketing Management 17 (2), 111–118. Johne, A., & Storey, E.C., (1998). New service development : a review of the literature and annotated bibliography. European Journal of Marketing, 32 (3), 184-251. Johnson, S.P., Menor, L.J., Roth, A.V., & Chase, R.B., (2000). A critical evaluation of the new service development process: integrating service innovation and service design. Sage Publications, Thousand Oaks, CA, 1–32. Jong, J. d., Bruins, A., Dolfsma, W., & Meijaard, J., (2003). Innovation in service firms explored: what, how and why? Literature review. Business and Policy Research, Available at: http://www.ondernemerschap.nl/pdf-ez/B200205.pdf (Accessed online 01November 2008) Jong J. & Vermeulen P., (2003). Organising successful new service development: a literatue review. Management Decision Journal, 41 (9), 844-855. Kelly, D., (2000). New service development: initiation strategies. International Journal of Service Industry Management, 11 (1), 45-62. Klivans J. M., (1990). Launching a financial service: A case study in persistence. The Journal of business strategy, 11 (5) 8-11. Kumar K., (1990). Post Implementation Evaluation of Computer-Based Information Systems: Current Practices. Communications of the ACM, 33 (2). Available at : http://www.portal.acm.org (Accessed online 24October 2008) Langeard, E. & Eiglier, P., (1983). ‘Strategic management of service development’, In Berry, L.L. et al. (Eds), Emerging Perspectives on Services, AMA, Chicago, IL, 6872. Levitt, T., (1976). The industrialization of service. Harvard Business Review. 54(5): 63-74. Liang, W.-Y. (2003). The analytic hierarchy process in project evaluation An R&D case study in Taiwan. Benchmarking: An International Journal, 10 (5), 445-456. 90
Lopes, M. D., & Flavell, R. (1998). Project appraisal a framework to assess non-financial aspects of projects during the project life cycle. International Journal of Project Management, 16 (4), 223-233. Lovelock, C. (1984). Developing and implementing new services. Chicago: American Marketing Association. Lovelock, C. H. (1983). Classifying Services to Gain Strategic Marketing Insights. Journal of Marketing, 47(3), 9-20. Luftfartsverket Group website: http://www.lfv.se Maister, D.H., & Lovelock. C. H, (Summer 1982). Managing Facilitator Services. Sloan Management Review. 23 (Summer 1982),19–31. McNamara J.F (1994). Surveys and experiments in education research. Lancaster. PA, Technomic. Menor, L. J., Tatikonda, M. V., & Sampson, S. E. (2002). New service development: areas for exploitation and exploration. Journal of Operations Management, 20 (2), 135157. Menor, L.J., & Roth, A.V., (2007). New service development competence in retail banking: Construct development and measurement validation. Journal of Operations Management, 25 (4), 825–846. Messner, J. I., & Sanvido, V. E. (2001). An information model for project evaluation. Engineering, Construction and Architectural Management, 8 (5-6), 393-402. Meyer, M.H & De Tore, A. (2001). Perspective: creating a platform-based approach for developing new services. Journal of Product Innovation Management, 18(3), 188204. Milis, K., & Mercken, R. (2004). The use of the balanced scorecard for the evaluation of Information and Communication Technology projects. International Journal of Project Management, 22 (2), 87-97. Mohamed, S., & McCowan, A. K. (2001). Modelling project investment decision under uncertainty using possibility theory. International journal of project management, 19 (4), 231-241. Müller, R. (2003). Commercial Aspects of Project Management, An Introduction. Umeå School of Business and Administration, Umeå University , 1-14.
Nelson, R. R. (2006). Tracks in the Snow; IT projects are usually judged successes or failures when they go live. But look back and you will see the real judgment comes later, and that requires a new set of value criteria. CIO. Framingham, 19 (22), 1. OECD. (2000). Promoting innovation and growth in services. Organisation for Economic Cooperation and Development: Paris . Oral, M., Kettani,O., & Lang, P. (1991). A methodology for collective evaluation and selection of industrial R&D projects. Management Science, 37 (7), 871- 885. Örtengren, K. (2004). A summary of the theory behind the LFA method : The Logical Framework Approach. Sida , 1-40. Parsons, G.L. (1983). Information technology: a new competitive weapon. Sloan Management Review, Fall, 25 (1), 3-14. PMI. (2004). A guide to the Project management Body of Knowledge. Pennsylvania: PMI. Remenyi, D. (1995). Effective measurement and management of IT benefits and cost. Paper presented at the Management Training and Education Seminar, MTE, Melbourne. Riel A. & Lievens A. (2003). New service development in high tech sectors: a decision making perspective. In Van Riel, Allard C.R (2002), Effective Decision Making in the High Tech Service Innovation Process, Doctoral Dissertation, Maastricht University, Maastricht, Datawyse/Maastricht University Press. Saunders, M., Lewis, P., & Thornhill, A., 2007. Research Methods for Business Students. England: Pearson Education Limited. Scheuing, E.E & Johnson, M.E. (1989). New product development and management in financial institutions. International Journal of Bank Marketing, 7 (2), 17-21. Scriven, M. (1967). The methodology of evaluation, In Tyler, R. (Ed.), Perspectives of Curriculum Evaluation. Rand McNally, Chicago, IL, 39-83. Scriven M. 1980. The Logic of Evaluation. Edgepress: California. Segone M. 1998. Democratic evaluation. Working Paper, UNICEF. Schneider, B., & Bowen, D.E., (1984). New service design, development and implementation and the employee. In George, W.R., & Marshall, C.E. (Ed.), Developing New Services, American Marketing Association, Chicago, IL, 82-101. Shostack, G. L. (1984a). Designing Services that Deliver. Harvard Business Review 62 (January-Febuary) , 133-139. 92
Shostack, G.L. (1984b). ‘Service design in the operating environment’, in George, W.R. and Marshall, C.E. (Eds), Developing New Services, American Marketing Association, Chicago, IL, 27-43. Small, K. A., (1998). Project Evaluation. Chapter 5 for Transportation Policy and Economics: A handbook in honor of John R. Meyer. The University of California Transportation Center. Stevens, F., Lawrenz, F., & Sharp, L. (1993). User friendly handbook for project management: Science, Mathematics, Engineering and Technology Education. Washington DC: National Science Foundation. Stemler, S. (2001). An overview of content analysis. Practical Assessment, Research & Evaluation, 7 (17), 137. Stockdale, R., & Standing, C. (2006). An interpretive approach to evaluating information systems: A content, context, process framework. European Journal of Operational Research 173 (3) , 1090–1102. Stockdale, R., Standing, C., & Love, P.E. (2006). Propagation of a parsimonious framework for evaluating information systems in construction. Automation in Construction 15 (6) , 729 – 736. Storey E.C. (July 1993). The Impact of the New Product Development Project on the Success of Fmancial Services. The Service Industries Journal, 13 (3) , 40-54. Storey, E.C., & Easingwood, C. (1995). Determinants of new product performance: a study in the financial services sector. International Journal of service industry management 7 (1), 32-55. Storey, E.C. & Easingwood, C. (1994). New service success and the augmented service offering. Proceedings of Product Development & Management Association 18th International Conference, Boston, 5-10 Nov. 178-89. Storey C. & Kelly D. (2001) Measuring the Performance of New Service Development Activities. The Service Industries Journal, 21 (2) ,71–90. Strauss A. & Corbin J. (1990). Basics of Qualitative Research. Techniques and Procedures for developing Grounded Theory, second edition. New Burry Park, CA: Sage Publications Inc. Suwardy T., Ratnatunga J., Sohal A.S., & Speight G. (2003). IT projects: evaluation, outcomes and impediments. Benchmarking: An International Journal, 10 (4) General review.
Thomas, D.R.E. (1978). Strategy in different in service businesses. Harvard Business Review, July-August, 158-67. Tukel O. I & Walter O. R (2001). An empirical investigation of project evaluation criteria. International Journal of Operations & Production Management, 21 (3). 400 – 416. Twiss, B., (1986). Managing Technological Innovation, Pitman, London. Uhl A. (2000). The Limits of Evaluation. European Monitoring Centre for Drugs and Drug Addiction, Lisbon. Van der Aa, W. a. (2002). Realizing innovation in services. Scandinavian Journal of Management, 18 (2) 155-171. Vakola M. (2000) ‘Exploring the relationship between the use of evaluation in business process re-engineering and organisational learning and innovation’. Journal of Management Development, 19 (10), 812-835. Valeri & Rozenfeld (2004). Improving the flexibility of New Product Development (NPD) through a new quality gate approach. Society for Design and Process Science, 8 (3), 17-36. Wholey J, Hatry H, Newcomer K. 1994. Handbook of Practical Program Evaluation. Jossey-Bass: San Francisco, CA. Ye S. & Tiong R.L.K., (2000). NPV- at - risk method in infrastructure project investment evaluation. Journal of construction engineering and management (May-June) , 227233. Yeo, K., a& Qiu, F. (2003). The value of management flexibility—a real option approach to investment evaluation. International Journal of Project Management, 21 (4), 243250.