Internet Research A framework and methodology for evaluating e-commerce Web sites Rian van der MerweJames Bekker
Article information: To cite this document: Rian van der MerweJames Bekker, (2003),"A framework and methodology for evaluating e-commerce Web sites", Internet Research, Vol. 13 Iss 5 pp. 330 - 341 Permanent link to this document: http://dx.doi.org/10.1108/10662240310501612
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
Downloaded on: 16 June 2015, At: 01:01 (PT) References: this document contains references to 21 other documents. To copy this document:
[email protected] The fulltext of this document has been downloaded 4601 times since 2006*
Users who downloaded this article also downloaded: Chung-Shing Lee, (2001),"An analytical framework for evaluating e-commerce business models and strategies", Internet Research, Vol. 11 Iss 4 pp. 349-359 http://dx.doi.org/10.1108/10662240110402803 Mei Cao, Qingyu Zhang, John Seydel, (2005),"B2C e-commerce web site quality: an empirical examination", Industrial Management & Data Systems, Vol. 105 Iss 5 pp. 645-661 http://dx.doi.org/10.1108/02635570510600000 Eric Allen, Jerry Fjermestad, (2001),"E-commerce marketing strategies: an integrated framework and case analysis", Logistics Information Management, Vol. 14 Iss 1/2 pp. 14-23 http://dx.doi.org/10.1108/09576050110360106
Access to this document was granted through an Emerald subscription provided by emerald-srm:499410 []
For Authors If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service information about how to choose which publication to write for and submission guidelines are available for all. Please visit www.emeraldinsight.com/authors for more information.
About Emerald www.emeraldinsight.com Emerald is a global publisher linking research and practice to the benefit of society. The company manages a portfolio of more than 290 journals and over 2,350 books and book series volumes, as well as providing an extensive range of online products and additional customer resources and services. Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committee on Publication Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archive preservation. *Related content and download information correct at time of download.
Introduction
A framework and methodology for evaluating e-commerce Web sites Rian van der Merwe and James Bekker
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
The authors Rian van der Merwe is Research Assistant and Student, School of Marketing, Curtin University of Technology, Perth, Australia. James Bekker is a Lecturer in Information Systems and Computer Simulation, Department of Industrial Engineering, University of Stellenbosch, South Africa. Keywords Electronic commerce, Internet, Function evaluation Abstract The corporate Web site is an indispensable part of any ecommerce venture. If users find it difficult to do electronic transactions, they will visit another online source, or go to a physical source instead. Evaluating the performance of ecommerce Web sites has received some attention in the past few years, but there is still a pressing need for a more comprehensive approach. This paper attempts to address this need by providing an e-commerce Web site evaluation framework and method built on solid business principles, and using multidimensional scaling to analyse evaluation data and present the results in a graphic manner for easy interpretation. Electronic access The Emerald Research Register for this journal is available at http://www.emeraldinsight.com/researchregister The current issue and full text archive of this journal is available at http://www.emeraldinsight.com/1066-2243.htm Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . pp. 330-341 # MCB UP Limited . ISSN 1066-2243 DOI 10.1108/10662240310501612
The Internet and e-commerce have become extremely important avenues for firms in many industries to interact with their customers and other stakeholders. E-commerce, with the Web as primary infrastructure, offers several tangible and intangible benefits (Kim et al., 2003). For any firm with an online presence, the Web site is the platform used to communicate with customers and to facilitate business transactions. Web sites that do not provide positive experiences may cause customers to decide that it is easier to go to a physical store rather than purchase online. Forrester Research estimates that ``poor Web design will result in a loss of 50 per cent of potential sales due to users being unable to find what they want, and a loss of 40 per cent of potential repeat visits due to initial negative experience'' (Cunliffe, 2000). The question is, however, how should companies go about conceptualising and continually improving their Web sites. Managers need information on Web site performance in order to improve effectiveness and usability. Currently, no generally accepted method exists to evaluate the performance of Web sites systematically and comprehensively. There is a need for a complete and accurate method to evaluate the performance of Web sites, not only individually but also in comparison with the Web sites of other companies in the same industry. This article attempts to address this need by focusing on two objectives: (1) To develop a framework and criteria for the comprehensive evaluation of e-commerce Web sites. (2) To use this framework and sound statistical reasoning to develop a method that can be used to evaluate Web sites quantitatively. This paper starts with a brief review of the literature on Web site evaluation, followed by a discussion of a framework and method for evaluation. The paper concludes with the identification of further managerial applications and research opportunities, and also points out some of the limitations of the methodology.
330
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
Web site evaluation: a review of the literature Boyd Collins developed the first formal approach to the evaluation of Web sites in late 1995. He founded the Infofilter project, a model intended for librarians who needed to evaluate the quality of information on the Internet. The model was based on six criteria, developed by combining evaluation criteria for printed media, and considering what was relevant for Web sites. These criteria are content, authority, organisation, searchability, graphic design and innovative use. Although innovative at the time, it is clear that the Internet and its uses have changed to such an extent that these criteria are no longer sufficient to evaluate Web sites. The Infofilter project ceased operation in July 1997 because its participants realised that it was becoming obsolete. It seems, however, that many authors have not realised that these criteria have to a large extent lost their relevance. Using common Internet search engines to find literature on Web site evaluation reveals hundreds of Web sites claiming to provide Web site evaluation criteria (see Alexander and Tate, 1996). When studied closely it becomes clear that most of these sites use the Collins criteria as a platform to which they arbitrarily seem to add their own. There is little evidence of any research on which these are based, which makes them unreliable to a large extent. The biggest problem with them is that they are based on traditional marketing strategies for printed media. The past few years have also witnessed an increase in the number of research articles on Web site evaluation published in academic journals. The advice given in these publications ranges from extremely rigid (``Use no more than three images per page'') (D'Angelo and Little, 1998) to very vague (``Design the Web site for content, not appearance'') (D'Angelo and Little, 1998). Generally, however, these publications are much more reliable than the multitude of Web sites claiming to provide criteria for Web site evaluation. They even move a step beyond mere front-end evaluation. Berthon et al. (1996) developed a model for the evaluation of Web site effectiveness based on hierarchy of effects models, which are well known in buying behaviour. Other approaches
include the static-quality/dynamic-quality approach by Day (1997); user-defined criteria developed by Misic and Johnson (1999); the Web site quality evaluation method (QEM) by Olsina et al. (1999) and automatic classification and evaluation by Bauer and Scharl (2000). Literature on Web site evaluation that focuses specifically on e-commerce sites is scarce. One approach is the Web assessment model (WA model), which was developed in 1997 at the Competence Centre for Electronic Markets (CCEM) at the University of St Gallen in Switzerland. Schubert and Selz (1999) describe the development of this model in detail. Liu et al. (2000) made effective progress in the field by developing criteria for the development of e-commerce Web sites using the inputs of Fortune 1,000 companies' Web masters. While the approaches to Web site evaluation discussed above have made some contributions, they only focus on a specific aspect of a Web site, and none provides a method for the comprehensive evaluation of e-commerce Web sites. Ideally, these approaches need to be combined and extended to develop a framework that evaluates every aspect of an e-commerce Web site, as proposed by Kim et al. (2003).
An e-commerce Web site evaluation framework For reasons that become clear in the next few paragraphs, the proposed framework for the evaluation of an e-commerce Web site addresses three different levels of detail. The first level consists of five criteria categories, which serve as a structure for evaluation, covering all the relevant aspects of e-commerce Web sites. The second level involves four criteria groups within each of the criteria categories, and provides more detail and definition to the categories. On the third and most detailed level, five individual criteria are developed within each of the criteria groups, and are used to perform the evaluation. The levels are as follows. Level 1 The purpose of the framework and its criteria categories is to find a way to comprehensively
331
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
Rian van der Merwe and James Bekker
evaluate e-commerce Web sites. To fulfil this purpose, the customer buying cycle is used as a framework for e-commerce Web site evaluation (see Robinson et al., 1967; Schoel and Guiltinan, 1992, for two approaches to the customer buying cycle). This was done for two reasons: (1) Comprehensive evaluation of all aspects of e-commerce Web sites. By following every action a customer will perform on an e-commerce Web site when a purchase is made, the buying cycle will ensure that all the relevant parts of the Web site are evaluated. (2) Total customer focus. The customer buying cycle puts the focus on the processes that customers follow when they pay for products/services. They are the users of the Web sites, so any evaluation framework must focus specifically on their experience of the sites. As mentioned above, there are several views in the marketing literature on customer buying processes (see Berthon et al., 1996; Churchill et al., 1993). A basic thread runs through these approaches, and this paper regards the customer buying cycle as consisting of four distinct phases. These phases are shown below with an appropriate example to explain each of them: (1) Need recognition. A potential customer realises a specific need, for example, he/she would like to read a book on a particular subject, say aviation in the Second World War. (2) Gather information. The customer goes to a bookstore and finds all the resources on aviation during the Second World War. (3) Evaluate information. The customer reads the summaries of the books on the subject, and decides which of the resources he/she is going to buy. (4) Make purchase. The customer goes to the checkout counter and buys the book he/she has selected.
Figure 1 shows how these links were made and an explanation follows below. During the need-recognition phase, potential customers will visit a Web site that they believe will be able to fulfil their specific needs. The interface is very important at this stage, because this determines the user's first impression of the site. The interface refers to the visual aspects of the site ± how attractive it is visually and what backgrounds, fonts and colours are used, etc. Although Berthon et al. (1996) state that, on the Internet, the buying cycle actually starts before the need-recognition phase, when the customer tries to find the site, the approach followed in this paper focuses specifically on the Web site itself. During the information-gathering phase, customers explore the site and search for the products/services in which they are interested. During this phase, navigation is the most important aspect of the site. This refers to the process of moving around between the different pages of the Web site. Aspects such as the straightforward, logical structure of the site, and menus that are easy to understand are important in this phase. During the third phase of the buying cycle (evaluate information), customers make choices about which products/services they want to purchase. The content of the site is, therefore, very important. Content refers to the actual information on the site, for example, the amount and quality of information on the site. In the final phase of the buying cycle (make purchase), customers purchase the products/ services they have chosen. The reliability of the site is extremely important at this stage and is defined as consisting of two aspects: (1) The degree to which a customer is able to use the order process on the site easily and effectively. (2) The degree to which the company is able to fulfil its promises and obligations to customers every time a purchase is made.
To use this buying cycle as a framework for the evaluation of e-commerce Web sites, it is necessary to find links between each of the buying phases and a specific aspect of a Web site that is most important during that phase.
Finally, several technical aspects of a Web site are essential to its proper functioning. As shows, these aspects are important during all the phases of the buying cycle, and are, therefore, the backbone of the site.
332
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
Figure 1 Development of e-commerce Web site evaluation framework
Level 2 With the first level of the evaluation framework completed, the second level to be developed is the criteria groups within each category. These groups were derived from the literature on Web site evaluation using popular criteria listed by several authors (e.g. Berthon et al. 1996; Abels et al. 1999; Cunliffe, 2000; Bauer and Scharl, 2000; Kim et al., 2003). These criteria were then classified according to the researchers' judgement under the most appropriate category defined on level 1. Four groups per criteria category were identified, as shown in Figure 2. Each of these groups is explained in Table I.
Level 3 The third level of the evaluation framework consists of specific criteria within each criteria group. Although it cannot be completely eliminated, the purpose of the criteria on this level is to minimize subjectivity during the evaluation process. These criteria are proposed by the researchers and are debatable. Five criteria were developed within each of the 20 criteria groups described above, resulting in a total of 100 criteria. The criteria are shown in Table II-VI. Note that language is not part of the criteria, as it is assumed that e-commerce is
Figure 2 E-commerce Web site evaluation framework and criteria groups
333
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Table I Description of e-commerce Web site evaluation criteria groups Phase
Criteria group
This criteria group evaluates/measures
Interface
Graphic design principles
The effective use of colour, text, backgrounds, and other general graphic design principles The effectiveness of the graphics and multimedia used on the site Whether or not the text is concise and relevant, and the style good The degree to which the interface is designed to handle exceptions, for example, text-only versions of pages
Graphics and multimedia Style and text Flexibility and compatibility Navigation
Logical structure Ease of use Search engine
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
Navigational necessities
The organisation and menu system of the site The ease of navigation to find the pages that the user is looking for The search engine's ability to find the correct pages easily and provide clear descriptions of the search results Other important aspects of navigation like the absence of broken links and ``under-construction'' pages
Content
Product/service-related information Company and contact information Information quality Interactivity
Whether or not the products/services are described precisely and thoroughly Whether or not it is easy to find information on the company, its employees and its principals The currency and relevance of the content on the site How much input the user has on the content displayed on the site
Reliability
Stored customer profile
The registering process and how the company uses the stored customer profile The effectiveness and ease of use of the online order process The company's actions from order placement until the order is delivered How the company communicates and helps its online customers
Order process After-order to order receipt Customer service Technical
Speed Security Software and database System design
Different aspects of the loading speed of the site Security systems and the ways used by the company to protect customers' privacy on the site Flexibility in terms of different software used. Also looks at the data software and data communication systems used on the site The correct functioning of the site and how well it integrates with internal and external systems
done in English, even within non-English countries.
Evaluation concept Evaluation data for a given Web site are gathered by going through the list of evaluation criteria and awarding a value to each item. This value will be an indication of how well the Web site adheres to the specific criterion. All the individual values for the criteria in a specific group are then summed to a unique score for that group. The group scores within each category are then summed, which generates a scale of criteria category scores indicating how
well a Web site performs in each of the categories. As soon as more than one site is evaluated using this procedure, the data can be presented in tabular form. The result is a matrix of values achieved, with the column variable being criteria categories and the row vector being different Web sites within the same industry. A given matrix therefore contains the results for different Web sites in one industry. The values in the cells of the matrix represent the score of each site for a specific criteria category, as calculated from the evaluation criteria. An example is shown in Table VII. One of the problems associated with this type of table is the lack of ease and efficiency of interpretation of the contained information.
334
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Table II E-commerce Web site evaluation criteria ± interface
Table IV E-commerce Web site evaluation criteria ± content
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
1 Interface
3 Content
Group 1: Graphic design principles
Home page concise and clear Effective use of white space Effective and consistent use of colour Effective and consistent use of backgrounds Effective graphics/typeface/colour combinations
Group 1: Product/servicerelated content
Extensive product/service information available Price information and cost benefits communicated Adequate breadth of product range Adequate amount of advertising of own products Adequate amount of advertising by other companies
Group 2: Graphics and multimedia
Site visually attractive Graphics and multimedia make a contribution to the understanding and navigation of site Icons easy to understand Not excessively used Size of media: no negative impact on loading times
Full company information available Group 2: Terms and conditions easily accessed Company and contact information E-mail addresses of employees available Telephone and fax numbers available Postal and physical addresses available
Group 3: Style and text
Style of pages consistent Typefaces consistent and easy to read Correct spelling and grammar Text concise and relevant Purpose of site made clear on home page
Group 4: Flexibility and compatibility
Pages sized to fit in browser window Printable versions of certain pages available Text-only version available Foreign language support available Accommodation made for disabled users
Group 3: Content current and updated Information quality Content relevant to purpose of site Content concise and non-repetitive High perceived quality of product/service Quality of advertisements high Group 4: Interactivity
User able to customise content to fit needs Large amount of personalisation possible Easy access to online community High perceived value of online community High degree of interactivity in community
Table III E-commerce Web site evaluation criteria ± navigation 2 Navigation
Table V E-commerce Web site evaluation criteria ± reliability 4 Reliability
Group 1: Logical structure
Intelligible, straightforward organising scheme Content logically structured in different sections and levels Menus understandable and straightforward Site map/table of contents available Consistent navigation throughout site
Group 2: Ease of use
Easy Easy Easy Easy Easy
Group 3: Search engine and help function
Easy to use search engine Search engine accurate Good description of search engine findings No search engine errors Help function easy to use
Online order tracking available Group 3: After-order to order Effective payment settlement Confirmation of order dispatch sent to customer receipt On-time delivery Full order delivery
Group 4: Navigational necessities
No broken links No ``under-construction'' pages Links clearly discernible, well labelled and defined Clear label of current position on site Effective use of frames, non-frames version available
Group 4: Customer service
to to to to to
find site explore specific idea or subject return to main page find specific information access complete product/service range
Group 1: Stored customer profile
Easy to register on site High perceived benefits from registering Easy to log on to site Adjustable customer profile stored Guided ordering using customer profile available
Group 2: Order process
Transparent, interactive and easy order process Easy selection of generic services Alternative methods of ordering/payment available Good stock availability Acknowledgement of order sent to customer
335
Feedback forms available Good after-sales support Quick reply to e-mail enquiries General courtesy of company good Good after-sales relationship building
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Table VI E-commerce Web site evaluation criteria ± technical
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
4 Technical Group 1: Speed
Fast home page loading speed Fast sub-page loading speed Good perceived use of cacheing Good consideration of non-broadband users Good perceived speed of database
Group 2: Security
Security systems accredited Secure payment systems used Privacy of users protected Security protocols well communicated Security certificates adequate
Group 3: Software and database
Good cross-browser capability Users advised on ideal browser and resolution Database software adequate for size of database Good data transfer between systems No perceived duplication of data in database
Group 4: System design
Precise operation and computation Good resolution compatibility Good integration with systems of users Good integration of different systems on site Global accessibility to products/service available
The next section introduces correspondence analysis as the technique used to analyse the evaluation data and represent it graphically, which enables easy and efficient interpretation.
Correspondence analysis In order to quantify, interpret and assess the evaluation data, it is imperative to find a suitable technique to analyse data that are presented in this form. The purpose of multidimensional scaling techniques is to condense large amounts of data to as few dimensions as possible, while still describing it adequately. Formally, multidimensional scaling can be defined as ``[a set of mathematical techniques] concerned primarily with the
spatial representation of relationships among behavioural data'' (Green et al., 1989). To paraphrase this definition, multidimensional scaling techniques seek to find an understandable visual representation of the similarities, dissimilarities or ordered rankings between different data points. All that is needed to perform multidimensional scaling is a matrix of numbers that expresses all the similarities, dissimilarities or rankings within a group of objects ± like the matrix discussed at the end of the previous section. Correspondence analysis is part of multidimensional scaling and will be used here. It originated with a French analyst called Jean-Paul BenzeÂcri in the early 1960s (see Greenacre, 1984), and can be defined as ``a technique for displaying the rows and columns of a data matrix (primarily a two-way contingency table) as points in dual lowdimensional vector spaces'' (Greenacre, 1984). The strength of this technique is that it provides a way to examine relations not only between row variables and column variables individually but also between row and column variables together. This implies that, if used in the context under discussion, it would be possible to compare the similarities that different Web sites have with each other, or the similarities that different criteria groups have with each other. Furthermore, it would also be possible to compare individual Web sites with the different criteria groups. The mathematical concepts of correspondence analysis are discussed extensively in Greenacre (1984) and Weller and Romney (1990). Correspondence analysis enables one to examine data and visually show the complex relationships between different Web sites, between different criteria, and between sites and criteria, on a low-dimensional scale (usually two dimensions).
Table VII Example of a two-way contingency matrix for Web site evaluation Criteria category Criteria category Criteria category Criteria category Criteria category 1 2 3 4 5 Web Web Web Web
site site site site
A B C D
67 87 78 37
78 74 64 57
63 48 49 85
336
78 93 56 73
23 67 83 65
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
E-commerce Web site evaluation method A Web site evaluation method is now proposed. The term method, as used in this context, is defined as all the procedures followed to evaluate Web sites ± from the gathering of evaluation data to the interpretation of the output results. Since evaluation must be done on the same basis, the method requires that the e-commerce Web sites be evaluated by industry. Each industry must be clearly defined, while Web sites must be classified and assigned to a specific industry. To ensure objectivity, it is suggested that at least three evaluators do the evaluation. The three-step evaluation method follows, and it is supported by an example (Van der Merwe, 2001). Step 1: Gather e-commerce Web site evaluation data and present the data in a two-way contingency matrix Each of the evaluators individually assesses the Web site in question against the 100 evaluation criteria. The scoring method requires that the evaluator scores the Web site according to the level to which he agrees with the statement made by each criterion. For each criterion a score between 2 (strongly disagree) and +2 (strongly agree) is awarded. Only four scoring options are provided (2, 1, 1 or 2) so the evaluator is forced to decide whether he agrees or not ± a neutral answer (0) is not possible. A fifth option, not applicable, is provided for cases in which the criterion is not at all relevant to the sites in the industry under evaluation. In this case, a score of 0 is awarded to all the sites in that industry, which ensures that it has no positive or negative effect on the outcome of the evaluation. After the evaluations are completed the group should go through the criteria together, discuss their opinions, and award an agreed final score to each criterion. Any subjectivity that exists within the criterion is thus reduced to a large extent. This process can be facilitated through online group collaboration (see Bamber et al., 1995). The sum of the final scores of the five criteria in each criteria group gives the 20 criteria group totals. The four group totals within each of the five criteria categories in the evaluation framework are then summed and
these five totals provide a scale of evaluation of data for the site. This process is repeated for all the Web sites that are evaluated in a specific industry. When all the sites have been evaluated, a contingency matrix is created which is used for correspondence analysis. As an example, this method is applied to six Web sites in the online books industry. After an experimental evaluation, the matrix shown in Table VIII was obtained. Step 2: Perform correspondence analysis on the matrix and represent the results graphically In this step, computer software is used to perform correspondence analysis on the contingency matrix and to plot the results. An example of the plot that resulted from correspondence analysis of the data in Table VIII is shown in Figure 3. Step 3: Interpret the results in the graphical representation and make recommendations The interpretation process should be done in three distinct phases: (1) Use the correspondence analysis graph to draw basic conclusions about the performance of each of the Web sites and the profile groups that can be identified. This is done by assessing the proximities of points to each other. Three aspects of the graph are considered: the criteria category profiles on their own, the Web site categories on their own, and the criteria category profiles and Web site profiles together. (2) Use these preliminary findings and combine them with the original data in the contingency matrix to refine the conclusions. (3) For each Web site, use its individual criteria group scores to make recommendations on possible ways to improve the site, and how to do so. The following conclusions can be drawn from the correspondence analysis results shown in Table VIII and Figure 3.
337
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Table VIII Contingency matrix for the books industry Site name
Site address
Amazon.com Books Barnes & Noble Exclusive Books Kalahari.net PC Books, Inc. VS Online
www.amazon.com/books www.bn.com www.exclusivebooks.com www.kalahari.net www.pcbooks.co.za www.vsonline.co.za
Interface Navigation 17 15 14 ±15 ±19 6
35 29 35 11 ±4 14
Content 30 25 18 13 1 ±9
Reliability Technical 31 27 27 24 16 6
28 19 23 12 22 11
Total 141 115 117 45 16 28
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
Figure 3 Correspondence analysis graph for the book industry
Analysis of criteria category profiles Based on the lack of proximity between criteria
The other three sites have quite different profiles from this group and from each other.
category profiles, it is evident that the criteria
Analysis of criteria category and Web site profiles together First, it is evident that Amazon.com Books, Barnes and Noble and Exclusive Books all lie more or less in the centre of the graph, with the five criteria forming a rough circle around them. This implies that these three sites generally received the highest scores for the criteria. From their close proximity to the Navigation criteria category, it can be deduced that these three sites not only received higher scores for the Navigation category than for the other categories, but also that their scores for
categories form no distinctly similar groups in this case. This implies that their profiles differ. In other words, two or more specific criteria do not feature considerably more on some sites than on others. Analysis of Web site profiles Based on proximities, it is evident that the sites of Amazon.com Books, Barnes and Noble and Exclusive Books form a group of similar profiles. It appears therefore that these sites received similar scores for some of the criteria.
338
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
Navigation are higher than the scores that the other three sites received for this category. Kalahari.net was positioned almost halfway between the content and reliability criteria categories. Although its total score was not very high, it seems that, compared to the other categories, this is where Kalahari.net received the highest scores. PC Books and VS Online lie on the periphery of the graph and not in close proximity to any of the categories. It is, therefore, clear that they have not received particularly high scores for any of the criteria. Summary Based on the above discussion, the following conclusions can be made: . Clearly, Amazon.com, Barnes and Noble and Exclusive Books are the industry leaders in terms of Web site effectiveness. Particularly in terms of navigation, the other sites can learn a lot from these three. . Kalahari.net needs major improvements, but its overall performance is not too bad. They should start by improving the interface, technical and navigation aspects of the site. . VS Online and PC Books are in a bad condition and drastically need to improve their sites in all areas. . When the correspondence analysis graph and the contingency matrix are analysed together, it is clear that all sites need to improve their interfaces. Kalahari.net and PC Books in particular have to make drastic improvements in this area. The graphical analysis can then be extended by looking at the individual site results for each site and deciding on specific criteria groups and individual criteria that have to be addressed. Managers can use the information from the evaluation results to identify : . areas where their Web sites need improvement; . other Web sites that perform well in the aspects that need improvement (for benchmarking purposes); and . the areas in which their Web sites perform well.
This completes the proposed method for comprehensive evaluation of e-commerce Web sites. The method is summarised in Figure 4.
Conclusions, recommendations, managerial and research implications At present, firms engaged in e-commerce are limited in their ability to gauge their online performance based on objective criteria. Their ability to compare their performance with other companies in the same industry is also restricted. The technique presented in this paper is structured and repeatable, and gives managers the opportunity to evaluate the as-is situation of their Web sites (assuming that they wish to do so) with respect to those of other companies. They can use the approach to identify where they perform well, and where there is room for improvement. Managers can determine which sites within their industry perform better than they do in particular areas, and why. This objective information can be used to make strategic decisions on how to improve the efficiency and performance of their Web sites. Practising and academic marketing researchers can also apply the technique. Practitioners can identify who the leaders are in a specific industry, as well as why they do so well. Academics can conduct cross-industry research, which may reveal aspects of Web sites within one industry that may have been neglected in others. They will be able to quantify the performance of Web sites and use this as a dependent or independent variable in research directed at determining the consequences of Web site performance. There are, however, a few reservations about the individual criteria: . Although the objectivity of criteria was a major driving force behind the development process, the criteria still entail a certain measure of human subjectivity, as mentioned above. Although the evaluation framework and criteria groups seem to be reasonably inclusive, it may be necessary to modify or even replace some of the criteria to ensure greater objectivity in the process. The criteria of level 3 requires further evaluation, and should perhaps be
339
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
Figure 4 Summary of the e-commerce Web site evaluation method
.
identified through consensus of experts in the field. Some of the criteria cannot be evaluated without intricate knowledge of the ordering process of the companies being evaluated. These criteria are essential to the evaluation; therefore, e-commerce companies that are evaluated must be prepared to explain their process.
With regard to correspondence analysis, the following must be noted: . Interpretation of correspondence analysis results tends to become less accurate as the number of sites to be evaluated decreases. It is, therefore, recommended that the graph is used to make initial conclusions, and that the contingency matrix is then used to validate and elaborate on these conclusions. . One drawback of the technique is that the results deal with relative positions. Suppose, for example, that the point of a specific Web site lies very close to a certain criteria category on the plot. This does mean that the site scored well in that category, compared to its other scores, but it does not necessarily mean that it received the highest score for that category of all the sites that were evaluated. Other sites with high overall scores and a more even distribution of scores may well have scored higher in that category. Also, no conclusions can be drawn from the correspondence analysis graph regarding the absolute scores of the Web sites. The correspondence analysis graph has no way of communicating this piece of information to the user. The recommendation is that
the graphs be studied with the contingency matrix close at hand, so that an accurate conclusion can also be drawn about the absolute performance of the Web sites in the industry under consideration. It is recommended that further study be conducted in the wider field of statistical mapping. In this paper, the focus was on correspondence analysis as a specific technique within this field. Other multidimensional techniques such as metric scaling or cluster analysis may be better suited to perform the analysis of evaluation data. Furthermore, the technique of intercoder reliability (see Kassarjian, 1977) could be useful to assess the reliability of the evaluators themselves. Further research should evaluate the performance of the proposed method against that of other existing evaluation methods (see, for example, Kim et al., 2003). It is also important to note that it takes a considerable amount of time to complete an evaluation. The researchers found that it took an average of two to three working days per industry to evaluate the Web sites thoroughly. The evaluators have to spend time getting to know the site (if they have not used it considerably before), and a dedicated effort is needed to evaluate each of the 100 evaluation criteria. It is, however, not a trivial undertaking to evaluate an e-commerce Web site, and it is therefore essential that enough time be spent to perform an accurate and useful evaluation. Although many factors have to be considered for an Internet company to be successful, the Web site is one of the essential aspects that have to be effectively in place for e-commerce business to succeed. The evaluation method
340
A framework and methodology for evaluating e-commerce Web sites
Internet Research: Electronic Networking Applications and Policy Volume 13 . Number 5 . 2003 . 330-341
Rian van der Merwe and James Bekker
proposed in this paper aims to assist managers to achieve this, should they wish to improve their e-commerce enterprises.
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
References Abels, E.G., White, M.D. and Hahn, K. (1999), ``A user-based design process for Web sites'', OCLC Systems & Services, Vol. 15 No. 1, pp. 35-44. Alexander, J. and Tate, M. (1996), ``Evaluation criteria'', available at: http://hoku.lcc.hawaii.edu/kaz/100/ evalCriteria.htm Bamber, M., Hill, M.C. and Watson, R.T. (1995), ``Group support systems can improve your meetings'', The CPA Journal, Vol. 65 No. 3, pp. 32-7. Bauer, C. and Scharl, A. (2000), ``Quantitative evaluation of Web site content and structure'', Internet Research: Electronic Networking Applications and Policy, Vol. 10 No. 1, pp. 31-43. Berthon, P.R., Pitt, L.F. and Watson, R.T. (1996), ``The World Wide Web as an advertising medium: toward understanding of conversion efficiency'', Journal of Advertising Research, Vol. 36 No. 1, pp. 43-53. Churchill, G.A. Jr, Ford, N.M. and Walker, O.C. Jr (1993), Sales Force Management, 4th ed., Richard D. Irwin, Homewood, IL. Cunliffe, D. (2000), ``Developing usable Web sites ± a review and model'', Internet Research: Electronic Networking Applications and Policy, Vol. 10 No. 4, pp. 295-307. D'Angelo, J. and Little, S.K. (1998), ``Successful Web pages: what are they and do they exist?'', Information Technology and Libraries, Vol. 17 No. 2, pp. 71-81. Day, A. (1997), ``A model for monitoring Web site effectiveness'', Internet Research: Electronic Networking Applications and Policy, Vol. 7 No. 2, pp. 109-15. Green, P.E., Carmone, F.J. and Smith, S.M. (1989), Multidimensional Scaling: Concepts and Applications, Allyn and Bacon, Needham Heights, MA. Greenacre, M.J. (1984), Theory and Applications of Correspondence Analysis, Academic Press, London.
Kassarjian, H.H. (1977), ``Content analysis and consumer research'', Journal of Consumer Research, Vol. 4, June, pp. 8-18. Kim, S.-E., Shaw, T. and Schneider, H. (2003), ``Web site design benchmarking within industry groups'', Internet Research: Electronic Networking Applications and Policy, Vol. 13 No. 1, pp. 17-26. Liu, C., Arnett, K.P. and Litecky, C. (2000), ``Design quality of Web sites for electronic commerce: Fortune 1000 Webmasters' evaluations'', Electronic Markets, Vol. 10 No. 2, pp. 120-9. Misic, M.M. and Johnson, K.L. (1999), ``Benchmarking: a tool for Web site evaluation and improvement'', Internet Research: Electronic Network Applications and Policy, Vol. 9 No. 5, pp. 383-92. Olsina, L., Godoy, D., Lafuente, G.J. and Rossi, G. (1999), ``Specifying quality characteristics and attributes for Web sites'', paper presented at the 1st ICSE Workshop on Web Engineering (WebE-99), Los Angeles, CA, available at: http://citeseer.nj.nec.com/265461.html Robinson, P.J., Faris, C.W. and Wind, Y. (1967), Industrial Buying and Creative Marketing, Allyn and Bacon, Boston, MA. Schoel, W.F. and Guiltinan, J.P. (1992), Marketing, 5th ed., Allyn and Bacon, Needham Heights, MA. Schubert, P. and Selz, D. (1999), ``Web assessment ± measuring the effectiveness of electronic commerce sites going beyond traditional marketing paradigms'', Proceedings of the 32nd Hawaii International Conference on System Sciences, Maui, Hawaii. Internet and the Digital Economy Track, 01/99, IEEE Computer Society Press, CA, available at: www.businessmedia.org/netacademy/publications.nsf/ all_pk/1142 Van der Merwe, R. (2001), ``E-commerce Web site evaluation'', MSc Master's thesis, Department of Industrial Engineering, University of Stellenbosch, South Africa. Weller, S.C. and Romney, A.K. (1990), Metric Scaling: Correspondence Analysis, Sage University Paper Series on Quantitative Applications in the Social Sciences, Series nr. 07-075, Sage Publications, Newbury Park, CA.
341
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
This article has been cited by: 1. Gajendra Sharma, Wang Lijuan. 2015. The effects of online service quality of e-commerce Websites on user satisfaction. The Electronic Library 33:3, 468-485. [Abstract] [Full Text] [PDF] 2. Ping Sun, David A. Cárdenas, Rich Harrill. 2015. Chinese customers’ evaluation of travel agency website quality: A decision tree analysis. Journal of Hospitality Marketing & Management 150416144358002. [CrossRef] 3. Devashish Das Gupta, Utkarsh. 2014. Assessing the website effectiveness of top ten tourist attracting nations. Information Technology & Tourism 14, 151-175. [CrossRef] 4. Gaylen N. Chandler, J. Christian Broberg, Thomas H. Allison. 2014. Customer Value Propositions in Declining Industries: Differences between Industry Representative and High-Growth Firms. Strategic Entrepreneurship Journal n/a-n/a. [CrossRef] 5. Corrado lo Storto. 2013. Evaluating ecommerce websites cognitive efficiency: An integrative framework based on data envelopment analysis. Applied Ergonomics 44, 1004-1014. [CrossRef] 6. Layla Hasan, Anne Morris, Steve Probets. 2013. E‐commerce websites for developing countries – a usability evaluation framework. Online Information Review 37:2, 231-251. [Abstract] [Full Text] [PDF] 7. Abida Ellahi, Rahat H. Bokhari. 2013. Key quality factors affecting users' perception of social networking websites. Journal of Retailing and Consumer Services 20, 120-129. [CrossRef] 8. Layla Hasan, Anne Morris, Steve Probets. 2012. A comparison of usability evaluation methods for evaluating e-commerce websites. Behaviour & Information Technology 31, 707-737. [CrossRef] 9. Vasiliki Vrana, Kostas Zafiropoulos. 2011. Associations Between USPs and Design Characteristics of Mediterranean Countries' Websites. Journal of Hospitality Marketing & Management 20, 766-790. [CrossRef] 10. José Carlos Pinho, Maria de Lurdes Martins, Isabel Macedo. 2011. The effect of online service quality factors on internet usage. International Journal of Quality & Reliability Management 28:7, 706-722. [Abstract] [Full Text] [PDF] 11. Chad Lin, Yu‐An Huang, Rosemary Stockdale. 2011. Developing a B2B web site effectiveness model for SMEs. Internet Research 21:3, 304-325. [Abstract] [Full Text] [PDF] 12. Wen-Hsien Tsai, Wen-Chin Chou, Chien-Wen Lai. 2010. An effective evaluation model and improvement analysis for national park websites: A case study of Taiwan. Tourism Management 31, 936-952. [CrossRef] 13. Wen-Chih Chiou, Chin-Chao Lin, Chyuan Perng. 2010. A strategic framework for website evaluation based on a review of the literature from 1995–2006. Information & Management 47, 282-290. [CrossRef] 14. Hyung Jun Ahn. 2010. Evaluating customer aid functions of online stores with agent-based models of customer behavior and evolution strategy. Information Sciences 180, 1555-1570. [CrossRef] 15. Gülçin Büyüközkan, Jbid Arsenyan, Gürdal Ertek. 2010. Evaluation of E-Learning Web Sites Using Fuzzy Axiomatic Design Based Approach. International Journal of Computational Intelligence Systems 3, 28-42. [CrossRef] 16. R. Dale Wilson. 2010. Using clickstream data to enhance business‐to‐business web site performance. Journal of Business & Industrial Marketing 25:3, 177-187. [Abstract] [Full Text] [PDF] 17. Gulcin Yucel, Ahmet F. Ozok. 2010. Quantifying ergonomic quality of governmental websites. Electronic Government, an International Journal 7, 233. [CrossRef] 18. Jean-Claude Usunier, Nicolas Roulin, Björn Sven Ivens. 2009. Cultural, National, and Industry-Level Differences in B2B Web Site Design and Content. International Journal of Electronic Commerce 14, 41-88. [CrossRef] 19. Robert A. Opoku. 2009. Mapping Destination Personality in Cyberspace: An Evaluation of Country Web Sites Using Correspondence Analysis. Journal of Internet Commerce 8, 70-87. [CrossRef] 20. Francisco Javier Miranda, Ramón Sanguino, Tomás M. Bañegil. 2009. Quantitative assessment of European municipal web sites. Internet Research 19:4, 425-441. [Abstract] [Full Text] [PDF] 21. Fotini Patsioura, Maro Vlachopoulou, Vicky Manthou. 2009. A new advertising effectiveness model for corporate advertising web sites. Benchmarking: An International Journal 16:3, 372-386. [Abstract] [Full Text] [PDF] 22. Indranil Bose, Alvin Chung Man Leung. 2008. Assessing anti-phishing preparedness: A study of online banks in Hong Kong. Decision Support Systems 45, 897-912. [CrossRef] 23. Eleni Panopoulou, Efthimios Tambouris, Konstantinos Tarabanis. 2008. A framework for evaluating web sites of public authorities. Aslib Proceedings 60:5, 517-546. [Abstract] [Full Text] [PDF] 24. Stewart Adam, Michael D Featherstone. 2007. A comparison of web use in marketing by local government in the United States and Australia. Journal of Database Marketing & Customer Strategy Management 14, 297-310. [CrossRef]
Downloaded by Universiti Tunku Abdul Rahman At 01:01 16 June 2015 (PT)
25. GÜLÇIN BÜYÜKÖZKAN, DA RUAN. 2007. EVALUATING GOVERNMENT WEBSITES BASED ON A FUZZY MULTIPLE CRITERIA DECISION-MAKING APPROACH. International Journal of Uncertainty, Fuzziness and KnowledgeBased Systems 15, 321-343. [CrossRef] 26. Young Namkung, Seo-Young Shin, Il-Sun Yang. 2007. A Grounded Theory Approach to Understanding the Website Experiences of Restaurant Customers. Journal of Foodservice Business Research 10, 77-99. [CrossRef] 27. Gülçin Büyüközkan, Da Ruan, Orhan Feyzioğlu. 2007. Evaluating e-learning web site quality in a fuzzy environment. International Journal of Intelligent Systems 22:10.1002/int.v22:5, 567-586. [CrossRef] 28. Ray Welling, Lesley White. 2006. Web site performance measurement: promise and reality. Managing Service Quality: An International Journal 16:6, 654-670. [Abstract] [Full Text] [PDF] 29. R. Ufuk Bilsel, Gülçin Büyüközkan, Da Ruan. 2006. A fuzzy preference-ranking model for a quality evaluation of hospital web sites. International Journal of Intelligent Systems 21:10.1002/int.v21:11, 1181-1197. [CrossRef] 30. Ray Welling, Lesley White. 2006. Measuring the Value of Corporate Web Sites. Journal of Internet Commerce 5, 127-145. [CrossRef] 31. Talha Harcar, John E. Spillan. 2006. Exploring Latin American family decision-making using correspondence analysis. Journal of World Business 41, 221-232. [CrossRef] 32. COSTAS ZAFIROPOULOS, VASILIKI VRANA, DIMITRIOS PASCHALOUDIS. 2006. Patterns of Information Services on Hotel Websites: The Case of Greece. Anatolia 17, 55-74. [CrossRef] 33. Nikos Manouselis, Constantina Costopoulou. 2006. Quality in metadata: a schema for e‐commerce. Online Information Review 30:3, 217-237. [Abstract] [Full Text] [PDF] 34. Charles Oppenheim, Louise Ward. 2006. Evaluation of web sites for B2C e‐commerce. Aslib Proceedings 58:3, 237-260. [Abstract] [Full Text] [PDF] 35. Wayne Huang, Taowen Le, X. Li, S. Gandha. 2006. Categorizing web features and functions to evaluate commercial web sites. Industrial Management & Data Systems 106:4, 523-539. [Abstract] [Full Text] [PDF] 36. Costas Zafiropoulos, Vasiliki Vrana, Dimitrios Paschaloudis. 2006. The internet practices of hotel companies: an analysis from Greece. International Journal of Contemporary Hospitality Management 18:2, 156-163. [Abstract] [Full Text] [PDF] 37. Irene Pollach. 2005. Corporate self‐presentation on the WWW. Corporate Communications: An International Journal 10:4, 285-301. [Abstract] [Full Text] [PDF] 38. Jürgen Kai‐Uwe Brock, Yu (Josephine) Zhou. 2005. Organizational use of the internet. Internet Research 15:1, 67-87. [Abstract] [Full Text] [PDF] 39. Muneesh Kumar, Mamta SareenEvaluating Web Sites of Municipal Corporations 143-157. [CrossRef]
