Quality Assessment of the University Educational Process: an Application of the ECSI Model
Bruno Chiandotto, Matilde Bini, Bruno Bertaccini1 Statistics Department “G. Parenti”, University of Florence, Italy
Summary. In a university, students represent the final users as well as the principal actors of the formative services. A measure of their perceived quality is essential for planning changes that would increase the level of the quality of these services. This perceived quality is analysed in this paper with the ECSI (European Customer Satisfaction Index) methodology. The ECSI, which implements a structural equation model, is aimed to represent the satisfaction of the students with some latent variables gauged through a set of observable indicators. We extend the ECSI to the data obtained from graduates of the University of Florence employed one year after graduation. Keywords: Customer satisfaction; ECSI; Structural equation models; University education.
1. Introduction We can say that a service is of good quality if it satisfies the consumers’ needs (Fornell, 1992; Fornell et al, 1996). The development of appropriate activities aimed at improving the quality represents one of the main strategies of the organisations willing to increase their productivity and competitiveness (Montgomery, 1997). Hence, studies on customer satisfaction must be accomplished with the purpose of gathering the appropriate information on purchase experience in order to improve the quality of the service. Customer satisfaction studies have become increasingly important due to the definition and use of new indices (CSI – Customer Satisfaction Indices) and barometers for the evaluation of large markets or that of the entire produc1
The three authors of this paper contributed the initial idea, the structure and the design of the work, whereas M. Bini and B. Bertaccini elaborated and implemented the model.
44
B. Chiandotto, M. Bini, B. Bertaccini
tion of a country. These new tools derive from two kinds of analyses: transactionspecific satisfaction and cumulative satisfaction (Johnson et al., 2000). The initial interest, focused on single episodes of consumption, shifted towards the overall psychological satisfaction after the use of a service. The assessment is based, therefore, on the upgraded experience of multiple transactions between the consumer and productive organization. The first model was the barometer proposed in Sweden in 1989 (SCSB – Swedish Customer Satisfaction Barometer), followed a few years later by the American index (ACSI – American Customer Satisfaction Index, 1994), then by the Norwegian barometer (NCSB – Norwegian Customer Satisfaction Barometer, 1996) and, finally, by the European Community index (ECSI – European Customer Satisfaction Index, see ECSI Technical Committee, 1998). These models, based on validated theories concerning the consumers’ behaviours, their satisfaction and the quality of the products purchased, consist of causal links among latent factors, each one representing the values of a specific set of measurable indicators. Their structure is under continuous review and is subject to modifications in relation to the context. The differences among the proposals are due both to the number of the latent factors and to the number of the causal nexuses involved in the analysis. Our work is aimed to extend the use of CSI indices to the university education framework. We chose the ECSI model since its basic structure and the relative latent factors are consistent with it. In order to draw a picture of the quality of the educational programmes realised by universities, we analysed data on graduates employed one year after graduation. For estimating the effects of ECSI we adopted the SEM  Structural Equation Models approach based on maximum likelihood estimation2. SEM procedures were preferred to PLS ones because the former  model specification is more flexible,  allow the significance testing of the omitted parameters (such as error covariances), loading of the latent variables, inclusion of ordinal and categorical variables, implementation of a twolevel data structure, and handling of missing data,  can be implemented by means of specific software (Müthen & Müthen, 2003). We present the ASCI/ECSI models in Section 2, describe the data set used in our analysis in Section 3, and discuss the results in Section 4. Section 5 is devoted to some final remarks. 2
The technique initially suggested for estimating latent variables in CSI models (Fornell, 1992) was Partial Least SquaresPLS (Wold, 1975). We abandoned this technique because it assumes normality for the estimation of latent variables. Nevertheless, the PLS approach may optimally predict the dependent variable, both with small samples and skewed distributions.
Quality Assessment of the University Educational Process: an Application of the ECSI Model
45
2. Structure of the ACSI / ECSI model The structure of the CSI models includes some latent factors, each of which explains the variability of a set of, usually observable, indicators. The main feature of this approach is the connection between latent factors deriving from causal relationships between the sources and consequences of satisfaction. The ECSI model has evolved from the ACSI one. In the European model, the expectations of the consumer, the quality and value perceived, the satisfaction and loyalty concepts are very similar to the American one. The models differ in two fundamental aspects: the ECSI does not include complaints, but includes corporate image as a latent variable; this latter aspect may exert a direct effect upon the consumer’s expectation, satisfaction and loyalty. The causes of satisfaction foreseen in these models are (for the relationships between factors see Figure 1): Perceived Quality (QUA): it refers to the assessment of recent consumer experiences concerning the characteristics of the service (perceived quality of hardware – QUAHW), the assistance supplied both during, and after the consumer’s experience (perceived quality of humanware – QUAUW). Both subdimensions are presumed to exert a direct, positive effect on the overall satisfaction; Value (VALU): it represents the value of the quality perceived in relation to price. This factor may positively determine the overall satisfaction and be influenced by the perception of quality;
+
IMAG
+
+
EXPE
+
+
VALU
+ QUAHW
LOYA
+
+ +
+
/+
SATI

QUAUW
Figure 1. ACSI / ECSI Model: structure and expected relationships.
COMP
46
B. Chiandotto, M. Bini, B. Bertaccini
Image (IMAG): it reflects the sensations generated by the productbrandmanufacturer association. This factor may positively influence value, overall satisfaction and loyalty, and produce a direct effect on perceived quality, nevertheless the ECSI model considers these two aspects exogenous; Expectations (EXPE): it is the level of quality that the purchaser/user expects to receive and is the result of previous consumption/use experiences. It is considered an exogenous factor capable of positive influence on both value and overall satisfaction. The consequences of satisfaction are: Complaints (COMP): it refers to the type and extent of the complaints and, above all, to the manner in which they are dealt with. In the ACSI model, an increase in the level of overall satisfaction is expected to decrease the incidence of complaints; Loyalty (LOYA): it is a proxy variable of the power of profit of the organisation since it generates the intention to purchase again, tolerance of variation in price, intention to recommend the service to others. High levels of image and overall satisfaction may activate the loyalty of the consumer. We considered the European version of the customer satisfaction index as the most appropriate for representing the subject matter because adequate informative sources were available (see Section 3). Based on the information available and the preliminary analyses performed and, above all, on previous knowledge of the phenomenon, we defined the structural section of the reference model for the following analyses (see Figure 2).
IMAG
+
+
EXPE
+
VALU
+ QUAHW
+
+
+
+
+
+
SATI
LOYA
+
+
QUAUW
Figure 2. ECSI model expectations for the assessment of the quality of university education.
Quality Assessment of the University Educational Process: an Application of the ECSI Model
47
3. The database Customer satisfaction surveys are based on questionnaires specifically devised for the purpose and aimed at covering all aspects that may be connected with the causes and consequences of the consumer/user experience. For assessing the quality of education issued by the University of Florence we created a database containing the data from the ALMALAUREA survey on graduates and that on ‘Professional opportunities and Florence University students graduated in the year 2000’ produced by the VALMON GROUP in collaboration with the Statistics Department ‘G. Parenti’ of Florence3. The latter data have been collected with a telephone interview lasting about 20 minutes and using a CATI  Computer Aided Telephone Interviewing system4. The cognitive requirements of the two basic surveys (Chiandotto et al., 2003, 2004) were different from those entailed in the assessment of customer satisfaction. However, the surveys allowed the definition of a complete picture of the quality of university process by students at the graduation stage as well as some time upon degree achievement. However, the enticing prospect of being able to combine information surveyed at the time of graduation with that obtained one year after it (which also referred to certain aspects of the graduates’ employment) actually implied the exclusion from the analysis of graduates who did not found an occupation within one year after graduation. The variables capable of assessing certain aspects of satisfaction are: the reason for enrolling in university and choosing a particular course, the quality of the relationships with professors, assistant teachers, nonteaching staff, fellow students, the quality of university facilities (classrooms, laboratories, libraries, canteens), satisfaction about occupation (consistency between job and studies, between job and own cultural interests, acquired skills) and overall satisfaction; the hypothesis of entering university again and doing the same subjects; the intention to continue with educational activities. Nearly all the variables are dichotomous or fivegrade ordinal. Thanks to the number of eligible interviewees, we could delete the records with incomplete information. Thus, the selected cases totalled 1,753. 3
4
ALMALAUREA survey involves graduates of the summer session monitored three (or five) years after the end of their studies. The Florentine survey extended the observation of graduates to the years 1998, 1999, 2000 and 2001 one year after graduation. The Valmon Group, coordinated by B. Chiandotto, is composed of students and professors of the Department of Statistics, Florence University. A mail questionnaire was sent to graduates who were not possible to contact by telephone. The purpose of this survey was to detect possible differences with graduates contacted by telephone.
48
B. Chiandotto, M. Bini, B. Bertaccini
Table 1. Description of the measurement section of the theoretic ECSI model Latent structures
Observed variables
IMAG: Reason for attending University: MOTISCR1, MOTISCR2, MOTISCR3 EXPE: QUAHW:
Reason for choosing specific subjects: MOTCOR1, MOTCOR2, MOTCOR3, MOTCOR4, MOTCOR6 Evaluation of classrooms, libraries, laboratories and canteen (scale 15): STRAULE, STRBIB, STRLIB, STRMENSE
Evaluation of relationship with professors, assistants, nonteaching staff, QUAUW: fellow students, and overall judgement (scale 15): RAPDOC, RAPCOL, RAPNDOC, RAPSTUD, GIUDIZIO VALU: SATI: LOYA:
Satisfaction for coherence with studies and conformity with own cultural interests (scale 15): SODDCOER, SODDICUL Satisfaction for acquired skills, and overall judgement (scale 15): SODDPROF, SODDTO Hypothesis of reattendance, intention to continue studies and educational activities: IPREISC, OKUNIV, OKCOR, INTSTUD, ATTFORM
In Table 1, it is presented the measurement section of the ECSI model of the relationships between the directly observed variables and latent structures.
4. The ECSI model for the quality of university education assessment As suggested by some Authors (see Bollen, 1989), a model for confirmatory factor analysis should be used at a first stage to validate a structural equation model5, that is, to evaluate the quality of the indicators employed as a gauge of 5
The structural equation models originate from the convergence of two scientific traditions, the ergonometric, which derives from the theory of a network of causal links between variables, and psychometric, which comes from the notion of latent variables. The structural equation model is usually referred to as the LISREL model (Jöreskog, 1973; Jöreskog, 1990; Jöreskog & Sörbom, 1993; Corbetta, 2002). This model assumes two components: a structural part, designed to explain the causal links between latent variables, and a measurement one for evaluating the variability on the observed variables. The model may be expressed as: y = η + Bξ + ζ (structural) and x = Λxξ + δ, y = Λyη + ε (measurement), where y and x are, respectively, endogenous and exogenous vectors of observed variables, η and ξ are vectors of underlying latent variables, Β, Γ, Λy and Λx are matrices of coefficients and ε and δ are the measurement error terms. In the SEM it is normally assumed that the data follow a multivariate normal distribution, so that the
Quality Assessment of the University Educational Process: an Application of the ECSI Model
49
latent components and eliminate alien sources of variability. Nevertheless, the use of this procedure would have been inappropriate for our purposes: the lack of a specific questionnaire would necessitate in fact the application of an exploratory factor analysis for unveiling the suitable number of latent structures and the items appropriate for measuring these structures. Once the measurement section of the model has been defined, the estimation procedure is optimised by applying a confirmatory factor analysis and estimating the effects.
4.1 Exploratory factor analysis The dual aim of the exploratory factor analysis is to detect the optimal number of latent factors and identify the variables that prove to be appropriate for measuring them. The analysis highlighted between five and eight factors and we chose seven factors. The factorial loadings are presented in Table 2. The variables uncorrelated with the factors are marked with asterisks. All the other variables are clearly associated with only one factor, with the exception of that regarding consistency between occupation and studies (SODDCOER). On this ground, we identified the measurement section of the model (Table 3).
4.2 Confirmatory factor analysis A typical configuration of confirmatory factorial analysis (CFA) is a model of structural equations that foresees all the possible links between latent structures. The purpose of CFA is to assess the quality of the measurement structures of the model identified in the previous step. By applying the WLSMV (Weighted Least Square Mean and Variance) estimation logarithm, recommended by the MPlus software, the model suggested by the explorative factorial analysis fails to converge. Nevertheless, the resulting estimates are the starting values for the following models. We performed another confirmatory factor analysis that accounted for the results of the exploratory analysis and the first implementation of the ECSI model (Table 4).
vector of means and the matrix of covariance contain the information required for estimation. Nevertheless, there are estimation procedures that use nonnormal continuous data but require the entire data matrix. The statistical tests of fitting generally depend on sample size. Likewise, when the sample is limited in size, there is a risk of accepting the model even if the fitting is poor. Several indices have been proposed for measuring the proximity of the data to the model (Hox & Bechger, 1998).
50
B. Chiandotto, M. Bini, B. Bertaccini
Table 2. Factor loadings of 7factor exploratory factor analysis
* * *
*
Table 3. Measurement section suggested by the exploratory factor analysis A
Quality of relationships with professors, assistants, nonteaching staff, fellow students
B Evaluation of classrooms, libraries, laboratories, canteens C Level of usage of skills acquired at university D Reason for attending university and choosing particular subjects E
Opinion concerning the hypothesis of attending university again and studying the same subjects
F Intention to continue studies and educational activities Consistency between occupation and studies, consistency between occupation and G own cultural interests, acquired skills, overall job satisfaction
Quality Assessment of the University Educational Process: an Application of the ECSI Model
51
Table 4. Measurement of confirmatory factor analysis according to the exploratory one IMAG: MOTISCR1 MOTISCR2*0.68 MOTISCR3*0.264 EXPE: MOTCOR2 MOTCOR4 MOTCOR6 QUAHW: STRAULE STRBIB*1.06 STRLAB*0.817 SERMENSE*0.568 QUAUW: RAPDOC RAPCOL*0.98 RAPNDOC*0.815 RAPSTUD*0.554 VALU: SODDCOER SODDICUL*0.884 SODDPROF*0.876 SODDTOT*.755 SATI: OKUNIV OKCOR*1.006 IPREISC*.913 LOYA: INT_STUD ATTFORM*1.163
The adaptation indices denote a model that is capable of describing the subject matter in a more satisfactory manner (TFI = 0.984 e RMSEA = 0.028). This makes it possible to rely on the measurement section detected and concentrate solely on the structural section, estimating the causal links between the latent components to verify the strength of the expected relationships (Figure 2).
4.3 Structural equation models The CFA model was reconfigured into a complete ECSISEM model, where the regression equations between the latent components are specified. The term ‘complete’ describes the particular configuration assumed by the causal links between the latent components, because of its typical temporal and sequential features. That is to say, in the complete model each factor on the left side of the graph is assumed capable of exerting both a direct and an indirect influence on factors on its right. For instance, all the theoretic causes of consumption experience (IMAG, EXPE, QUAHW, QUAUW) are assumed as explanatory factors of both the actual components of the satisfaction (VALU e SATI) and the loyalty (LOYA). Moreover, since it is plausible to assume that the consumption experience can activate a cognitive process whereby a value is attributed to a service purchased before the arousal of actual feelings of satisfaction, it is logical to consider VALU as a possible cause of SATI and LOYA. The implementation of a complete model required new links within the model originally proposed (see Figure 3 – added links in bold print). Due to the extreme complexity of the structural section, the complete ECSISEM model did not converge. The estimates produced by the software used have nonetheless been useful guidelines for improving the model since they suggest which causal links had to be eliminated.
52
B. Chiandotto, M. Bini, B. Bertaccini
IMAG
EXPE
VALU
SATI
LOYA
QUAHW
QUAUW
Figure 3. Complete ECSISEM model: structural section highlighting the added links
IMAG
EXPE
VALU
SATI
LOYA
QUAHW
QUAUW
Figure 4. Structure of the first ECSISEM convergence model
Simplifications applied to the structural section for subsequent steps have generated the model shown in Figure 4. Even in this case, the normal adaptation indices indicate that the model effectively fits the data (TFI = 0.982; RMSEA = 0.029). Being aware that this procedure is tricky and open to criticism, but wishing to define a more satisfactory model from an interpretative point of view, the nonsignificant links have been gradually eliminated. Just the essential results obtained during this step will be discussed here. It is sufficient to notice that the adaptation indices always emphasise sensible modifications tending towards the value that represents perfect adaptation.
Quality Assessment of the University Educational Process: an Application of the ECSI Model
53
IMAG
EXPE
0.26
VALU QUAHW
0.53
SATI
0.15
LOYA
0.20 0.09
QUAUW
Figure 5. Structure of the final ECSISEM mode.
Figure 5 shows the final ECSISEM model for which no improvement was appreciable. The values of the estimated coefficients are shown on the arrows in the graph that represent the causal links differing from zero: note that all the effects are oriented in the expected direction. This graph does not show the correlation relationships that nevertheless existed between the latent components. Once again, the good adaptation of the model is substantiated by the values given by the TLI (0.985) and RMSEA (0.027) indices.
5. Conclusions The interest on quality assessment of educational activities in Italian universities is based on law (Laws 168/89 and 537/93). It foresees internal controls on the efficiency of processes and effectiveness of results of university management. Especially in recent years, numerous projects for the assessment of university education in terms of efficiency and effectiveness have been proposed and realised. The success achieved by university graduates in the labour market may be considered the mirror image of the quality of educational processes. This success may be measured in terms of either objective indicators, like salary and career, or subjective ones, such as satisfaction for the occupation. Our work evaluated the ECSISEM models as a tool for the analysis of university education quality. The results of our analysis not only confirm the validity of the application of ECSISEM models in this subject matter but also stimulate interest towards the implementation of more detailed analyses, which, with the collection of appropriate data, could help the development of theoretical and methodological aspects.
54
B. Chiandotto, M. Bini, B. Bertaccini
A suitable questionnaire could be studied to collect opinions on satisfaction. As far as the methodological aspects are concerned, an evaluation of the effects caused by the presence of abnormal observations could be performed by applying the forward search algorithm and implementing new models for groups of graduates who attended the same programme, in order to test the existence of specific causal links of satisfaction.
References BOLLEN K.A. (1989) Structural Equation with Latent Variables, Wiley, New York. CHIANDOTTO B., BERTACCINI B. (2003) Profilo e sbocchi occupazionali dei laureati e diplomati dell’Ateneo fiorentino nell’anno 1999, Gruppo VALMON  Università degli Studi di Firenze. CHIANDOTTO B., BACCI S., BERTACCINI B. (2004) Profilo e sbocchi occupazionali dei laureati e diplomati dell’Ateneo fiorentino nell’anno 2000, Gruppo VALMON  Università degli Studi di Firenze. CORBETTA P. (2002) Metodi di analisi multivariata per le scienze sociali. I Modelli di Equazioni Strutturali, Il Mulino, Bologna. ECSI Technical Committee (1998) European Customer Satisfaction Index: Foundation and Structure for Harmonized National Pilot Projects. Report prepared for the ECSI Steering Committee, October. FORNELL C. (1992) A National Customer Satisfaction Barometer, the Swedish Experience, Journal of Marketing, 56: 621. FORNELL C., JOHNSON M.D., ANDERSON E.W., CHA J., BRYANT B.E. (1996) The American Customer Satisfaction Index, Nature, Purpose and Findings, Journal of Marketing, 60: 718. HOX J.J., BECHGER T.M. (1998) An Introduction to Structural Equation Modeling, Family Science Review, 11: 354373. JOHNSON M.D., GUSTAFSSON A., ANDREASSEN T.W., LERVIK L., CHA J. (2001) The Evolution and Future of National Customer Satisfaction Index Models, Journal of Economic Psychology, 22: 217245. JÖRESKOG K.G. (1973) A General Method for Estimating a Linear Structural Equation System. In: GOLDBERGER A.S., DUNCAN O.D. (eds) Structural Equation Models in the Social Sciences, New York, Academic Press: 85112. JÖRESKOG K.G. (1990) New Developments in LISREL. Analysis of Ordinal Variables Using Polychoric Correlations and Weighted Least Squares, Quality and Quantity, 24: 387404. JÖRESKOG K.G., SÖRBOM D. (1993) New Features in PRELIS2, Scientific Software International Chicago, IL. MONTGOMERY D. (1997) Introduction to Statistical Quality Control, Wiley, New York. MÜTHEN B.O., MÜTHEN L.K. (2003) MPlus 3.0 User Guide (http://www.statmodel.com/verhistory.shtml). WOLD H. (1975) Path Models with Latent Variables, the NIPALS Approach. In: BLALOCK H.M., AGANBEGIAN A., BORODKIN F.M., BOUDON R., CAPPECCHI V. (eds) Quantitative Sociology. International Perspectives on Mathematical and Statistical Modeling, Academic Press, New York: 307353.