Quality Assessment of the University Educational Process

Quality Assessment of the University Educational Process: an Application of the ECSI Model Bruno Chiandotto, Matilde Bini, Bruno Bertaccini1 Statistic...

0 downloads 19 Views 270KB Size
Quality Assessment of the University Educational Process: an Application of the ECSI Model

Bruno Chiandotto, Matilde Bini, Bruno Bertaccini1 Statistics Department “G. Parenti”, University of Florence, Italy

Summary. In a university, students represent the final users as well as the principal actors of the formative services. A measure of their perceived quality is essential for planning changes that would increase the level of the quality of these services. This perceived quality is analysed in this paper with the ECSI (European Customer Satisfaction Index) methodology. The ECSI, which implements a structural equation model, is aimed to represent the satisfaction of the students with some latent variables gauged through a set of observable indicators. We extend the ECSI to the data obtained from graduates of the University of Florence employed one year after graduation. Keywords: Customer satisfaction; ECSI; Structural equation models; University education.

1. Introduction We can say that a service is of good quality if it satisfies the consumers’ needs (Fornell, 1992; Fornell et al, 1996). The development of appropriate activities aimed at improving the quality represents one of the main strategies of the organisations willing to increase their productivity and competitiveness (Montgomery, 1997). Hence, studies on customer satisfaction must be accomplished with the purpose of gathering the appropriate information on purchase experience in order to improve the quality of the service. Customer satisfaction studies have become increasingly important due to the definition and use of new indices (CSI – Customer Satisfaction Indices) and barometers for the evaluation of large markets or that of the entire produc1

The three authors of this paper contributed the initial idea, the structure and the design of the work, whereas M. Bini and B. Bertaccini elaborated and implemented the model.

44

B. Chiandotto, M. Bini, B. Bertaccini

tion of a country. These new tools derive from two kinds of analyses: transaction-specific satisfaction and cumulative satisfaction (Johnson et al., 2000). The initial interest, focused on single episodes of consumption, shifted towards the overall psychological satisfaction after the use of a service. The assessment is based, therefore, on the upgraded experience of multiple transactions between the consumer and productive organization. The first model was the barometer proposed in Sweden in 1989 (SCSB – Swedish Customer Satisfaction Barometer), followed a few years later by the American index (ACSI – American Customer Satisfaction Index, 1994), then by the Norwegian barometer (NCSB – Norwegian Customer Satisfaction Barometer, 1996) and, finally, by the European Community index (ECSI – European Customer Satisfaction Index, see ECSI Technical Committee, 1998). These models, based on validated theories concerning the consumers’ behaviours, their satisfaction and the quality of the products purchased, consist of causal links among latent factors, each one representing the values of a specific set of measurable indicators. Their structure is under continuous review and is subject to modifications in relation to the context. The differences among the proposals are due both to the number of the latent factors and to the number of the causal nexuses involved in the analysis. Our work is aimed to extend the use of CSI indices to the university education framework. We chose the ECSI model since its basic structure and the relative latent factors are consistent with it. In order to draw a picture of the quality of the educational programmes realised by universities, we analysed data on graduates employed one year after graduation. For estimating the effects of ECSI we adopted the SEM - Structural Equation Models approach based on maximum likelihood estimation2. SEM procedures were preferred to PLS ones because the former - model specification is more flexible, - allow the significance testing of the omitted parameters (such as error covariances), loading of the latent variables, inclusion of ordinal and categorical variables, implementation of a two-level data structure, and handling of missing data, - can be implemented by means of specific software (Müthen & Müthen, 2003). We present the ASCI/ECSI models in Section 2, describe the data set used in our analysis in Section 3, and discuss the results in Section 4. Section 5 is devoted to some final remarks. 2

The technique initially suggested for estimating latent variables in CSI models (Fornell, 1992) was Partial Least Squares-PLS (Wold, 1975). We abandoned this technique because it assumes normality for the estimation of latent variables. Nevertheless, the PLS approach may optimally predict the dependent variable, both with small samples and skewed distributions.

Quality Assessment of the University Educational Process: an Application of the ECSI Model

45

2. Structure of the ACSI / ECSI model The structure of the CSI models includes some latent factors, each of which explains the variability of a set of, usually observable, indicators. The main feature of this approach is the connection between latent factors deriving from causal relationships between the sources and consequences of satisfaction. The ECSI model has evolved from the ACSI one. In the European model, the expectations of the consumer, the quality and value perceived, the satisfaction and loyalty concepts are very similar to the American one. The models differ in two fundamental aspects: the ECSI does not include complaints, but includes corporate image as a latent variable; this latter aspect may exert a direct effect upon the consumer’s expectation, satisfaction and loyalty. The causes of satisfaction foreseen in these models are (for the relationships between factors see Figure 1):  Perceived Quality (QUA): it refers to the assessment of recent consumer experiences concerning the characteristics of the service (perceived quality of hardware – QUAHW), the assistance supplied both during, and after the consumer’s experience (perceived quality of humanware – QUAUW). Both sub-dimensions are presumed to exert a direct, positive effect on the overall satisfaction;  Value (VALU): it represents the value of the quality perceived in relation to price. This factor may positively determine the overall satisfaction and be influenced by the perception of quality;

+

IMAG

+

+

EXPE

+

+

VALU

+ QUAHW

LOYA

+

+ +

+

-/+

SATI

-

QUAUW

Figure 1. ACSI / ECSI Model: structure and expected relationships.

COMP

46

B. Chiandotto, M. Bini, B. Bertaccini

Image (IMAG): it reflects the sensations generated by the product-brandmanufacturer association. This factor may positively influence value, overall satisfaction and loyalty, and produce a direct effect on perceived quality, nevertheless the ECSI model considers these two aspects exogenous;  Expectations (EXPE): it is the level of quality that the purchaser/user expects to receive and is the result of previous consumption/use experiences. It is considered an exogenous factor capable of positive influence on both value and overall satisfaction. The consequences of satisfaction are:  Complaints (COMP): it refers to the type and extent of the complaints and, above all, to the manner in which they are dealt with. In the ACSI model, an increase in the level of overall satisfaction is expected to decrease the incidence of complaints;  Loyalty (LOYA): it is a proxy variable of the power of profit of the organisation since it generates the intention to purchase again, tolerance of variation in price, intention to recommend the service to others. High levels of image and overall satisfaction may activate the loyalty of the consumer. We considered the European version of the customer satisfaction index as the most appropriate for representing the subject matter because adequate informative sources were available (see Section 3). Based on the information available and the preliminary analyses performed and, above all, on previous knowledge of the phenomenon, we defined the structural section of the reference model for the following analyses (see Figure 2). 

IMAG

+

+

EXPE

+

VALU

+ QUAHW

+

+

+

+

+

+

SATI

LOYA

+

+

QUAUW

Figure 2. ECSI model expectations for the assessment of the quality of university education.

Quality Assessment of the University Educational Process: an Application of the ECSI Model

47

3. The database Customer satisfaction surveys are based on questionnaires specifically devised for the purpose and aimed at covering all aspects that may be connected with the causes and consequences of the consumer/user experience. For assessing the quality of education issued by the University of Florence we created a database containing the data from the ALMALAUREA survey on graduates and that on ‘Professional opportunities and Florence University students graduated in the year 2000’ produced by the VALMON GROUP in collaboration with the Statistics Department ‘G. Parenti’ of Florence3. The latter data have been collected with a telephone interview lasting about 20 minutes and using a CATI - Computer Aided Telephone Interviewing system4. The cognitive requirements of the two basic surveys (Chiandotto et al., 2003, 2004) were different from those entailed in the assessment of customer satisfaction. However, the surveys allowed the definition of a complete picture of the quality of university process by students at the graduation stage as well as some time upon degree achievement. However, the enticing prospect of being able to combine information surveyed at the time of graduation with that obtained one year after it (which also referred to certain aspects of the graduates’ employment) actually implied the exclusion from the analysis of graduates who did not found an occupation within one year after graduation. The variables capable of assessing certain aspects of satisfaction are:  the reason for enrolling in university and choosing a particular course,  the quality of the relationships with professors, assistant teachers, nonteaching staff, fellow students,  the quality of university facilities (classrooms, laboratories, libraries, canteens),  satisfaction about occupation (consistency between job and studies, between job and own cultural interests, acquired skills) and overall satisfaction;  the hypothesis of entering university again and doing the same subjects;  the intention to continue with educational activities. Nearly all the variables are dichotomous or five-grade ordinal. Thanks to the number of eligible interviewees, we could delete the records with incomplete information. Thus, the selected cases totalled 1,753. 3

4

ALMALAUREA survey involves graduates of the summer session monitored three (or five) years after the end of their studies. The Florentine survey extended the observation of graduates to the years 1998, 1999, 2000 and 2001 one year after graduation. The Valmon Group, coordinated by B. Chiandotto, is composed of students and professors of the Department of Statistics, Florence University. A mail questionnaire was sent to graduates who were not possible to contact by telephone. The purpose of this survey was to detect possible differences with graduates contacted by telephone.

48

B. Chiandotto, M. Bini, B. Bertaccini

Table 1. Description of the measurement section of the theoretic ECSI model Latent structures

Observed variables

IMAG: Reason for attending University: MOTISCR1, MOTISCR2, MOTISCR3 EXPE: QUAHW:

Reason for choosing specific subjects: MOTCOR1, MOTCOR2, MOTCOR3, MOTCOR4, MOTCOR6 Evaluation of classrooms, libraries, laboratories and canteen (scale 15): STRAULE, STRBIB, STRLIB, STRMENSE

Evaluation of relationship with professors, assistants, non-teaching staff, QUAUW: fellow students, and overall judgement (scale 1-5): RAPDOC, RAPCOL, RAPNDOC, RAPSTUD, GIUDIZIO VALU: SATI: LOYA:

Satisfaction for coherence with studies and conformity with own cultural interests (scale 1-5): SODDCOER, SODDICUL Satisfaction for acquired skills, and overall judgement (scale 1-5): SODDPROF, SODDTO Hypothesis of re-attendance, intention to continue studies and educational activities: IPREISC, OKUNIV, OKCOR, INTSTUD, ATTFORM

In Table 1, it is presented the measurement section of the ECSI model of the relationships between the directly observed variables and latent structures.

4. The ECSI model for the quality of university education assessment As suggested by some Authors (see Bollen, 1989), a model for confirmatory factor analysis should be used at a first stage to validate a structural equation model5, that is, to evaluate the quality of the indicators employed as a gauge of 5

The structural equation models originate from the convergence of two scientific traditions, the ergonometric, which derives from the theory of a network of causal links between variables, and psychometric, which comes from the notion of latent variables. The structural equation model is usually referred to as the LISREL model (Jöreskog, 1973; Jöreskog, 1990; Jöreskog & Sörbom, 1993; Corbetta, 2002). This model assumes two components: a structural part, designed to explain the causal links between latent variables, and a measurement one for evaluating the variability on the observed variables. The model may be expressed as: y = η + Bξ + ζ (structural) and x = Λxξ + δ, y = Λyη + ε (measurement), where y and x are, respectively, endogenous and exogenous vectors of observed variables, η and ξ are vectors of underlying latent variables, Β, Γ, Λy and Λx are matrices of coefficients and ε and δ are the measurement error terms. In the SEM it is normally assumed that the data follow a multivariate normal distribution, so that the

Quality Assessment of the University Educational Process: an Application of the ECSI Model

49

latent components and eliminate alien sources of variability. Nevertheless, the use of this procedure would have been inappropriate for our purposes: the lack of a specific questionnaire would necessitate in fact the application of an exploratory factor analysis for unveiling the suitable number of latent structures and the items appropriate for measuring these structures. Once the measurement section of the model has been defined, the estimation procedure is optimised by applying a confirmatory factor analysis and estimating the effects.

4.1 Exploratory factor analysis The dual aim of the exploratory factor analysis is to detect the optimal number of latent factors and identify the variables that prove to be appropriate for measuring them. The analysis highlighted between five and eight factors and we chose seven factors. The factorial loadings are presented in Table 2. The variables uncorrelated with the factors are marked with asterisks. All the other variables are clearly associated with only one factor, with the exception of that regarding consistency between occupation and studies (SODDCOER). On this ground, we identified the measurement section of the model (Table 3).

4.2 Confirmatory factor analysis A typical configuration of confirmatory factorial analysis (CFA) is a model of structural equations that foresees all the possible links between latent structures. The purpose of CFA is to assess the quality of the measurement structures of the model identified in the previous step. By applying the WLSMV (Weighted Least Square Mean and Variance) estimation logarithm, recommended by the MPlus software, the model suggested by the explorative factorial analysis fails to converge. Nevertheless, the resulting estimates are the starting values for the following models. We performed another confirmatory factor analysis that accounted for the results of the exploratory analysis and the first implementation of the ECSI model (Table 4).

vector of means and the matrix of covariance contain the information required for estimation. Nevertheless, there are estimation procedures that use non-normal continuous data but require the entire data matrix. The statistical tests of fitting generally depend on sample size. Likewise, when the sample is limited in size, there is a risk of accepting the model even if the fitting is poor. Several indices have been proposed for measuring the proximity of the data to the model (Hox & Bechger, 1998).

50

B. Chiandotto, M. Bini, B. Bertaccini

Table 2. Factor loadings of 7-factor exploratory factor analysis

* * *

*

Table 3. Measurement section suggested by the exploratory factor analysis A

Quality of relationships with professors, assistants, non-teaching staff, fellow students

B Evaluation of classrooms, libraries, laboratories, canteens C Level of usage of skills acquired at university D Reason for attending university and choosing particular subjects E

Opinion concerning the hypothesis of attending university again and studying the same subjects

F Intention to continue studies and educational activities Consistency between occupation and studies, consistency between occupation and G own cultural interests, acquired skills, overall job satisfaction

Quality Assessment of the University Educational Process: an Application of the ECSI Model

51

Table 4. Measurement of confirmatory factor analysis according to the exploratory one IMAG: MOTISCR1 MOTISCR2*-0.68 MOTISCR3*0.264 EXPE: MOTCOR2 MOTCOR4 MOTCOR6 QUAHW: STRAULE STRBIB*1.06 STRLAB*0.817 SERMENSE*0.568 QUAUW: RAPDOC RAPCOL*0.98 RAPNDOC*0.815 RAPSTUD*0.554 VALU: SODDCOER SODDICUL*0.884 SODDPROF*0.876 SODDTOT*.755 SATI: OKUNIV OKCOR*1.006 IPREISC*.913 LOYA: INT_STUD ATTFORM*1.163

The adaptation indices denote a model that is capable of describing the subject matter in a more satisfactory manner (TFI = 0.984 e RMSEA = 0.028). This makes it possible to rely on the measurement section detected and concentrate solely on the structural section, estimating the causal links between the latent components to verify the strength of the expected relationships (Figure 2).

4.3 Structural equation models The CFA model was re-configured into a complete ECSI-SEM model, where the regression equations between the latent components are specified. The term ‘complete’ describes the particular configuration assumed by the causal links between the latent components, because of its typical temporal and sequential features. That is to say, in the complete model each factor on the left side of the graph is assumed capable of exerting both a direct and an indirect influence on factors on its right. For instance, all the theoretic causes of consumption experience (IMAG, EXPE, QUAHW, QUAUW) are assumed as explanatory factors of both the actual components of the satisfaction (VALU e SATI) and the loyalty (LOYA). Moreover, since it is plausible to assume that the consumption experience can activate a cognitive process whereby a value is attributed to a service purchased before the arousal of actual feelings of satisfaction, it is logical to consider VALU as a possible cause of SATI and LOYA. The implementation of a complete model required new links within the model originally proposed (see Figure 3 – added links in bold print). Due to the extreme complexity of the structural section, the complete ECSI-SEM model did not converge. The estimates produced by the software used have nonetheless been useful guidelines for improving the model since they suggest which causal links had to be eliminated.

52

B. Chiandotto, M. Bini, B. Bertaccini

IMAG

EXPE

VALU

SATI

LOYA

QUAHW

QUAUW

Figure 3. Complete ECSI-SEM model: structural section highlighting the added links

IMAG

EXPE

VALU

SATI

LOYA

QUAHW

QUAUW

Figure 4. Structure of the first ECSI-SEM convergence model

Simplifications applied to the structural section for subsequent steps have generated the model shown in Figure 4. Even in this case, the normal adaptation indices indicate that the model effectively fits the data (TFI = 0.982; RMSEA = 0.029). Being aware that this procedure is tricky and open to criticism, but wishing to define a more satisfactory model from an interpretative point of view, the non-significant links have been gradually eliminated. Just the essential results obtained during this step will be discussed here. It is sufficient to notice that the adaptation indices always emphasise sensible modifications tending towards the value that represents perfect adaptation.

Quality Assessment of the University Educational Process: an Application of the ECSI Model

53

IMAG

EXPE

0.26

VALU QUAHW

0.53

SATI

0.15

LOYA

0.20 0.09

QUAUW

Figure 5. Structure of the final ECSI-SEM mode.

Figure 5 shows the final ECSI-SEM model for which no improvement was appreciable. The values of the estimated coefficients are shown on the arrows in the graph that represent the causal links differing from zero: note that all the effects are oriented in the expected direction. This graph does not show the correlation relationships that nevertheless existed between the latent components. Once again, the good adaptation of the model is substantiated by the values given by the TLI (0.985) and RMSEA (0.027) indices.

5. Conclusions The interest on quality assessment of educational activities in Italian universities is based on law (Laws 168/89 and 537/93). It foresees internal controls on the efficiency of processes and effectiveness of results of university management. Especially in recent years, numerous projects for the assessment of university education in terms of efficiency and effectiveness have been proposed and realised. The success achieved by university graduates in the labour market may be considered the mirror image of the quality of educational processes. This success may be measured in terms of either objective indicators, like salary and career, or subjective ones, such as satisfaction for the occupation. Our work evaluated the ECSI-SEM models as a tool for the analysis of university education quality. The results of our analysis not only confirm the validity of the application of ECSI-SEM models in this subject matter but also stimulate interest towards the implementation of more detailed analyses, which, with the collection of appropriate data, could help the development of theoretical and methodological aspects.

54

B. Chiandotto, M. Bini, B. Bertaccini

A suitable questionnaire could be studied to collect opinions on satisfaction. As far as the methodological aspects are concerned, an evaluation of the effects caused by the presence of abnormal observations could be performed by applying the forward search algorithm and implementing new models for groups of graduates who attended the same programme, in order to test the existence of specific causal links of satisfaction.

References BOLLEN K.A. (1989) Structural Equation with Latent Variables, Wiley, New York. CHIANDOTTO B., BERTACCINI B. (2003) Profilo e sbocchi occupazionali dei laureati e diplomati dell’Ateneo fiorentino nell’anno 1999, Gruppo VALMON - Università degli Studi di Firenze. CHIANDOTTO B., BACCI S., BERTACCINI B. (2004) Profilo e sbocchi occupazionali dei laureati e diplomati dell’Ateneo fiorentino nell’anno 2000, Gruppo VALMON - Università degli Studi di Firenze. CORBETTA P. (2002) Metodi di analisi multivariata per le scienze sociali. I Modelli di Equazioni Strutturali, Il Mulino, Bologna. ECSI Technical Committee (1998) European Customer Satisfaction Index: Foundation and Structure for Harmonized National Pilot Projects. Report prepared for the ECSI Steering Committee, October. FORNELL C. (1992) A National Customer Satisfaction Barometer, the Swedish Experience, Journal of Marketing, 56: 6-21. FORNELL C., JOHNSON M.D., ANDERSON E.W., CHA J., BRYANT B.E. (1996) The American Customer Satisfaction Index, Nature, Purpose and Findings, Journal of Marketing, 60: 7-18. HOX J.J., BECHGER T.M. (1998) An Introduction to Structural Equation Modeling, Family Science Review, 11: 354-373. JOHNSON M.D., GUSTAFSSON A., ANDREASSEN T.W., LERVIK L., CHA J. (2001) The Evolution and Future of National Customer Satisfaction Index Models, Journal of Economic Psychology, 22: 217-245. JÖRESKOG K.G. (1973) A General Method for Estimating a Linear Structural Equation System. In: GOLDBERGER A.S., DUNCAN O.D. (eds) Structural Equation Models in the Social Sciences, New York, Academic Press: 85-112. JÖRESKOG K.G. (1990) New Developments in LISREL. Analysis of Ordinal Variables Using Polychoric Correlations and Weighted Least Squares, Quality and Quantity, 24: 387-404. JÖRESKOG K.G., SÖRBOM D. (1993) New Features in PRELIS2, Scientific Software International Chicago, IL. MONTGOMERY D. (1997) Introduction to Statistical Quality Control, Wiley, New York. MÜTHEN B.O., MÜTHEN L.K. (2003) MPlus 3.0 User Guide (http://www.statmodel.com/verhistory.shtml). WOLD H. (1975) Path Models with Latent Variables, the NIPALS Approach. In: BLALOCK H.M., AGANBEGIAN A., BORODKIN F.M., BOUDON R., CAPPECCHI V. (eds) Quantitative Sociology. International Perspectives on Mathematical and Statistical Modeling, Academic Press, New York: 307-353.