Assessment of Fit

The CALIS Procedure

Assessment of Fit

This section contains a collection of formulas used in computing indices to assess the goodness of fit by PROC CALIS. The following notation is used:

N for the sample size
n for the number of manifest variables
t for the number of parameters to estimate
$NM = \{ (N - 1) & {if the CORR or COV matrix is analyzed} \ & {or the intercept... ...trix is analyzed} \ & {and the intercept variable is not used in the model} \ .$
df for the degrees of freedom
${\gamma}= X$ for the t vector of optimal parameter estimates
S = (s_ij) for the n ×n input COV, CORR, UCOV, or UCORR matrix
$C=(c_{ij}) = \hat{{{\Sigma}}} = {{\Sigma}}(\hat{{\gamma}})$ for the predicted model matrix
W for the weight matrix (W = I for ULS, W = S for default GLS, and W = C for ML estimates)
U for the n² ×n² asymptotic covariance matrix of sample covariances
$\Phi(x|\lambda,df)$ for the cumulative distribution function of the noncentral chi-squared distribution with noncentrality parameter $\lambda$

The following notation is for indices that allow testing nested models by a $\chi^2$ difference test:

f₀ for the function value of the independence model
df₀ for the degrees of freedom of the independence model
f_min = F for the function value of the fitted model
df_min = df for the degrees of freedom of the fitted model

The degrees of freedom df_min and the number of parameters t are adjusted automatically when there are active constraints in the analysis. The computation of many fit statistics and indices are affected. You can turn off the automatic adjustment using the NOADJDF option. See the section "Counting the Degrees of Freedom" for more information.

Residuals

PROC CALIS computes four types of residuals and writes them to the OUTSTAT= data set:

Raw Residuals
Res = S- C, Res_ij = s_ij - c_ij
The raw residuals are displayed whenever the PALL, the PRINT, or the RESIDUAL option is specified.
Variance Standardized Residuals
$VSRes_{ij} = {s_{ij} - c_{ij} \over \sqrt{s_{ii} s_{jj}}}$
The variance standardized residuals are displayed when you specify
- the PALL, the PRINT, or the RESIDUAL option and METHOD=NONE, METHOD=ULS, or METHOD=DWLS
- RESIDUAL=VARSTAND
The variance standardized residuals are equal to those computed by the EQS 3 program (Bentler 1989).
Asymptotically Standardized Residuals

$ASRes_{ij} = {s_{ij} - c_{ij} \over \sqrt{c_{ij,ij}}} , {where}$

$c_{ij,ij} = diag(U- JCov({\gamma}) J^')_{ij}$
The matrix J is the n² ×t Jacobian matrix , and is the t ×t asymptotic covariance matrix of parameter estimates (the inverse of the information matrix). Asymptotically standardized residuals are displayed when one of the following conditions is met:
- The PALL, the PRINT, or the RESIDUAL option is specified, and METHOD=ML, METHOD=GLS, or METHOD=WLS, and the expensive information and Jacobian matrices are computed for some other reason.
- RESIDUAL= ASYSTAND is specified.
The asymptotically standardized residuals are equal to those computed by the LISREL 7 program (Jreskog and Srbom 1988) except for the denominator NM in the definition of matrix U.
Normalized Residuals

$NRes_{ij} = {s_{ij} - c_{ij} \over \sqrt{u_{ij,ij}}}$
where the diagonal elements u_ij,ij of the n² ×n² asymptotic covariance matrix U of sample covariances are defined for the following methods.
- GLS as $u_{ij,ij} = {1 \over NM} (s_{ii} s_{jj} + s^2_{ij})$
- ML as $u_{ij,ij} = {1 \over NM} (c_{ii} c_{jj} + c^2_{ij})$
- WLS as u_ij,ij = w_ij,ij
Normalized residuals are displayed when one of the following conditions is met:
- The PALL, the PRINT, or the RESIDUAL option is specified, and METHOD=ML, METHOD=GLS, or METHOD=WLS, and the expensive information and Jacobian matrices are not computed for some other reason.
- RESIDUAL=NORM is specified.
The normalized residuals are equal to those computed by the LISREL VI program (Jreskog and Srbom 1985) except for the definition of the denominator NM in matrix U.

For estimation methods that are not BGLS estimation methods (Browne 1982, 1984), such as METHOD=NONE, METHOD=ULS, or METHOD=DWLS, the assumption of an asymptotic covariance matrix U of sample covariances does not seem to be appropriate. In this case, the normalized residuals should be replaced by the more relaxed variance standardized residuals. Computation of asymptotically standardized residuals requires computing the Jacobian and information matrices. This is computationally very expensive and is done only if the Jacobian matrix has to be computed for some other reason, that is, if at least one of the following items is true:

The default, PRINT, or PALL displayed output is requested, and neither the NOMOD nor NOSTDERR option is specified.
Either the MODIFICATION (included in PALL), PCOVES, or STDERR (included in default, PRINT, and PALL output) option is requested or RESIDUAL=ASYSTAND is specified.
The LEVMAR or NEWRAP optimization technique is used.
An OUTRAM= data set is specified without using the NOSTDERR option.
An OUTEST= data set is specified without using the NOSTDERR option.

Since normalized residuals use an overestimate of the asymptotic covariance matrix of residuals (the diagonal of U), the normalized residuals cannot be larger than the asymptotically standardized residuals (which use the diagonal of $U- JCov({\gamma}) J^'$ ).

Together with the residual matrices, the values of the average residual, the average off-diagonal residual, and the rank order of the largest values are displayed. The distribution of the normalized and standardized residuals is displayed also.

Goodness-of-Fit Indices Based on Residuals

The following items are computed for all five kinds of estimation:ULS, GLS, ML, WLS, and DWLS. All these indices are written to the OUTRAM= data set. The goodness of fit (GFI), adjusted goodness of fit (AGFI), and root mean square residual (RMR) are computed as in the LISREL VI program of J $\ddot{o}$ reskog and S $\ddot{o}$ rbom (1985).

Goodness-of-Fit Index
The goodness-of-fit index for the ULS, GLS, and ML estimation methods is
$GFI = 1 - {Tr( (W^{-1}(S- C))^2 ) \over Tr( (W^{-1}S)^2 )}$
but for WLS and DWLS estimation, it is
$GFI = 1 - {Vec(s_{ij} - c_{ij})^' W^{-1} Vec(s_{ij} - c_{ij}) \over Vec(s_{ij})^' W^{-1} Vec(s_{ij})}$
where W = diag for DWLS estimation, and Vec(s_ij - c_ij) denotes the vector of the n(n+1)/2 elements of the lower triangle of the symmetric matrix S- C. For a constant weight matrix W, the goodness-of-fit index is 1 minus the ratio of the minimum function value and the function value before any model has been fitted. The GFI should be between 0 and 1. The data probably do not fit the model if the GFI is negative or much larger than 1.
Adjusted Goodness-of-Fit Index
The AGFI is the GFI adjusted for the degrees of freedom of the model
$AGFI = 1 - {n(n+1) \over 2df} (1 - GFI)$
The AGFI corresponds to the GFI in replacing the total sum of squares by the mean sum of squares.

Caution:
- Large n and small df can result in a negative AGFI. For example, GFI=0.90, n=19, and df=2 result in an AGFI of -8.5.
- AGFI is not defined for a saturated model, due to division by df=0.
- AGFI is not sensitive to losses in df.
The AGFI should be between 0 and 1. The data probably do not fit the model if the AGFI is negative or much larger than 1. For more information, refer to Mulaik et al. (1989).
Root Mean Square Residual
The RMR is the mean of the squared residuals:
$RMR = \sqrt{{2 \over n(n+1)} \sum_i^n \sum_j^i (s_{ij} - c_{ij})^2}$
Parsimonious Goodness-of-Fit Index
The PGFI (Mulaik et al. 1989) is a modification of the GFI that takes the parsimony of the model into account:
$PGFI = {df_{min} \over df_0} GFI$
The PGFI uses the same parsimonious factor as the parsimonious normed Bentler-Bonett index (James, Mulaik, and Brett 1982).

Goodness-of-Fit Indices Based on the $\chi^2$

The following items are transformations of the overall $\chi^2$ value and in general depend on the sample size N. These indices are not computed for ULS or DWLS estimates.

Uncorrected ${\chi}^2$
The overall $\chi^2$ measure is the optimum function value F multiplied by N - 1 if a CORR or COV matrix is analyzed, or multiplied by N if a UCORR or UCOV matrix is analyzed. This gives the likelihood ratio test statistic for the null hypothesis that the predicted matrix C has the specified model structure against the alternative that C is unconstrained. The $\chi^2$ test is valid only if the observations are independent and identically distributed, the analysis is based on the nonstandardized sample covariance matrix S, and the sample size N is sufficiently large (Browne 1982; Bollen 1989b; J $\ddot{o}$ reskog and S $\ddot{o}$ rbom 1985). For ML and GLS estimates, the variables must also have an approximately multivariate normal distribution. The notation Prob>Chi**2 means "the probability under the null hypothesis of obtaining a greater $\chi^2$ statistic than that observed."
$\chi^2 = NM * F$
where F is the function value at the minimum.
${\chi}^2_0$ Value of the Independence Model
The ${\chi}^2_0$ value of the independence model
$\chi^2_0 = NM * f_0$
and the corresponding degrees of freedom df₀ can be used (in large samples) to evaluate the gain of explanation by fitting the specific model (Bentler 1989).
RMSEA Index (Steiger and Lind 1980)
The Steiger and Lind (1980) root mean squared error approximation (RMSEA) coefficient is
$\epsilon_{\alpha} = \sqrt{\max({F \over df} - {1 \over NM},0) }$
The lower and upper limits of the confidence interval are computed using the cumulative distribution function of the noncentral chi-squared distribution $\Phi(x|\lambda,df) = \alpha$ , with x=NM*F, $\lambda_L$ satisfying $\Phi(x|\lambda_L,df) = 1-\frac{\alpha}2$ ,and $\lambda_U$ satisfying $\Phi(x|\lambda_U,df) = \frac{\alpha}2$ :
$(\epsilon_{\alpha_L} ; \epsilon_{\alpha_U}) = (\sqrt{\lambda_L \over NM*df} ; \sqrt{\lambda_U \over NM*df})$
Refer to Browne and Du Toit (1992) for more details. The size of the confidence interval is defined by the option ALPHARMS= $\alpha$ , $0 \leq \alpha \leq 1$ .The default is $\alpha=0.1$ , which corresponds to the 90% confidence interval for the RMSEA.
Probability for Test of Close Fit (Browne and Cudeck 1993)
The traditional exact $\chi^2$ test hypothesis $H_{0}\colon \epsilon_{\alpha} = 0$ is replaced by the null hypothesis of close fit $H_{0}\colon \epsilon_{\alpha} \le 0.05$ and the exceedance probability P is computed as
$P = 1 - \Phi(x|\lambda^*,df)$
where x=NM*F and $\lambda^* = 0.05^2 * NM * df$ .The null hypothesis of close fit is rejected if P is smaller than a prespecified level (for example, P < 0.05).
Expected Cross Validation Index (Browne and Cudeck 1993)
For GLS and WLS, the estimator c of the ECVI is linearly related to AIC:
$c = F(S,C) + {2t \over NM}$
For ML estimation, c_ML is used.
$c_{ML} = F_{ML}(S,C) + {2t \over NM-n-1}$
The confidence interval (c_L ; c_U) for c is computed using the cumulative distribution function $\Phi(x|\lambda,df)$ of the noncentral chi-squared distribution,
$(c_L ; c_U) = ({\lambda_L + nnt \over NM} ; {\lambda_U + nnt \over NM})$
with nnt = n(n+1)/2 + t, x=NM * F, $\Phi(x|\lambda_U,df) = 1-\frac{\alpha}2$ , and $\Phi(x|\lambda_L,df) = \frac{\alpha}2$ .The confidence interval (c^*_L ; c^*_U) for c_ML is
$(c^*_L ; c^*_U) = ({\lambda^*_L + nnt \over NM-n-1} ; {\lambda^*_U + nnt \over NM-n-1})$
where nnt = n(n+1)/2 + t, x=(NM-n-1) * F, $\Phi(x|\lambda^*_U,df) = 1-\frac{\alpha}2$ and $\Phi(x|\lambda^*_L,df) = \frac{\alpha}2$ .Refer to Browne and Cudeck (1993). The size of the confidence interval is defined by the option ALPHAECV= $\alpha$ , $0 \leq \alpha \leq 1$ .The default is $\alpha=0.1$ , which corresponds to the 90% confidence interval for the ECVI.
Comparative Fit Index (Bentler 1989)

$CFI = 1 - {{\max(NM*f_{min}-df_{min},0)} \over {\max(NM*f_0-df_0},0)}$
Adjusted ${\chi}^2$ Value (Browne 1982)
If the variables are n-variate elliptic rather than normal and have significant amounts of multivariate kurtosis (leptokurtic or platykurtic), the $\chi^2$ value can be adjusted to
$\chi^2_{ell} = {\chi^2 \over \eta_2}$
where $\eta_2$ is the multivariate relative kurtosis coefficient.
Normal Theory Reweighted LS ${\chi}^2$ Value
This index is displayed only if METHOD=ML. Instead of the function value F_ML, the reweighted goodness-of-fit function F_GWLS is used,
$\chi^2_{GWLS} = NM * F_{GWLS}$
where F_GWLS is the value of the function at the minimum.
Akaike's Information Criterion (AIC) (Akaike 1974; Akaike 1987)
This is a criterion for selecting the best model among a number of candidate models. The model that yields the smallest value of AIC is considered the best.
$AIC = \chi^2 - 2df$
Consistent Akaike's Information Criterion (CAIC) (Bozdogan 1987)
This is another criterion, similar to AIC, for selecting the best model among alternatives. The model that yields the smallest value of CAIC is considered the best. CAIC is preferred by some people to AIC or the $\chi^2$ test.
$CAIC = \chi^2 - (ln(N) + 1) df$
Schwarz's Bayesian Criterion (SBC) (Schwarz 1978; Sclove 1987)
This is another criterion, similar to AIC, for selecting the best model. The model that yields the smallest value of SBC is considered the best. SBC is preferred by some people to AIC or the $\chi^2$ test.
$SBC = \chi^2 - ln(N) df$
McDonald's Measure of Centrality (McDonald and Hartmann 1992)

$CENT = exp( - {(\chi^2 - df) \over 2N} )$
Parsimonious Normed Fit Index (James, Mulaik, and Brett 1982)
The PNFI is a modification of Bentler-Bonett's normed fit index that takes parsimony of the model into account,
$PNFI = {df_{min} \over df_0} {(f_0 - f_{min}) \over f_0}$
The PNFI uses the same parsimonious factor as the parsimonious GFI of Mulaik et al. (1989).
Z-Test (Wilson and Hilferty 1931)
The Z-Test of Wilson and Hilferty assumes an n-variate normal distribution:
$Z = {{\sqrt[3]{\chi^2 \over df} - (1 - {2 \over 9 df})} \over {\sqrt{2 \over 9 df}}}$
Refer to McArdle (1988) and Bishop, Fienberg, and Holland (1977, p. 527) for an application of the Z-Test.
Nonnormed Coefficient (Bentler and Bonett 1980)

$\rho = {{f_0 / df_0 - f_{min} / df_{min}} \over {f_0 / df_0 - 1 / NM }}$
Refer to Tucker and Lewis (1973).
Normed Coefficient (Bentler and Bonett 1980)

$\Delta = {f_0 - f_{min} \over f_0}$
Mulaik et al. (1989) recommend the parsimonious weighted form PNFI.
Normed Index $\rho_1$ (Bollen 1986)

$\rho_1 = {{ f_0 / df_0 - f_{min} / df_{min}} \over {f_0 / df_0} }$
$\rho_1$ is always less than or equal to 1; $\rho_1 \lt 0$ is unlikely in practice. Refer to the discussion in Bollen (1989a).
Nonnormed Index $\Delta_2$ (Bollen 1989a)

$\Delta_2 = {{f_0 - f_{min}} \over {f_0 - {df \over NM}}}$
is a modification of Bentler & Bonett's $\Delta$ that uses df and "lessens the dependence" on N. Refer to the discussion in Bollen (1989b). $\Delta_2$ is identical to Mulaik et al.'s (1989) IFI2 index.
Critical N Index (Hoelter 1983)

$CN = { \chi^2_{crit} \over F} + 1$
where $\chi^2_{crit}$ is the critical chi-square value for the given df degrees of freedom and probability $\alpha= 0.05$ , and F is the value of the estimation criterion (minimization function). Refer to Bollen (1989b, p. 277). Hoelter (1983) suggests that CN should be at least 200; however, Bollen (1989b) notes that the CN value may lead to an overly pessimistic assessment of fit for small samples.

Squared Multiple Correlation

The following are measures of the squared multiple correlation for manifest and endogenous variables and are computed for all five estimation methods: ULS, GLS, ML, WLS, and DWLS. These coefficients are computed as in the LISREL VI program of J $\ddot{o}$ reskog and S $\ddot{o}$ rbom (1985). The DETAE, DETSE, and DETMV determination coefficients are intended to be global means of the squared multiple correlations for different subsets of model equations and variables. These coefficients are displayed only when you specify the PDETERM option with a RAM or LINEQS model.

R² Values Corresponding to Endogenous Variables

$R^2_i = 1 - {\hat{var({\zeta}_i)} \over \hat{var(eta_i)} }$
Total Determination of All Equations

$DETAE = 1 - {det(\hat{{{\Theta}}},\hat{{{\Psi}}}) \over det(\hat{Cov(y,x,eta)})}$
Total Determination of the Structural Equations

$DETSE = 1 - {det(\hat{{{\Psi}}}) \over det(\hat{Cov(eta )})}$
Total Determination of the Manifest Variables

$DETMV = 1 - {det(\hat{{{\Theta}}}) \over det(S)}$

Caution: In the LISREL program, the structural equations are defined by specifying the BETA matrix. In PROC CALIS, a structural equation has a dependent left-hand-side variable that appears at least once on the right-hand side of another equation, or the equation has at least one right-hand-side variable that is the left-hand-side variable of another equation. Therefore, PROC CALIS sometimes identifies more equations as structural equations than the LISREL program does.

Chapter Contents
Previous
Next
Top