mxCompare: not working for WLS models

[tbates]

WLS models: diff-df seems inconsistent with diff-ep, p-value for comparison = 1 when should be <<1 , AIC for comparison model is much lower, when it should be slightly higher.

# make a litres variable in built-in mtcars dataframe
mtcars$litres = mtcars$disp/61.02

# Build DWLS w1 and w2
w1 = umxRAM("DWLS", data = mtcars, group = "am", type="DWLS",
	umxPath(c("wt", "litres"), to = "mpg"),
	umxPath("wt", with = "litres"),
	umxPath(var = c("wt", "litres", "mpg"))
)
w2 = umxModify(w1, regex=c("litres_to_mpg.*"), name="drop_l2mpg", comp=TRUE)

# Compare models
mxCompare(w1, w2)

DWLS Model Several queries:

1. ep differs by 2, but diffdf shows as 187???
2. P-value is 1? (same model with cov gave p-value ~.05 for comparison)
3. AIC is massively lower, when it should be slightly higher

base comparison ep     minus2LL  df       AIC    diffLL diffdf  p
1 DWLS       <NA> 12 3.952173e-18  84 -168.0000        NA     NA NA
2 DWLS drop_l2mpg 10 1.497845e-01 271 -533.7941 0.1497845    187  1

Same thing for a 1-group model: large change in df, p-value =1.0 (despite RMSEA getting bad)

w1 = umxRAM(data = mtcars, type="DWLS", "#DWLS 
   mpg ~ wt + litres"
)
w2 = umxModify(w1, update="litres_to_mpg", name="drop_l2mpg")
mxCompare(w1, w2)

  base comparison ep   minus2LL  df        AIC   diffLL diffdf  p
1 DWLS       <NA>  6  0.1988082  90  -179.8012       NA     NA NA
2 DWLS drop_l2mpg  5 35.3204724 523 -1044.1758 35.12166    433  1

footnotes:

ML Model To show what comparison should look like for the multi-group model:

# Build ML raw-data model m1 and m2 dropping litres -> mpg
m1 = umxRAM("ML", data = mtcars, group = "am",
	umxPath(c("wt", "litres"), to = "mpg"),
	umxPath("wt", with = "litres"),
	umxPath(v.m. = c("wt", "litres", "mpg"))
)
m2 = umxModify(m1, regex=c("litres_to_mpg.*"), name="drop_l2mpg", comp=TRUE)
  base comparison ep minus2LL df      AIC   diffLL diffdf          p
1   ML       <NA> 18 292.3695 78 136.3695       NA     NA         NA
2   ML drop_l2mpg 16 298.2125 80 138.2125 5.842923      2 0.05385492

FYI: For CRAN version 2.17.4, it appeared to be the case that the AIC was not printed for models with cov data, but appears to be fixed in current travis version (2.17.3.171)...

[over-soul]

I am having the same issue.

[RMKirkpatrick]

mxCompare reports AIC as NA for a comparison of 2 cov models... (despite AIC() working on each model)

That may be by design. I know that @mhunter1 deliberately chose not to put AIC in summary() output for WLS models.

[tbates]

But those 2 are cov models, solved by ML, and mxCompare should be showing AIC. AIC() returns a value so not sure why mxCompare isn't printing it.

The show-stopper issues for WLS regard ep, diffdf, P-value.

Oddly, mxCompare for WLS prints AIC (contrary to the decision to remove it).

[tbates]

Updated from CRAN version of OpenMx to the latest travis version the bug of cov data models not showing AIC appears fixed in the travis version

PS: the cran version is showing 2.17.4, but the travis version (newer) is reporting 2.17.3.171

OpenMx version: 2.17.3.171 [GIT v2.17.3-171-g5833c2d8] R version: R version 4.0.0 (2020-04-24) Platform: x86_64-apple-darwin17.0 MacOS: 11.0 Default optimizer: SLSQP NPSOL-enabled?: Yes OpenMP-enabled?: Yes

[RMKirkpatrick]

PS: the cran version is showing 2.17.4, but the travis version (newer) is reporting 2.17.3.171

I guess travis is building from a clone of the repository that lacks the latest tag?

[RMKirkpatrick]

Something relevant I just noticed... Look at the conditional starting at what is currently line 1019 of R/MxSummary.R . The behavior of AIC.MxModel depends partly upon whether model@output$Minus2LogLikelihood is NULL. However, look at line 700 of src/glue.cpp . If the backend was requested to compute the fitfunction value, then the 'Minus2LogLikelihood' element of the output slot is always populated with the fit, irrespective of the fit units. I bet that is legacy behavior from OpenMx version 1, when ML was the only kind of fitfunction available. But I bet that it is at least a partial cause of this issue. I bet there's another bug in mxCompare() as well.

[tbates]

Given the like bug in AIC/WLS fitfunction units, I wonder if issue #279 – logLik erroring with WLS models and seeing a NULL model@output$chi is also relevant here ...

[RMKirkpatrick]

@tbates It is at least tangentially relevant, but I think #279 reflects a bug in the backend. I will look into both this issue and #279 today.

[RMKirkpatrick]

I may have this one figured out. Fingers crossed...

[RMKirkpatrick]

See 34548e5 . @tbates , does this look better?:

> mxCompare(w1, w2)
  base comparison ep minus2LL  df       AIC diffLL diffdf  p
1 DWLS       <NA>  6       NA 275 -527.6498     NA     NA NA
2 DWLS drop_l2mpg  5       NA 276 -519.8687     NA      1 NA

[RMKirkpatrick]

34548e5 passes 'make test' on my local machine. However, it failed to build on travis.

[mcneale]

What caused the failure?

[RMKirkpatrick]

@mcneale Something related to package repository keys: https://travis-ci.org/github/OpenMx/OpenMx/builds/721768050 .

[jpritikin]

What caused the failure?

Disregard. Travis R is temporarily broken.

[tbates]

Hi Rob, thank you so much.

• estimated parameters is right now (1 not 2)
• diffdf is right (1 not 187 (!) )

But now p is NA, and AIC is displayed. Also minus2LL, diffLL are NA

I guess mxCompare should show model@output$fit not the contents of $Minus2LogLikelihood in the LL and diffLL columns? (see #297)

The p-value is missing. Without fit, p-value, or AIC, how can users can meaningfully compare models?

[RMKirkpatrick]

@tbates I only tried to fix what was obviously broken. Let's discuss how mxCompare() ought to behave during today's meeting.

[tbates]

As of OpenMx 2.18.1.14 [GIT v2.18.1-14-g8d49b0ac] (travis build from 2020-09-04),

1. AIC is not being displayed correctly: AIC for model 1 of mxCompare is not showing at all, and an incorrect value is shown for model 2. AIC in mxCompare should come from a call to the AIC() function.
2. Also seem to be back to wrong df? diffDF =187 should be 2.

Using the example model from the OP of this issue:

mxCompare(w1, w2)

base comparison ep minus2LL  df       AIC diffLL diffdf  p
1 DWLS       <NA> 12       NA  84        NA     NA     NA NA
2 DWLS drop_l2mpg 10       NA 271 -533.7941     NA    187 NA


AIC(w1)
[1] 0

AIC(w2)
[1] -12.41189

[tbates]

Can someone outline what needs fixing in mxCompare and I or someone else could have a go at fixing this? (student on a WLS project with a clock, but unable to proceed). Or if there's a work around to directly access and compute the missing DFs, delta DF, fit, AIC, p-values?

[over-soul]

Hi all,

I have a paper waiting for this fix and was wondering when you think it might be ready?

Thank you!

[tbates]

In OpenMx version: GIT v2.18.1-233-ga849f4e3, mxCompare on the two WLS models at the top of this issue yields sensible ep, chisq, df, and AIC, but I'm getting NaN for the Satorra-Bentler scaled chisq fit-difference and for the p value...

Perhaps mxCompare is not coping with the zero chisq of model 1?

mxCompare(w1,w2)
  base comparison ep    chisq df      AIC SBchisq diffdf   p
1 DWLS       <NA>  6 0.000000  0 12.00000      NA     NA  NA
2 DWLS drop_l2mpg  5 4.495419  1 14.49542     NaN      1 NaN

[tbates]

Any news on allowing comparison of WLS models? Or viable workaround to allow comparison of WLS models?