Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

model comparison with k-fold cross validation #18

Open
bnicenboim opened this issue May 10, 2024 · 0 comments
Open

model comparison with k-fold cross validation #18

bnicenboim opened this issue May 10, 2024 · 0 comments

Comments

@bnicenboim
Copy link

Hi,
I just discovered this package, it looks very cool, now I'm trying to do k-fold cross validation with the log-score rule, what I would do with brms with:

k <- loo::kfold_split_random(K = 10, N = nrow(df))
k1 <- kfold(m1, chains = 1, folds = k)
k2 <- kfold(m2, chains = 1, folds = k)
loo_compare(k1, k2)

I guess I should start with

folds <- vfold_cv(houses_train, v = 10)

fit_folds1 <- my_workflow1 |>
  tune::fit_resamples(folds)

fit_folds2 <- my_workflow2 |>
  tune::fit_resamples(folds)

But I'm not sure how I should compare the models. (I can provide a complete reprex, if it's not clear)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant