Fitting Linear and Generalized Linear Models to out of the memory data sets in Divide and Recombine approach
big_drglm.RmdTo fit Generalized Linear Models (GLMs) on large data sets that exceed memory limits, you can use the ‘big.drglm’ function. The ‘biglm’ package has been used for such applications, but it doesn’t support factors. The ‘speedglm’ package does support factors, but its application on large CSV files for fitting GLMs hasn’t been demonstrated. We’ll show how to fit large datasets in chunks for model fitting.
Now , lets create a toy data set
set.seed(123)
#Number of rows to be generated
n <- 1000000
#creating dataset
dataset <- data.frame(
Var_1 = round(rnorm(n, mean = 50, sd = 10)),
Var_2 = round(rnorm(n, mean = 7.5, sd = 2.1)),
Var_3 = as.factor(sample(c("0", "1"), n, replace = TRUE)),
Var_4 = as.factor(sample(c("0", "1", "2"), n, replace = TRUE)),
Var_5 = as.factor(sample(0:15, n, replace = TRUE)),
Var_6 = round(rnorm(n, mean = 60, sd = 5))
)
# Save the dataset to a temporary file
temp_file <- tempfile(fileext = ".csv")
write.csv(dataset, file = temp_file, row.names = FALSE)
# Path to the temporary file
dataset_path <- temp_file
dataset_path # Display the path to the temporary file## [1] "/tmp/RtmpCHttjW/file18b52c2d9157.csv"We’ve saved the data set to disk, assuming the data set is too large to fit into memory. We’ll explore how to handle such data sets. The ‘drglm’ package offers a function called ‘make.data’, which is a slight modification of the ‘biglm’ and ‘speedglm’ packages. It’s specifically designed for large CSV files that can’t be loaded into memory.
# Path to the temporary file
dataset_path <- temp_file
dataset_path # Display the path to the temporary file## [1] "/tmp/RtmpCHttjW/file18b52c2d9157.csv"
# Initialize the data reading function with the data set path and chunk size
da <- drglm::make.data(dataset_path, chunksize = 100000)Fitting MLR Models
# Fitting MLR Model
nmodel <-
drglm::big.drglm(da, formula = Var_1 ~ Var_2+ factor(Var_3)+
factor(Var_4)+ factor(Var_5)+ Var_6, 10, family="gaussian")
# View the results table
print(nmodel)## Estimate standard error t value Pr(>|t|)
## (Intercept) 49.9938921629 0.132222414 378.10451704 0.0000000
## Var_2 -0.0045648136 0.004721587 -0.96679652 0.3336458
## factor(Var_3)1 0.0140777358 0.020007935 0.70360764 0.4816772
## factor(Var_4)1 -0.0070996373 0.024495862 -0.28983006 0.7719462
## factor(Var_4)2 0.0031706649 0.024509469 0.12936490 0.8970689
## factor(Var_5)1 -0.0572865412 0.056620740 -1.01175896 0.3116533
## factor(Var_5)2 -0.0110496857 0.056615948 -0.19516914 0.8452605
## factor(Var_5)3 -0.0448607620 0.056694044 -0.79127821 0.4287817
## factor(Var_5)4 -0.0268086198 0.056646008 -0.47326582 0.6360235
## factor(Var_5)5 0.0466234380 0.056633526 0.82324801 0.4103670
## factor(Var_5)6 -0.0270480470 0.056580123 -0.47804857 0.6326156
## factor(Var_5)7 0.0433609651 0.056668648 0.76516675 0.4441723
## factor(Var_5)8 -0.0297390739 0.056712763 -0.52438062 0.6000138
## factor(Var_5)9 0.0432931453 0.056669237 0.76396203 0.4448899
## factor(Var_5)10 0.0672852618 0.056660012 1.18752643 0.2350200
## factor(Var_5)11 -0.0583903308 0.056635285 -1.03098856 0.3025462
## factor(Var_5)12 0.0091184125 0.056623543 0.16103571 0.8720653
## factor(Var_5)13 0.0049039721 0.056698929 0.08649144 0.9310758
## factor(Var_5)14 -0.0151239426 0.056584592 -0.26728023 0.7892534
## factor(Var_5)15 0.0548865463 0.056635643 0.96911668 0.3324870
## Var_6 0.0004866552 0.001996329 0.24377510 0.8074050
## 95% CI
## (Intercept) [ 49.73 , 50.25 ]
## Var_2 [ -0.01 , 0 ]
## factor(Var_3)1 [ -0.03 , 0.05 ]
## factor(Var_4)1 [ -0.06 , 0.04 ]
## factor(Var_4)2 [ -0.04 , 0.05 ]
## factor(Var_5)1 [ -0.17 , 0.05 ]
## factor(Var_5)2 [ -0.12 , 0.1 ]
## factor(Var_5)3 [ -0.16 , 0.07 ]
## factor(Var_5)4 [ -0.14 , 0.08 ]
## factor(Var_5)5 [ -0.06 , 0.16 ]
## factor(Var_5)6 [ -0.14 , 0.08 ]
## factor(Var_5)7 [ -0.07 , 0.15 ]
## factor(Var_5)8 [ -0.14 , 0.08 ]
## factor(Var_5)9 [ -0.07 , 0.15 ]
## factor(Var_5)10 [ -0.04 , 0.18 ]
## factor(Var_5)11 [ -0.17 , 0.05 ]
## factor(Var_5)12 [ -0.1 , 0.12 ]
## factor(Var_5)13 [ -0.11 , 0.12 ]
## factor(Var_5)14 [ -0.13 , 0.1 ]
## factor(Var_5)15 [ -0.06 , 0.17 ]
## Var_6 [ 0 , 0 ]In this similar manner , we can fit other family models using the above data set.
Fitting logistic Regression Model
# Fitting Logistic Model
bmodel <- drglm::big.drglm(da,formula = Var_3 ~ Var_1+ Var_2+
factor(Var_4)+ factor(Var_5)+ Var_6,
10, family="binomial")
# View the results table
print(bmodel)## Estimate Odds Ratio standard error z value Pr(>|z|)
## (Intercept) 0.0498850493 1.0511503 0.0281923787 1.7694516 0.07681854
## Var_1 0.0001406428 1.0001407 0.0001999858 0.7032641 0.48189121
## Var_2 -0.0010289335 0.9989716 0.0009441471 -1.0898021 0.27580035
## factor(Var_4)1 -0.0009157951 0.9990846 0.0048982015 -0.1869656 0.85168762
## factor(Var_4)2 0.0008660010 1.0008664 0.0049009500 0.1767006 0.85974354
## factor(Var_5)1 -0.0090198819 0.9910207 0.0113218905 -0.7966763 0.42563905
## factor(Var_5)2 -0.0103609021 0.9896926 0.0113209121 -0.9152003 0.36008649
## factor(Var_5)3 -0.0111773346 0.9888849 0.0113364057 -0.9859681 0.32414876
## factor(Var_5)4 -0.0051583819 0.9948549 0.0113269975 -0.4554059 0.64881723
## factor(Var_5)5 -0.0166414412 0.9834963 0.0113247263 -1.4694784 0.14170306
## factor(Var_5)6 -0.0170752441 0.9830697 0.0113137869 -1.5092422 0.13123691
## factor(Var_5)7 -0.0115591956 0.9885074 0.0113313552 -1.0201071 0.30767768
## factor(Var_5)8 -0.0190175646 0.9811621 0.0113399851 -1.6770361 0.09353542
## factor(Var_5)9 -0.0024879742 0.9975151 0.0113313423 -0.2195657 0.82620940
## factor(Var_5)10 -0.0039725724 0.9960353 0.0113297226 -0.3506328 0.72586385
## factor(Var_5)11 -0.0189525009 0.9812260 0.0113250085 -1.6735088 0.09422718
## factor(Var_5)12 -0.0080661323 0.9919663 0.0113222078 -0.7124169 0.47620665
## factor(Var_5)13 -0.0167293199 0.9834098 0.0113376220 -1.4755581 0.14006256
## factor(Var_5)14 -0.0270868122 0.9732767 0.0113146115 -2.3939675 0.01666723
## factor(Var_5)15 -0.0148850714 0.9852252 0.0113248937 -1.3143674 0.18872258
## Var_6 -0.0006315246 0.9993687 0.0003991918 -1.5820079 0.11364778
## 95% CI
## (Intercept) [ -0.01 , 0.11 ]
## Var_1 [ 0 , 0 ]
## Var_2 [ 0 , 0 ]
## factor(Var_4)1 [ -0.01 , 0.01 ]
## factor(Var_4)2 [ -0.01 , 0.01 ]
## factor(Var_5)1 [ -0.03 , 0.01 ]
## factor(Var_5)2 [ -0.03 , 0.01 ]
## factor(Var_5)3 [ -0.03 , 0.01 ]
## factor(Var_5)4 [ -0.03 , 0.02 ]
## factor(Var_5)5 [ -0.04 , 0.01 ]
## factor(Var_5)6 [ -0.04 , 0.01 ]
## factor(Var_5)7 [ -0.03 , 0.01 ]
## factor(Var_5)8 [ -0.04 , 0 ]
## factor(Var_5)9 [ -0.02 , 0.02 ]
## factor(Var_5)10 [ -0.03 , 0.02 ]
## factor(Var_5)11 [ -0.04 , 0 ]
## factor(Var_5)12 [ -0.03 , 0.01 ]
## factor(Var_5)13 [ -0.04 , 0.01 ]
## factor(Var_5)14 [ -0.05 , 0 ]
## factor(Var_5)15 [ -0.04 , 0.01 ]
## Var_6 [ 0 , 0 ]Fitting Poisson Regression Model
# Fitting Poisson Regression Model
pmodel <- drglm::big.drglm(da,
formula = Var_5 ~ Var_1+ Var_2+ factor(Var_3)+
factor(Var_4)+ Var_6, 10, family="poisson")
# View the results table
print(pmodel)## Estimate Odds Ratio standard error z value Pr(>|z|)
## (Intercept) 1.993906e+00 7.3441637 1 1.993906e+00 0.04616233
## Var_1 8.520103e-05 1.0000852 1 8.520103e-05 0.99993202
## Var_2 5.068223e-04 1.0005070 1 5.068223e-04 0.99959561
## factor(Var_3)1 -2.064546e-03 0.9979376 1 -2.064546e-03 0.99835273
## factor(Var_4)1 -2.984466e-03 0.9970200 1 -2.984466e-03 0.99761874
## factor(Var_4)2 -2.436860e-03 0.9975661 1 -2.436860e-03 0.99805567
## Var_6 2.669898e-04 1.0002670 1 2.669898e-04 0.99978697
## 95% CI
## (Intercept) [ 0.03 , 3.95 ]
## Var_1 [ -1.96 , 1.96 ]
## Var_2 [ -1.96 , 1.96 ]
## factor(Var_3)1 [ -1.96 , 1.96 ]
## factor(Var_4)1 [ -1.96 , 1.96 ]
## factor(Var_4)2 [ -1.96 , 1.96 ]
## Var_6 [ -1.96 , 1.96 ]This package currently allows for fitting GLMs to very large data sets in CSV format. Future updates will enable users to fit GLMs to data sets in other formats.