Open CSV with a batch size

[davidegazze]

Hi,
I am starting to use this package. For my need, I would like to read a big csv file.
Due to memory issue I would like to do something like that:

N = 10000 // Total number of rows
M = 100 // batch size
start_index = 0
end_index = start_index + M
while (end_index < N) {
// Read only a part of the csv

Table table1 = Table.read().csv("filename.csv", start_index =start_index , end_index =end_index );
// Elaboration

.....
end_index = start_index + M
}

Does it possible?

[lwhite1]

You might be able to read the csv externally and feed the batches to tablesaw, but there’s no built in support

…

On Wed, Aug 9, 2023 at 12:31 PM Davide Gazzè ***@***.***> wrote: Hi, I am starting to use this package. For my need, I would like to read a big csv file. Due to memory issue I would like to do something like that: N = 10000 // Total number of rows M = 100 // batch size start_index = 0 end_index = start_index + M while (end_index < N) { // Read only a part of the csv Table table1 = Table.read().csv("filename.csv", start_index =start_index , end_index =end_index ); // Elaboration ..... end_index = start_index + M } Does it possible? — Reply to this email directly, view it on GitHub <#1226>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2FPASHAS7GXSUQO3JNDC3XUO3O7ANCNFSM6AAAAAA3KILVO4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[davidegazze]

Understood thank for your response