Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possible Bug with dataframes exceeding 1,048,576 rows - 413 Request Entity Too Large #204

Open
mlamias opened this issue Jul 21, 2022 · 0 comments

Comments

@mlamias
Copy link

mlamias commented Jul 21, 2022

After assigning the appropriate API URL endpoint to socrata_url in my R code below, I have been attempting to write a dataframe that has about 2.5 million rows to Socrata using:

write.socrata(data_for_uploading, socrata_url, "REPLACE", keyring::key_get("socrata_user"), keyring::key_get("socrata_pw"))

But I receive the following error message: 413 Request Entity Too Large

I tried limiting my import to about a 1.5 million rows in my dataframe, data_for_uploading, and then uploading to Socrata, and they were added then without any problems. Once my data frame exceeds about 1.8 million rows (or about 250 MB in memory), I receive the 413 error message. Is it possible there's a bug in the RSocrata code that is limiting the upload size? If so, is there a work-around for this aside from splitting my dataframe in chunks with each chunk <= 1,500,000 rows, inserting the first chunk with the "REPLACE" argument, and then using an "UPSERT" for the remaining chunks?

As additional information, I am able to upload the 2.5 million record CSV file to Socrata directly through the web interface, and Socrata support indicated that there shouldn't be any limitation, and advised me to check with the RSocrata developers.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant