Skip to content
This repository has been archived by the owner on Oct 31, 2018. It is now read-only.

Handle scenario where the entire input file isn't needed #27

Open
kayousterhout opened this issue Aug 6, 2015 · 0 comments
Open

Handle scenario where the entire input file isn't needed #27

kayousterhout opened this issue Aug 6, 2015 · 0 comments

Comments

@kayousterhout
Copy link
Member

There are some situations where the function in a task only needs to read part of an input file (e.g., rdd.take()). We always read the entire input split, which is potentially wasteful; ideally, we should figure out how much is necessary to read and only read that much.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant