Disk caching and resume

[vjpr]

Would be nice to have a way to auto-resume between process runs and cache responses.

The rate limits are way too low from Twitter, and you will always get rate-limited eventually.

All responses should be cached to disk along with the next_token. Should wait for the rate-limits before retrying. If the process crashes, it can be easily resumed automatically by reading the persisted next_token...which I assume stays valid over a 15min wait period.

Maybe a wrapper library is best for this.

node-twitter-api-v2 would need to allow passing in next_token and returning it from its fetch methods...which it looks like it does, just not documented.

Related: #187

[alkihis]

It could be a 2-phase implementation:

1. preRequest / postRequest hooks inside main package

preRequest executes before every request, taking request URL, params, etc in arguments. It can alter URL/parameters to change what request is going to do. We can create special exception, like RequestAlreadyAvailableInCacheException, that this function can throw if the request should not be made but served from cache (and response data to give can be part of exception data).

postRequest executes after every request, taking request URL, params, and response (if succeeded) in arguments. It can be used as a hook to save data, like next_token (injectable later through preRequest

2. Plugin packages, that expose pre-construct preRequest/postRequest hooks able to inject/save next_token or caching to disk.