Skip to content

Commit

Permalink
loadTestDataFromDb: Load data in chunks of 100 rows (#90)
Browse files Browse the repository at this point in the history
With the new data from our first traffic collection on the web
(tweaselORG/data.tweasel.org#3), the request
database has gotten quite huge and I have seen debug-adapter runs take
a long time and even fail due to timeouts.

Luckily, the timeouts at least can be solved quite easily. Turns out,
we were already trying to load all matching rows in a single request.
Now, we instead load it in chunks of 100 rows. Setting the _size
parameter was sufficient for that—we had already implemented handling
of the next_url in responses and row limits.

With this change, I haven't seen any timeouts anymore so far and while
still slower, it isn't too bad anymore.

I still want to work on making data.tweasel.org as a whole faster again,
but this is a good change in any case.
  • Loading branch information
baltpeter authored Oct 7, 2024
1 parent 2136b16 commit d6fb693
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion scripts/lib/test-data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ export const loadTestDataFromDb = async (adapter: Adapter, options?: LoadTestDat
let nextUrl = `https://data.tweasel.org/data/requests.json?_shape=objects&_where=${encodeURIComponent(
whereClause
)}&_json=headers&_json=cookies&_nocol=initiator&_nocol=platform&_nocol=runType&_nofacet=1&_nosuggest=1&_nocount=1&_size=${
options?.rowLimit || 'max'
options?.rowLimit || '100'
}`;
while (nextUrl && (!options?.rowLimit || requests.length < options.rowLimit)) {
const res = await fetch(nextUrl).then((r) => r.json());
Expand Down

0 comments on commit d6fb693

Please sign in to comment.