[WIP] Merge private API data into interception proxy #1817
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This refactor is preparation for an upcoming change which will give each test function a unique helper object.
Preparation for an upcoming change which will turn this helper into a class.
It isn’t intended be called as a constructor.
Motivation as in 369ad7e. To expand a little, in order to implement ECO-4821 (gathering detailed information about private API usage in the test suite), we will give each test function access to a unique helper object which is aware of the identity of the test case in which it is executing. This helper object will offer a method for recording the fact that a test function is calling a private API. By using the existing shared_helper object, we can also record the usage of private APIs by the various helper methods, access to which is currently funnelled through the shared_helper. The couple of methods that I’ve made static (randomString, whenPromiseSettles) are ones that are very frequently called _and_ which definitely don’t use any private APIs, so I’ve made them static so that, when we make the aforementioned change giving each test case its own helper object, we don’t need to pass a helper around just to enable these methods to be called. The choice of making just these two methods static was pretty arbitrary; once we’ve catalogued all of the private API usage it might turn out there some other ones that can be made static.
This removes the per-file SharedHelper instance and instead makes sure that each context which will call SharedHelper instance methods (e.g. test cases, test hooks) has a unique SharedHelper instance that is aware of the context from which it is being called. (Motivation for this change already explained in 18613c3.)
Preparation for recording private API usage by tests (ECO-4821).
This adds annotations to each place in the test suite where a private API is called. When the tests are run, these annotations are used to emit a JSON file containing private API usage information. (In ECO-4834, we’ll process this JSON file to generate some useful reports.) This JSON file also contains a list of all of the tests which get run, which will be used as part of the aforementioned reporting to allow us to generate a report of the tests that do not use any private APIs. The motivation for this gathering this data is that we intend to try and reuse the ably-js test suite as a unified test suite for all of our client libraries. In order to do this, we’ll need to address the usage of ably-js private APIs in the test suite, and to do that we need to first gather data about the current usage. I originally intended to put the changes into the current commit into a separate unified test suite branch, instead of merging them into `main`. But I don’t really want to maintain a long-running feature branch, and also I want to make sure that we add private API annotations to any new tests that we add to ably-js, hence I decided to put this commit into `main`. But maybe, once we’re done with the unified test suite, we’ll decide to remove these annotations. Given the context in which we’re gathering this data, in the interests of saving time I skipped adding some annotations to some files that I’m pretty sure will not form part of the unified test suite: - test/realtime/transports.test.js - test/browser/simple.test.js - test/browser/http.test.js - test/browser/connection.test.js - test/browser/modular.test.js - test/browser/push.test.js - test/rest/bufferutils.test.js If we decide at some point that we’d like to for whatever reason gather data on these tests, we can do that later. (Note I haven't been very consistent with how many times per test I record the usage of a given private API; it’s not a stat I’m particularly interested in here. But for the most part I’m doing it at the point of each invocation of the private API.) Resolves ECO-4821.
This uses the data gathered in 3a86ca3 to generate some CSV reports about private API usage in the test suite. We’ll use these reports as a starting point for deciding how to remove this private API usage when reusing the test suite as a unified test suite for all our client libraries. Resolves ECO-4834.
For upcoming interception proxy. TODO: once I’ve got rid of the subsequent commit with the old Python code, put this into the later commit instead
Implemented entirely as an mitmproxy addon. Abandoned because it didn’t give me sufficient control over WebSocket connection lifetimes.
This reverts commit eeebf5d. (I just want to keep it in the Git history for now, in case for some reason it’s useful to return to.)
- start-interception-proxy adapted from https://github.com/ably/sdk-test-proxy at 82e93a7 Some TODOs which aren’t really important right now because this is just a prototype: - TODO fix type checking for interception proxy — `npm run build` does it properly, but tried to reproduce the way we do it for modulereport and it didn’t work - TODO fix linting for interception proxy — doesn’t seem to be catching lint errors - TODO linting / type checking etc for Python code Also: > Add test:playwright:open-browser script > > Lets you open a headed browser which is configured to use the > interception proxy. Useful for local debugging of browser tests.
…07-16-merge-private-API-data-into-interception-proxy
lawrence-forooghian
force-pushed
the
ECO-4787-prototype-interception-proxy
branch
from
af3a497
to
bf8a5ce
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This merges #1800 into #1816. Once #1800 is merged, close this draft and incorporate its contents into #1816.