Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proxy mode for http proxies #734

Open
wants to merge 60 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
b18179c
plain_mode and tls_mode for proxy.
T-256 Jun 17, 2023
ebb5fea
fix uncovered `self._tls_mode`
T-256 Jun 17, 2023
225f1d4
lint
T-256 Jun 17, 2023
762927b
args added
T-256 Jun 17, 2023
c3c8776
sync added
T-256 Jun 17, 2023
394fd13
lint
T-256 Jun 17, 2023
7135a60
ProxyMode exposed
T-256 Jun 17, 2023
4f2a1df
lint
T-256 Jun 17, 2023
e717d64
cleanup
T-256 Jun 17, 2023
28c0e90
lint
T-256 Jun 17, 2023
b9c6606
update to IntFlag method
T-256 Jun 17, 2023
a1924b1
isort
T-256 Jun 17, 2023
63deacf
typecheck
T-256 Jun 17, 2023
74a93f9
typecheck again!
T-256 Jun 17, 2023
e63188a
pls :/
T-256 Jun 17, 2023
dc5f757
final lint
T-256 Jun 17, 2023
4849b76
remove is_tls
T-256 Jun 17, 2023
8bbc6b3
improve readablity
T-256 Jun 17, 2023
3f205eb
improve readability
T-256 Jun 19, 2023
e726700
lint
T-256 Jun 19, 2023
8de0110
docs
T-256 Jun 19, 2023
95e2bcb
fix tests
T-256 Jun 19, 2023
f769121
fix
T-256 Jun 19, 2023
81f5e47
doc
T-256 Jun 19, 2023
c62078e
doc
T-256 Jun 20, 2023
7d85951
doc
T-256 Jun 21, 2023
f163f48
doc
T-256 Jun 21, 2023
b74a6ce
Merge branch 'encode:master' into patch-1
T-256 Jun 30, 2023
067d76f
refactor ProxyMode
T-256 Jul 4, 2023
df51960
Merge branch 'encode:master' into patch-1
T-256 Jul 4, 2023
cef82e1
doc
T-256 Jul 6, 2023
fd9b2dc
Merge branch 'encode:master' into patch-1
T-256 Jul 7, 2023
fc56a07
Merge branch 'master' into patch-1
T-256 Jul 14, 2023
a815d7d
Update CHANGELOG.md
T-256 Jul 14, 2023
2218a7c
Link PR
T-256 Jul 17, 2023
3f7697e
Update CHANGELOG.md
T-256 Jul 17, 2023
899fb9b
Update docs/proxies.md
T-256 Jul 18, 2023
01379b0
Drop ProxyMode.DEFAULT
T-256 Jul 18, 2023
9a22d73
rename `FORWARD` to `GATEWAY`
T-256 Aug 27, 2023
5af7839
Update CHANGELOG.md
T-256 Aug 27, 2023
aac8aa3
Update CHANGELOG.md
T-256 Aug 27, 2023
2706c46
Merge branch 'encode:master' into patch-1
T-256 Aug 27, 2023
54e9a44
Update CHANGELOG.md
T-256 Aug 27, 2023
2f74621
rename `GATEWAY` to `FORWARD`
T-256 Aug 30, 2023
3d0d993
Merge branch 'master' into patch-1
tomchristie Sep 1, 2023
4a2dd15
typo
T-256 Sep 6, 2023
17b216d
Update docs/proxies.md
T-256 Sep 6, 2023
81b1903
Update docs/proxies.md
T-256 Sep 6, 2023
a244cde
Update docs/proxies.md
T-256 Sep 6, 2023
e1a88fd
Merge branch 'master' into patch-1
T-256 Oct 6, 2023
c18d351
Merge branch 'master' into patch-1
T-256 Oct 14, 2023
d275626
Merge branch 'master' into patch-1
T-256 Oct 28, 2023
4002450
Merge branch 'master' into patch-1
T-256 Nov 7, 2023
602ed8e
Merge branch 'master' into patch-1
T-256 Dec 4, 2023
053f0a0
Merge branch 'master' into patch-1
T-256 Dec 10, 2023
b894d10
Merge branch 'master' into patch-1
T-256 Feb 12, 2024
cc47c1c
Merge branch 'master' into patch-1
T-256 Feb 20, 2024
3a47db9
Merge branch 'master' into patch-1
T-256 Mar 9, 2024
a87dc08
Merge branch 'master' into patch-1
T-256 Mar 19, 2024
e6339ff
Merge branch 'master' into patch-1
T-256 Apr 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions docs/proxies.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,30 @@ proxy = httpcore.HTTPProxy(
)
```

## Proxy Connection Mode

There are two types of HTTP proxy:

1. Forwarding (HTTP [absolute-url](https://tools.ietf.org/html/rfc7230#section-5.3.2))
2. Tunneling (HTTP [`CONNECT` method](https://tools.ietf.org/html/rfc7231#section-4.3.6))

By default `httpcore` will use forwarding for http requests and tunneling for https requests.

You can change this behavior with `proxy_mode: httpcore.ProxyMode` parameter:
T-256 marked this conversation as resolved.
Show resolved Hide resolved

```py
import httpcore
import base64

proxy = httpcore.HTTPProxy(
proxy_url="http://127.0.0.1:8080/",
proxy_mode=httpcore.ProxyMode.TUNNEL,
)
```

Note that `ProxyMode.FORWARD` will enable forwarding https requests, so they will be visible for proxy server.
It means handling TLS stuffs like certificate validation would on proxy side.

## Proxy SSL

The `httpcore` package also supports HTTPS proxies for http and https destinations.
Expand Down
5 changes: 3 additions & 2 deletions httpcore/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
WriteError,
WriteTimeout,
)
from ._models import URL, Origin, Request, Response
from ._models import URL, Origin, ProxyMode, Request, Response
from ._ssl import default_ssl_context
from ._sync import (
ConnectionInterface,
Expand Down Expand Up @@ -75,8 +75,9 @@ def __init__(self, *args, **kwargs): # type: ignore
"request",
"stream",
# models
"Origin",
"URL",
"Origin",
"ProxyMode",
"Request",
"Response",
# async
Expand Down
61 changes: 34 additions & 27 deletions httpcore/_async/http_proxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from .._models import (
URL,
Origin,
ProxyMode,
Request,
Response,
enforce_bytes,
Expand All @@ -25,7 +26,6 @@
HeadersAsSequence = Sequence[Tuple[Union[bytes, str], Union[bytes, str]]]
HeadersAsMapping = Mapping[Union[bytes, str], Union[bytes, str]]


logger = logging.getLogger("httpcore.proxy")


Expand Down Expand Up @@ -75,6 +75,7 @@ def __init__(
uds: Optional[str] = None,
network_backend: Optional[AsyncNetworkBackend] = None,
socket_options: Optional[Iterable[SOCKET_OPTION]] = None,
proxy_mode: Optional[ProxyMode] = None,
) -> None:
"""
A connection pool for making HTTP requests.
Expand Down Expand Up @@ -110,6 +111,7 @@ def __init__(
`AF_INET6` address (IPv6).
uds: Path to a Unix Domain Socket to use instead of TCP sockets.
network_backend: A backend instance to use for handling network I/O.
proxy_mode: Allow HTTP connection be tunnelable and HTTPS be forwardable.
"""
super().__init__(
ssl_context=ssl_context,
Expand All @@ -136,6 +138,7 @@ def __init__(
self._ssl_context = ssl_context
self._proxy_ssl_context = proxy_ssl_context
self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
self._proxy_mode = proxy_mode
if proxy_auth is not None:
username = enforce_bytes(proxy_auth[0], name="proxy_auth")
password = enforce_bytes(proxy_auth[1], name="proxy_auth")
Expand All @@ -145,7 +148,9 @@ def __init__(
] + self._proxy_headers

def create_connection(self, origin: Origin) -> AsyncConnectionInterface:
if origin.scheme == b"http":
if (self._proxy_mode is ProxyMode.FORWARD) or (
self._proxy_mode is None and origin.scheme == b"http"
):
return AsyncForwardHTTPConnection(
proxy_origin=self._proxy_url.origin,
proxy_headers=self._proxy_headers,
Expand Down Expand Up @@ -298,31 +303,33 @@ async def handle_async_request(self, request: Request) -> Response:
raise ProxyError(msg)

stream = connect_response.extensions["network_stream"]

# Upgrade the stream to SSL
ssl_context = (
default_ssl_context()
if self._ssl_context is None
else self._ssl_context
)
alpn_protocols = ["http/1.1", "h2"] if self._http2 else ["http/1.1"]
ssl_context.set_alpn_protocols(alpn_protocols)

kwargs = {
"ssl_context": ssl_context,
"server_hostname": self._remote_origin.host.decode("ascii"),
"timeout": timeout,
}
async with Trace("start_tls", logger, request, kwargs) as trace:
stream = await stream.start_tls(**kwargs)
trace.return_value = stream

# Determine if we should be using HTTP/1.1 or HTTP/2
ssl_object = stream.get_extra_info("ssl_object")
http2_negotiated = (
ssl_object is not None
and ssl_object.selected_alpn_protocol() == "h2"
)
http2_negotiated = False

if self._remote_origin.scheme == b"https":
# Upgrade the stream to SSL
ssl_context = (
default_ssl_context()
if self._ssl_context is None
else self._ssl_context
)
alpn_protocols = ["http/1.1", "h2"] if self._http2 else ["http/1.1"]
ssl_context.set_alpn_protocols(alpn_protocols)

kwargs = {
"ssl_context": ssl_context,
"server_hostname": self._remote_origin.host.decode("ascii"),
"timeout": timeout,
}
async with Trace("start_tls", logger, request, kwargs) as trace:
stream = await stream.start_tls(**kwargs)
trace.return_value = stream

# Determine if we should be using HTTP/1.1 or HTTP/2
ssl_object = stream.get_extra_info("ssl_object")
http2_negotiated = (
ssl_object is not None
and ssl_object.selected_alpn_protocol() == "h2"
)

# Create the HTTP/1.1 or HTTP/2 connection
if http2_negotiated or (self._http2 and not self._http1):
Expand Down
6 changes: 6 additions & 0 deletions httpcore/_models.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import enum
from typing import (
Any,
AsyncIterable,
Expand Down Expand Up @@ -490,3 +491,8 @@ async def aclose(self) -> None:
)
if hasattr(self.stream, "aclose"):
await self.stream.aclose()


class ProxyMode(enum.IntEnum):
FORWARD = 1
TUNNEL = 2
61 changes: 34 additions & 27 deletions httpcore/_sync/http_proxy.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from .._models import (
URL,
Origin,
ProxyMode,
Request,
Response,
enforce_bytes,
Expand All @@ -25,7 +26,6 @@
HeadersAsSequence = Sequence[Tuple[Union[bytes, str], Union[bytes, str]]]
HeadersAsMapping = Mapping[Union[bytes, str], Union[bytes, str]]


logger = logging.getLogger("httpcore.proxy")


Expand Down Expand Up @@ -75,6 +75,7 @@ def __init__(
uds: Optional[str] = None,
network_backend: Optional[NetworkBackend] = None,
socket_options: Optional[Iterable[SOCKET_OPTION]] = None,
proxy_mode: Optional[ProxyMode] = None,
) -> None:
"""
A connection pool for making HTTP requests.
Expand Down Expand Up @@ -110,6 +111,7 @@ def __init__(
`AF_INET6` address (IPv6).
uds: Path to a Unix Domain Socket to use instead of TCP sockets.
network_backend: A backend instance to use for handling network I/O.
proxy_mode: Allow HTTP connection be tunnelable and HTTPS be forwardable.
"""
super().__init__(
ssl_context=ssl_context,
Expand All @@ -136,6 +138,7 @@ def __init__(
self._ssl_context = ssl_context
self._proxy_ssl_context = proxy_ssl_context
self._proxy_headers = enforce_headers(proxy_headers, name="proxy_headers")
self._proxy_mode = proxy_mode
if proxy_auth is not None:
username = enforce_bytes(proxy_auth[0], name="proxy_auth")
password = enforce_bytes(proxy_auth[1], name="proxy_auth")
Expand All @@ -145,7 +148,9 @@ def __init__(
] + self._proxy_headers

def create_connection(self, origin: Origin) -> ConnectionInterface:
if origin.scheme == b"http":
if (self._proxy_mode is ProxyMode.FORWARD) or (
self._proxy_mode is None and origin.scheme == b"http"
):
return ForwardHTTPConnection(
proxy_origin=self._proxy_url.origin,
proxy_headers=self._proxy_headers,
Expand Down Expand Up @@ -298,31 +303,33 @@ def handle_request(self, request: Request) -> Response:
raise ProxyError(msg)

stream = connect_response.extensions["network_stream"]

# Upgrade the stream to SSL
ssl_context = (
default_ssl_context()
if self._ssl_context is None
else self._ssl_context
)
alpn_protocols = ["http/1.1", "h2"] if self._http2 else ["http/1.1"]
ssl_context.set_alpn_protocols(alpn_protocols)

kwargs = {
"ssl_context": ssl_context,
"server_hostname": self._remote_origin.host.decode("ascii"),
"timeout": timeout,
}
with Trace("start_tls", logger, request, kwargs) as trace:
stream = stream.start_tls(**kwargs)
trace.return_value = stream

# Determine if we should be using HTTP/1.1 or HTTP/2
ssl_object = stream.get_extra_info("ssl_object")
http2_negotiated = (
ssl_object is not None
and ssl_object.selected_alpn_protocol() == "h2"
)
http2_negotiated = False

if self._remote_origin.scheme == b"https":
# Upgrade the stream to SSL
ssl_context = (
default_ssl_context()
if self._ssl_context is None
else self._ssl_context
)
alpn_protocols = ["http/1.1", "h2"] if self._http2 else ["http/1.1"]
ssl_context.set_alpn_protocols(alpn_protocols)

kwargs = {
"ssl_context": ssl_context,
"server_hostname": self._remote_origin.host.decode("ascii"),
"timeout": timeout,
}
with Trace("start_tls", logger, request, kwargs) as trace:
stream = stream.start_tls(**kwargs)
trace.return_value = stream

# Determine if we should be using HTTP/1.1 or HTTP/2
ssl_object = stream.get_extra_info("ssl_object")
http2_negotiated = (
ssl_object is not None
and ssl_object.selected_alpn_protocol() == "h2"
)

# Create the HTTP/1.1 or HTTP/2 connection
if http2_negotiated or (self._http2 and not self._http1):
Expand Down
Loading