idk/httptunnel
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

move to make a page

Browse Source
This commit is contained in:
idk
2019-03-12 16:36:29 -04:00
parent 205aba31cc
commit b8e4e915bf
5 changed files with 587 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -15,4 +15,4 @@ httpproxy-*
*.out
README.md.asc
doc/doc
docs/doc

269
docs/README.md Normal file
View File

@ -0,0 +1,269 @@
How to Write an HTTP Proxy for I2P in Go
========================================
In this short guide, I will show you how to create a standalone HTTP proxy which
will only ever make connections over the I2P network as an introduction to I2P
application development in Go. Although most of the readers will have an
understanding of HTTP proxies already, and there is already an HTTP proxy to I2P
in the core I2P software, there is a method to this madness:
Why another Go HTTP Proxy Tutorial?
-----------------------------------
There are tons of tutorials out there to write HTTP proxies in Go(1). There is a
ton of example code out there you can just copy-and-paste, modify to suit your
needs, and use. So why another one now?
Well one reason there are so many is that Go makes it very easy to write a
reasonably reliable HTTP proxy. It's a simple project that shows why Go can be
good at helping you do the things you need, if what you need is an HTTP client
or server. Likewise, the HTTP proxy makes a good introduction to why I2P can be
good at helping you do the things you need, if what you need to do is make HTTP
requests in privacy. So it exists to bridge the mental gap between the Go
application and I2P client in a way which is accessible to people who learned Go
from the internet like me.
*warning:* Be careful and do not use this example code unmodified in your
real application. It deliberately references an earlier state of the application
in the root of this repository. This is a teaching tool and not production code.
What is an HTTP Proxy?
----------------------
At it's simplest it's two things, it's an HTTP server, and an HTTP Client. When
the HTTP server recieves a request, it handles it by forwarding it to the HTTP
client, which then retrieves and returns it to the server, which forwards it
back to the requestor. In-between recieving the request and forwarding it to
the client, the server can make changes to how the requests are handled. So in
our example, we'll be leaving the example HTTP server in place, but modifying
the HTTP client to handle requests by routing them to I2P.
Step One: Setting up the HTTP Server Structure
----------------------------------------------
Creating a custom HTTP server in Go is easy. We just need to create a type
called Proxy, which implements the http.Server that forwards you to the client.
First, import the goSam library.
import (
"github.com/eyedeekay/goSam"
)
goSam implements a transport which is compatible with Go's http.Client. The
proxy will then need to contain a goSam.Client and an http.Client.
type Proxy struct {
Sam *goSam.Client
Client *http.Client
}
In order to implement the http Server interface, we need the ServeHTTP function
within the Proxy structure, which has the following signature:
func (p *Proxy) ServeHTTP(wr http.ResponseWriter, req *http.Request)
Where req is the request made by the user, and wr is the the stream where the
response will be written and returned to the user.
When you create an instance of this struct, you should make sure to first set
up an instance of goSam.Client and pass it to the struct, like this:
sam, err := goSam.NewClientFromOptions(
goSam.SetUnpublished(true),
)
if err != nil {
log.Fatal(err)
}
handler := &i2phttpproxy.Proxy{
Sam: sam,
}
Note that goSam has been set up to use an Unpublished leaseset in this case,
because it will be used as a client and not a service and doesn't need to be
discovered until it starts communicating with a service.
Step Two: Anonymize the application
-----------------------------------
If you're developing a privacy-aware application, it's important to anonymize
more than just the connection, but also the application as well. In the case of
an HTTP proxy, you should probably at least scrub some headers out. To do this,
make a slice with the headers you want to delete at the first hop, and a
function which deletes them from the underlying structure.
var hopHeaders = []string{
"Accept-Language",
"Connection",
"Keep-Alive",
"Proxy-Authenticate",
"Proxy-Authorization",
"Proxy-Connection",
"Trailers",
"Upgrade",
"X-Forwarded-For",
"X-Real-IP",
}
func delHopHeaders(header http.Header) {
for _, h := range hopHeaders {
header.Del(h)
}
....
If you want, you can also re-write the user agent string in this function as
well.
....
if header.Get("User-Agent") != "MYOB/6.66 (AN/ON)" {
header.Set("User-Agent", "MYOB/6.66 (AN/ON)")
}
}
Because our proxy will only be used for I2P addresses, we'll want ServeHTTP to
ignore non-I2P addresses. We can do this by examining and dropping the request
when the server recieves it, like this:
func (p *Proxy) ServeHTTP(wr http.ResponseWriter, req *http.Request) {
if req.URL.Scheme != "http" && !strings.HasSuffix(req.URL.Host, ".i2p") {
msg := "unsupported protocal scheme " + req.URL.Scheme
http.Error(wr, msg, http.StatusBadRequest)
log.Println(msg)
return
}
...
Once you're done, add the header-scrubbing function to the ServeHTTP function.
...
delHopHeaders(req.Header)
...
Step Three: Actually handle the request
---------------------------------------
Now that we've got the application anonymized, it's time to handle the request.
...
p.get(wr, req)
}
To keep the steps of the procedure cleanly separated, I chose to contain this
in it's own function. It just carries the same signature of the ServeHTTP
function, which
func (p *Proxy) get(wr http.ResponseWriter, req *http.Request) {
req.RequestURI = ""
resp, err := p.Client.Do(req)
if err != nil {
log.Println("ServeHTTP:", err)
return
}
defer resp.Body.Close()
wr.WriteHeader(resp.StatusCode)
io.Copy(wr, resp.Body)
}
But wait! Where's all the I2P stuff? What will this do if it's not connected to
I2P? It will just be an HTTP Proxy that doesn't work, because it's already
blacklisting non-I2P addresses.
Step Four: Set up the I2P Client
--------------------------------
Setting up the I2P Client is really easy, but a good example would be a little
long so I decided to extract it out to it's own function and walk through the
steps. In the end, we need to come up with a new http.Client which uses i2p as a
Transport. In order to ensure that it uses the same settings as the goSam
client, the function is part of the Proxy struct.
func (p *Proxy) NewClient() *http.Client {
return &http.Client{
In order to use the resulting http.Client with i2p, you'll need to alter it's
DialContext function to use the one from goSam instead.
Transport: &http.Transport{
DialContext: p.Sam.DialContext,
....
It may also be helpful to limit connections, set some timeouts(Go by default
expects you to set timeouts). Some example settings you might change would be:
....
MaxConnsPerHost: 1,
MaxIdleConns: 0,
MaxIdleConnsPerHost: 1,
DisableKeepAlives: false,
ResponseHeaderTimeout: time.Second * 600,
IdleConnTimeout: time.Second * 300,
},
CheckRedirect: nil,
Timeout: time.Second * 600,
}
}
Step Five: Create the main() function
-------------------------------------
Now it's finally time to put it all together and make our code runnable. First,
lets make it accept a flag which allows the user to determine which port it
runs on. I don't know what ports other people use:
func main() {
var addr = flag.String("addr", "127.0.0.1:7950", "The addr of the application.")
flag.Parse()
...
Next, set up the goSam client and hand it off to a new instance of *Proxy:
...
sam, err := goSam.NewClientFromOptions(
goSam.SetHost("127.0.0.1"),
goSam.SetPort("7656"),
goSam.SetUnpublished(true),
goSam.SetInLength(uint(2)),
goSam.SetOutLength(uint(2)),
goSam.SetInQuantity(uint(1)),
goSam.SetOutQuantity(uint(1)),
goSam.SetInBackups(uint(1)),
goSam.SetOutBackups(uint(1)),
goSam.SetReduceIdle(true),
goSam.SetReduceIdleTime(uint(2000000)),
)
if err != nil {
log.Fatal(err)
}
handler := &Proxy{
Sam: sam,
}
...
Use that SAM connection to set up an HTTP client:
...
handler.Client = handler.NewClient()
...
And serve it up. It's just that easy.
...
log.Println("Starting proxy server on", *addr)
if err := http.ListenAndServe(*addr, handler); err != nil {
log.Fatal("ListenAndServe:", err)
}
}
:)
Credits/Notes:
==============
1. [This guide is based on a guide for the clearnet](https://medium.com/@mlowicki/http-s-proxy-in-golang-in-less-than-100-lines-of-code-6a51c2f2c38c)
and that is only 10% because it was easier for me. 90% of the reason is to
illustrate that *using I2P in your application is, in all likelihood, not that*
*different than what you're already doing.*\* In all, this example is only 111
lines of code long minus comments.
* if the rest of your application only sends and recieves that which is
non-linkable and absolutely necessary to its operation.

BIN
docs/README.pdf Normal file
View File

Binary file not shown.

142
docs/httptunnel.go Normal file
View File

@ -0,0 +1,142 @@
package main
import (
"crypto/tls"
"flag"
"io"
"log"
"net/http"
"strings"
"time"
)
// tutorial line 48
import (
"github.com/eyedeekay/goSam"
)
func copyHeader(dst, src http.Header) {
for k, vv := range src {
for _, v := range vv {
dst.Add(k, v)
}
}
}
// tutorial line 94
var hopHeaders = []string{
"Accept-Language",
"Connection",
"Keep-Alive",
"Proxy-Authenticate",
"Proxy-Authorization",
"Proxy-Connection",
"Trailers",
"Upgrade",
"X-Forwarded-For",
"X-Real-IP",
}
// tutorial line 107
func delHopHeaders(header http.Header) {
for _, h := range hopHeaders {
header.Del(h)
}
// tutorial line 117
if header.Get("User-Agent") != "MYOB/6.66 (AN/ON)" {
header.Set("User-Agent", "MYOB/6.66 (AN/ON)")
}
}
// tutorial line 55
type Proxy struct {
Sam *goSam.Client
Client *http.Client
}
// NewClient is on 181, 246
func (p *Proxy) NewClient() *http.Client {
return &http.Client{
// tutorial line 187
Transport: &http.Transport{
DialContext: p.Sam.DialContext,
//tutorial line 195
MaxConnsPerHost: 1,
MaxIdleConns: 0,
MaxIdleConnsPerHost: 1,
DisableKeepAlives: false,
ResponseHeaderTimeout: time.Second * 600,
IdleConnTimeout: time.Second * 300,
TLSNextProto: make(map[string]func(authority string, c *tls.Conn) http.RoundTripper),
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
},
CheckRedirect: nil,
Timeout: time.Second * 600,
}
}
// ServeHTTP is on line 63, 126
func (p *Proxy) ServeHTTP(wr http.ResponseWriter, req *http.Request) {
log.Println(req.RemoteAddr, " ", req.Method, " ", req.URL)
if req.URL.Scheme != "http" && req.URL.Scheme != "https" && !strings.HasSuffix(req.URL.Host, ".i2p") {
msg := "unsupported protocal scheme " + req.URL.Scheme
http.Error(wr, msg, http.StatusBadRequest)
log.Println(msg)
return
}
// tutorial line 137
delHopHeaders(req.Header)
p.get(wr, req)
}
// tutorial lines 147, 155
func (p *Proxy) get(wr http.ResponseWriter, req *http.Request) {
req.RequestURI = ""
resp, err := p.Client.Do(req)
if err != nil {
log.Println("ServeHTTP:", err)
return
}
defer resp.Body.Close()
wr.WriteHeader(resp.StatusCode)
io.Copy(wr, resp.Body)
}
// main is on tutorial line 214
func main() {
var addr = flag.String("addr", "127.0.0.1:7950", "The addr of the application.")
flag.Parse()
// tutorial line 71, 222
sam, err := goSam.NewClientFromOptions(
goSam.SetHost("127.0.0.1"),
goSam.SetPort("7656"),
goSam.SetUnpublished(true),
goSam.SetInLength(uint(2)),
goSam.SetOutLength(uint(2)),
goSam.SetInQuantity(uint(1)),
goSam.SetOutQuantity(uint(1)),
goSam.SetInBackups(uint(1)),
goSam.SetOutBackups(uint(1)),
goSam.SetReduceIdle(true),
goSam.SetReduceIdleTime(uint(2000000)),
)
if err != nil {
log.Fatal(err)
}
handler := &Proxy{
Sam: sam,
}
// tutorial line 245
handler.Client = handler.NewClient()
// tutorial line 252
log.Println("Starting proxy server on", *addr)
if err := http.ListenAndServe(*addr, handler); err != nil {
log.Fatal("ListenAndServe:", err)
}
}

175
docs/index.html Normal file
View File

@ -0,0 +1,175 @@
<h1 id="how-to-write-an-http-proxy-for-i2p-in-go">How to Write an HTTP Proxy for I2P in Go</h1>
<p>In this short guide, I will show you how to create a standalone HTTP proxy which will only ever make connections over the I2P network as an introduction to I2P application development in Go. Although most of the readers will have an understanding of HTTP proxies already, and there is already an HTTP proxy to I2P in the core I2P software, there is a method to this madness:</p>
<h2 id="why-another-go-http-proxy-tutorial">Why another Go HTTP Proxy Tutorial?</h2>
<p>There are tons of tutorials out there to write HTTP proxies in Go(1). There is a ton of example code out there you can just copy-and-paste, modify to suit your needs, and use. So why another one now?</p>
<p>Well one reason there are so many is that Go makes it very easy to write a reasonably reliable HTTP proxy. It's a simple project that shows why Go can be good at helping you do the things you need, if what you need is an HTTP client or server. Likewise, the HTTP proxy makes a good introduction to why I2P can be good at helping you do the things you need, if what you need to do is make HTTP requests in privacy. So it exists to bridge the mental gap between the Go application and I2P client in a way which is accessible to people who learned Go from the internet like me.</p>
<p><em>warning:</em> Be careful and do not use this example code unmodified in your real application. It deliberately references an earlier state of the application in the root of this repository. This is a teaching tool and not production code.</p>
<h2 id="what-is-an-http-proxy">What is an HTTP Proxy?</h2>
<p>At it's simplest it's two things, it's an HTTP server, and an HTTP Client. When the HTTP server recieves a request, it handles it by forwarding it to the HTTP client, which then retrieves and returns it to the server, which forwards it back to the requestor. In-between recieving the request and forwarding it to the client, the server can make changes to how the requests are handled. So in our example, we'll be leaving the example HTTP server in place, but modifying the HTTP client to handle requests by routing them to I2P.</p>
<h2 id="step-one-setting-up-the-http-server-structure">Step One: Setting up the HTTP Server Structure</h2>
<p>Creating a custom HTTP server in Go is easy. We just need to create a type called Proxy, which implements the http.Server that forwards you to the client. First, import the goSam library.</p>
<pre><code> import (
&quot;github.com/eyedeekay/goSam&quot;
)
</code></pre>
<p>goSam implements a transport which is compatible with Go's http.Client. The proxy will then need to contain a goSam.Client and an http.Client.</p>
<pre><code> type Proxy struct {
Sam *goSam.Client
Client *http.Client
}
</code></pre>
<p>In order to implement the http Server interface, we need the ServeHTTP function within the Proxy structure, which has the following signature:</p>
<pre><code> func (p *Proxy) ServeHTTP(wr http.ResponseWriter, req *http.Request)
</code></pre>
<p>Where req is the request made by the user, and wr is the the stream where the response will be written and returned to the user.</p>
<p>When you create an instance of this struct, you should make sure to first set up an instance of goSam.Client and pass it to the struct, like this:</p>
<pre><code> sam, err := goSam.NewClientFromOptions(
goSam.SetUnpublished(true),
)
if err != nil {
log.Fatal(err)
}
handler := &amp;i2phttpproxy.Proxy{
Sam: sam,
}
</code></pre>
<p>Note that goSam has been set up to use an Unpublished leaseset in this case, because it will be used as a client and not a service and doesn't need to be discovered until it starts communicating with a service.</p>
<h2 id="step-two-anonymize-the-application">Step Two: Anonymize the application</h2>
<p>If you're developing a privacy-aware application, it's important to anonymize more than just the connection, but also the application as well. In the case of an HTTP proxy, you should probably at least scrub some headers out. To do this, make a slice with the headers you want to delete at the first hop, and a function which deletes them from the underlying structure.</p>
<pre><code> var hopHeaders = []string{
&quot;Accept-Language&quot;,
&quot;Connection&quot;,
&quot;Keep-Alive&quot;,
&quot;Proxy-Authenticate&quot;,
&quot;Proxy-Authorization&quot;,
&quot;Proxy-Connection&quot;,
&quot;Trailers&quot;,
&quot;Upgrade&quot;,
&quot;X-Forwarded-For&quot;,
&quot;X-Real-IP&quot;,
}
func delHopHeaders(header http.Header) {
for _, h := range hopHeaders {
header.Del(h)
}
....
</code></pre>
<p>If you want, you can also re-write the user agent string in this function as well.</p>
<pre><code> ....
if header.Get(&quot;User-Agent&quot;) != &quot;MYOB/6.66 (AN/ON)&quot; {
header.Set(&quot;User-Agent&quot;, &quot;MYOB/6.66 (AN/ON)&quot;)
}
}
</code></pre>
<p>Because our proxy will only be used for I2P addresses, we'll want ServeHTTP to ignore non-I2P addresses. We can do this by examining and dropping the request when the server recieves it, like this:</p>
<pre><code> func (p *Proxy) ServeHTTP(wr http.ResponseWriter, req *http.Request) {
if req.URL.Scheme != &quot;http&quot; &amp;&amp; !strings.HasSuffix(req.URL.Host, &quot;.i2p&quot;) {
msg := &quot;unsupported protocal scheme &quot; + req.URL.Scheme
http.Error(wr, msg, http.StatusBadRequest)
log.Println(msg)
return
}
...
</code></pre>
<p>Once you're done, add the header-scrubbing function to the ServeHTTP function.</p>
<pre><code> ...
delHopHeaders(req.Header)
...
</code></pre>
<h2 id="step-three-actually-handle-the-request">Step Three: Actually handle the request</h2>
<p>Now that we've got the application anonymized, it's time to handle the request.</p>
<pre><code> ...
p.get(wr, req)
}
</code></pre>
<p>To keep the steps of the procedure cleanly separated, I chose to contain this in it's own function. It just carries the same signature of the ServeHTTP function, which</p>
<pre><code> func (p *Proxy) get(wr http.ResponseWriter, req *http.Request) {
req.RequestURI = &quot;&quot;
resp, err := p.Client.Do(req)
if err != nil {
log.Println(&quot;ServeHTTP:&quot;, err)
return
}
defer resp.Body.Close()
wr.WriteHeader(resp.StatusCode)
io.Copy(wr, resp.Body)
}
</code></pre>
<p>But wait! Where's all the I2P stuff? What will this do if it's not connected to I2P? It will just be an HTTP Proxy that doesn't work, because it's already blacklisting non-I2P addresses.</p>
<h2 id="step-four-set-up-the-i2p-client">Step Four: Set up the I2P Client</h2>
<p>Setting up the I2P Client is really easy, but a good example would be a little long so I decided to extract it out to it's own function and walk through the steps. In the end, we need to come up with a new http.Client which uses i2p as a Transport. In order to ensure that it uses the same settings as the goSam client, the function is part of the Proxy struct.</p>
<pre><code> func (p *Proxy) NewClient() *http.Client {
return &amp;http.Client{
</code></pre>
<p>In order to use the resulting http.Client with i2p, you'll need to alter it's DialContext function to use the one from goSam instead.</p>
<pre><code> Transport: &amp;http.Transport{
DialContext: p.Sam.DialContext,
....
</code></pre>
<p>It may also be helpful to limit connections, set some timeouts(Go by default expects you to set timeouts). Some example settings you might change would be:</p>
<pre><code> ....
MaxConnsPerHost: 1,
MaxIdleConns: 0,
MaxIdleConnsPerHost: 1,
DisableKeepAlives: false,
ResponseHeaderTimeout: time.Second * 600,
IdleConnTimeout: time.Second * 300,
},
CheckRedirect: nil,
Timeout: time.Second * 600,
}
}
</code></pre>
<h2 id="step-five-create-the-main-function">Step Five: Create the main() function</h2>
<p>Now it's finally time to put it all together and make our code runnable. First, lets make it accept a flag which allows the user to determine which port it runs on. I don't know what ports other people use:</p>
<pre><code> func main() {
var addr = flag.String(&quot;addr&quot;, &quot;127.0.0.1:7950&quot;, &quot;The addr of the application.&quot;)
flag.Parse()
...
</code></pre>
<p>Next, set up the goSam client and hand it off to a new instance of *Proxy:</p>
<pre><code> ...
sam, err := goSam.NewClientFromOptions(
goSam.SetHost(&quot;127.0.0.1&quot;),
goSam.SetPort(&quot;7656&quot;),
goSam.SetUnpublished(true),
goSam.SetInLength(uint(2)),
goSam.SetOutLength(uint(2)),
goSam.SetInQuantity(uint(1)),
goSam.SetOutQuantity(uint(1)),
goSam.SetInBackups(uint(1)),
goSam.SetOutBackups(uint(1)),
goSam.SetReduceIdle(true),
goSam.SetReduceIdleTime(uint(2000000)),
)
if err != nil {
log.Fatal(err)
}
handler := &amp;Proxy{
Sam: sam,
}
...
</code></pre>
<p>Use that SAM connection to set up an HTTP client:</p>
<pre><code> ...
handler.Client = handler.NewClient()
...
</code></pre>
<p>And serve it up. It's just that easy.</p>
<pre><code> ...
log.Println(&quot;Starting proxy server on&quot;, *addr)
if err := http.ListenAndServe(*addr, handler); err != nil {
log.Fatal(&quot;ListenAndServe:&quot;, err)
}
}
</code></pre>
<p>:)</p>
<h1 id="creditsnotes">Credits/Notes:</h1>
<ol>
<li><a href="https://medium.com/@mlowicki/http-s-proxy-in-golang-in-less-than-100-lines-of-code-6a51c2f2c38c">This guide is based on a guide for the clearnet</a> and that is only 10% because it was easier for me. 90% of the reason is to illustrate that <em>using I2P in your application is, in all likelihood, not that</em> <em>different than what you're already doing.</em>* In all, this example is only 111 lines of code long minus comments.</li>
</ol>
<ul>
<li>if the rest of your application only sends and recieves that which is non-linkable and absolutely necessary to its operation.</li>
</ul>