[deprecated] A full-featured SPDY library for the Go language.

Related tags

Network spdy
Overview

Deprecated

With the release of Go1.6 and the addition of http2 to the standard library, this package is no longer under active development. It is highly recommended that former users of this package migrate to HTTP/2.

spdy

GoDoc

A full-featured SPDY library for the Go language.

Note that this implementation currently supports SPDY drafts 2 and 3.

See these examples for a quick intro to the package.

Note that using this package with Martini is likely to result in strange and hard-to-diagnose bugs. For more information, read this article. As a result, issues that arise when combining the two should be directed at the Martini developers.

Servers

The following examples use features specific to SPDY.

Just the handler is shown.

Use SPDY's pinging features to test the connection:

package main

import (
	"net/http"
	"time"

	"github.com/SlyMarbo/spdy"
)

func Serve(w http.ResponseWriter, r *http.Request) {
	// Ping returns a channel which will send an empty struct.
	if ping, err := spdy.PingClient(w); err == nil {
		select {
		case response := <- ping:
			if response != nil {
				// Connection is fine.
			} else {
				// Something went wrong.
			}
			
		case <-time.After(timeout):
			// Ping took too long.
		}
	} else {
		// Not SPDY
	}
	
	// ...
}

Sending a server push:

package main

import (
	"net/http"
	
	"github.com/SlyMarbo/spdy"
)

func Serve(w http.ResponseWriter, r *http.Request) {
	// Push returns a separate http.ResponseWriter and an error.
	path := r.URL.Scheme + "://" + r.URL.Host + "/example.js"
	push, err := spdy.Push(path)
	if err != nil {
		// Not using SPDY.
	}
	http.ServeFile(push, r, "./content/example.js")

	// Note that a PushStream must be finished manually once
	// all writing has finished.
	push.Finish()
	
	// ...
}
Comments
  • Gzip over SPDY

    Gzip over SPDY

    So I tried adding gzip to a spdy server & on every request I get (spdy) 2014/05/05 10:01:18 spdy3_conn.go:1098: Warning: Received PROTOCOL_ERROR on stream 1. Closing connection..

    tried with a vanilla server & then with martini. There is a high possibility that this is not an issue with your library, but I could not figure out where to ask this question.

    Q: What's the best way to go about gzipping with a spdy server.

    opened by netroy 15
  • (s *clientStreamV3)Read unimplemented

    (s *clientStreamV3)Read unimplemented

    The read method for client streams is currently TODO:

    func (s *clientStreamV3) Read(out []byte) (int, error) {
      // TODO
      return 0, nil
    }
    

    It would be helpful to log here at least until implemented. I'm currently trying to read the results of a Request().

    opened by rnapier 10
  • Clients MUST support gzip

    Clients MUST support gzip

    From the spec – http://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3-1

    “User-agents MUST support gzip compression. Regardless of the Accept-Encoding sent by the user-agent, the server may always send content encoded with gzip or deflate encoding.”

    This change adds transparent decompression of gripped content.

    I found this when I modified client code to use SPDY to talk to an origin server. The origin is using nginx. I was getting errors when trying to unmarshall JSON:

    invalid character '\\x1f' looking for beginning of value
    

    Enabling debug logging, I saw the request:

    Header:               map[Url:[/public/dashboards?slug=carers-allowance] Version:[HTTP/1.1] Host:[example.gov.uk] Scheme:[https] Accept:[application/json] User-Agent:[Performance-Platform-Client/1.0]  Method:[GET]]
    

    and a response:

    map[Status:[200] Content-Type:[application/json] Via:[1.1 varnish] Date:[Thu, 12 Feb 2015 19:52:58 GMT] Cache-Control:[max-age=300] Access-Control-Allow-Origin:[*] X-Frame-Options:[SAMEORIGIN] X-Varnish:[1873677853 1873677852] Age:[0] X-Cache:[HIT] Content-Encoding:[gzip] Version:[HTTP/1.1] Server:[nginx] Alternate-Protocol:[443:npn-spdy/2 443:npn-spdy/2] Strict-Transport-Security:[max-age=2628000]]
    Data:                 [1f 8b 08 00 00 00 00 00 00 ... 05 af e6 9f 47 2a 3c 00 00]
    

    This shows that the response is coming back compressed, even though the client didn’t ask for that.

    opened by jabley 8
  • Using `httputil.NewSingleHostReverseProxy` results in malformed HTTP headers

    Using `httputil.NewSingleHostReverseProxy` results in malformed HTTP headers

    I'm working on a simple SPDY reverse proxy, and I ran into this bug. The following code sends valid HTTP headers to the reverse-proxied host:

    targetURL := url.Parse("http://www.google.com")
    reverseProxy := httputil.NewSingleHostReverseProxy(targetURL)
    http.Handle("/", reverseProxy)
    if err = http.ListenAndServeTLS(":5005", "cert.pem", "key.pem", nil); err != nil {
      // handle error.
      log.Fatal(err)
    }
    

    but replacing http.ListenAndServeTLS with spdy.ListenAndServeTLS or spdy.ListenAndServeSPDY produces the following:

    GET / HTTP/1.1
    User-Agent: spdylay/1.2.3
    : host: localhost:5005
    : method: GET
    : path: /
    : scheme: https
    : version: HTTP/1.1
    Accept: */*
    Accept-Encoding: gzip, deflate
    X-Forwarded-For: ::1
    Host: www.google.com
    

    Any ideas on how to fix this?

    opened by keichan34 8
  • Concurrent streams limit

    Concurrent streams limit

    When using the same http.Client and doing concurrent requests to a single domain, I eventually start seeing "Max concurrent streams limit exceeded." errors.

    There doesn't appear to be an option to adjust this limit. Do I need to maintain multiple http.Clients?

    opened by chendo 7
  • Panic after httpClient is not used for a while

    Panic after httpClient is not used for a while

    I'm seeing this panic when trying to use the httpClient after being idle for a while:

    2014/10/07 09:33:51 http: panic serving 127.0.0.1:54822: runtime error: send on closed channel
    goroutine 281 [running]:
    net/http.func·011()
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/server.go:1100 +0xb7
    runtime.panic(0x2d70c0, 0x4b6a9e)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/runtime/panic.c:248 +0x18d
    github.com/SlyMarbo/spdy/spdy3.(*Conn).Request(0xc208094400, 0xc208521110, 0x57e668, 0xc2081f70c0, 0xc208168000, 0x0, 0x0, 0x0, 0x0)
        /Users/chendo/Code/Go/src/github.com/SlyMarbo/spdy/spdy3/requests.go:109 +0xe6c
    github.com/SlyMarbo/spdy/spdy3.(*Conn).RequestResponse(0xc208094400, 0xc208521110, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0)
        /Users/chendo/Code/Go/src/github.com/SlyMarbo/spdy/spdy3/requests.go:131 +0xd3
    github.com/SlyMarbo/spdy.(*Transport).RoundTrip(0xc20804e300, 0xc208521110, 0x73, 0x0, 0x0)
        /Users/chendo/Code/Go/src/github.com/SlyMarbo/spdy/transport.go:310 +0xed4
    net/http.send(0xc208521110, 0x56d6a8, 0xc20804e300, 0xc20853df34, 0x0, 0x0)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/client.go:195 +0x43d
    net/http.(*Client).send(0xc208024b70, 0xc208521110, 0x11, 0x0, 0x0)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/client.go:118 +0x15b
    net/http.(*Client).doFollowingRedirects(0xc208024b70, 0xc208521110, 0x3a7210, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/client.go:343 +0x97f
    net/http.(*Client).Do(0xc208024b70, 0xc208521110, 0x4bcac0, 0x0, 0x0)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/client.go:153 +0x193
    main.handler(0x5702c0, 0xc2085665a0, 0xc208521110)
        /Users/chendo/Code/Go/src/github.com/chendo/spdy-up/main.go:22 +0x1af
    net/http.HandlerFunc.ServeHTTP(0x3a7078, 0x5702c0, 0xc2085665a0, 0xc208521110)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/server.go:1235 +0x40
    net/http.(*ServeMux).ServeHTTP(0xc208024630, 0x5702c0, 0xc2085665a0, 0xc208521110)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/server.go:1511 +0x1a3
    net/http.serverHandler.ServeHTTP(0xc208004360, 0x5702c0, 0xc2085665a0, 0xc208521110)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/server.go:1673 +0x19f
    net/http.(*conn).serve(0xc2085e6400)
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/server.go:1174 +0xa7e
    created by net/http.(*Server).Serve
        /usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/http/server.go:1721 +0x313
    

    I suspect the spdy connection was closed but it wasn't cleaned up properly. I'm thinking of keeping the connection alive with pings to the server but there doesn't appear to be a way to catch this in an err and handle it appropriately which might be an issue.

    bug 
    opened by chendo 7
  • Reconsider spdy/2

    Reconsider spdy/2

    The current approach to spdy/2 creates a lot of duplicated code that is tedious to maintain and bloats the final binary by 300k in my experiements. Nginx, Firefox and Chrome have all dropped support for spdy/2. My preference is just to delete the spdy2 package, which is a simple change.

    opened by rnapier 6
  • Buffer race

    Buffer race

    I've got another one for you. I was debugging a concurrency issue with my app and compiled it with race detection enabled. I'm running into a race warning repeatedly in a handler that reads the post body with a

    buf, err := ioutil.ReadAll(req.Body)

    I'm not spawning any goroutines myself in that handler, and the normal net/http server does not trigger a race warning, so I think this is a library issue.

    edit: An additional note - the race is only be detected the first time I hit that handler, so perhaps it's an initialization issue?

    WARNING: DATA RACE
    Write by goroutine 18:
      bytes.(*Buffer).Read()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/bytes/buffer.go:252 +0x63
      github.com/SlyMarbo/spdy.(*readCloser).Read()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/client_conn.go:1 +0x83
      bytes.(*Buffer).ReadFrom()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/bytes/buffer.go:169 +0x402
      io/ioutil.readAll()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/io/ioutil/ioutil.go:32 +0x1ca
      io/ioutil.ReadAll()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/io/ioutil/ioutil.go:41 +0x4c
      github.com/pabbott0/bellows.postPosts()
          /Users/liver/devel/go/src/github.com/pabbott0/bellows/posts.go:65 +0xad
      runtime.call64()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/asm_amd64.s:340 +0x31
      reflect.Value.Call()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/reflect/value.go:345 +0xaa
      github.com/codegangsta/inject.(*injector).Invoke()
          /Users/liver/devel/go/src/github.com/codegangsta/inject/inject.go:98 +0x394
      github.com/codegangsta/martini.(*context).Invoke()
          /Users/liver/devel/go/src/github.com/codegangsta/martini/env.go:1 +0x67
      github.com/codegangsta/martini.(*routeContext).run()
          /Users/liver/devel/go/src/github.com/codegangsta/martini/router.go:264 +0x13e
      github.com/codegangsta/martini.(*route).Handle()
          /Users/liver/devel/go/src/github.com/codegangsta/martini/router.go:186 +0x10d
      github.com/codegangsta/martini.(*router).Handle()
          /Users/liver/devel/go/src/github.com/codegangsta/martini/router.go:90 +0x25c
      github.com/codegangsta/martini.Router.Handle·fm()
          /Users/liver/devel/go/src/github.com/pabbott0/bellows/bellows.go:67 +0x6d
      runtime.call64()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/asm_amd64.s:340 +0x31
      reflect.Value.Call()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/reflect/value.go:345 +0xaa
      github.com/codegangsta/inject.(*injector).Invoke()
          /Users/liver/devel/go/src/github.com/codegangsta/inject/inject.go:98 +0x394
      github.com/codegangsta/martini.(*context).run()
          /Users/liver/devel/go/src/github.com/codegangsta/martini/martini.go:152 +0x110
      github.com/codegangsta/martini.(*Martini).ServeHTTP()
          /Users/liver/devel/go/src/github.com/codegangsta/martini/martini.go:68 +0x60
      github.com/SlyMarbo/spdy.(*serverStreamV3).Run()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_server_stream.go:239 +0x1b5
    
    Previous write by goroutine 15:
      bytes.(*Buffer).Write()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/bytes/buffer.go:126 +0x57
      github.com/SlyMarbo/spdy.(*serverStreamV3).ReceiveFrame()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_server_stream.go:179 +0x5a4
      github.com/SlyMarbo/spdy.(*connV3).handleClientData()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_conn.go:485 +0x554
      github.com/SlyMarbo/spdy.(*connV3).processFrame()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_conn.go:1182 +0x13ac
      github.com/SlyMarbo/spdy.(*connV3).readFrames()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_conn.go:1229 +0x62a
    
    Goroutine 18 (running) created at:
      github.com/SlyMarbo/spdy.(*connV3).handleRequest()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_conn.go:714 +0xa2d
      github.com/SlyMarbo/spdy.(*connV3).processFrame()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_conn.go:1061 +0x1672
      github.com/SlyMarbo/spdy.(*connV3).readFrames()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_conn.go:1229 +0x62a
    
    Goroutine 15 (running) created at:
      github.com/SlyMarbo/spdy.(*connV3).Run()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/spdy3_conn.go:394 +0x113
      github.com/SlyMarbo/spdy.func·014()
          /Users/liver/devel/go/src/github.com/SlyMarbo/spdy/server_conn.go:391 +0x172
      net/http.(*conn).serve()
          /usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1116 +0x5f1
    
    bug 
    opened by pabbott0 5
  • http.Get(api.linode.com) results in 'Unsupported negotiated protocol' error

    http.Get(api.linode.com) results in 'Unsupported negotiated protocol' error

    First off, thank you for the terrific library. I was able to enable SPDY support in my web service in just a few minutes with it. :smile:

    I see inside client.go's init() http.DefaultClient to enable SPDY support, but when I make calls to api.linode.com I get Get https://api.linode.com: Error: Unsupported negotiated protocol "".

    This gist contains sample code that reproduces the issue. https://gist.github.com/arashpayan/3927d6aae5e8e01aa43d

    In the meantime, I just replace the http.DefaultClient at the beginning of my app with one without SPDY support.

    opened by arashpayan 4
  • Decompress crash

    Decompress crash

    This is a6b3ab986382ecd174a9927dbe8175d08bc2e709 (v6.0-19-ga6b3ab9), which is out of date now, but I wanted to capture it in case this race condition still exists. (If you believe this has been fixed in the latest; go ahead and close. I've already upgraded, but my QA was testing an older version.)

    This was during a somewhat rapid (~1/second) disconnect/reconnect cycle.

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xb code=0x1 addr=0x20 pc=0x55b436]
    
    goroutine 7513 [running]:
    runtime.panic(0x74cfe0, 0x975753)
            /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
    .../src/github.com/SlyMarbo/spdy.(*synStreamFrameV3_1).Decompress(0xc2084d8600, 0x0, 0x0, 0x0, 0x0)
            .../Godeps/_workspace/src/github.com/SlyMarbo/spdy/spdy3_frames.go:294 +0x76
    .../src/github.com/SlyMarbo/spdy.(*connV3).readFrames(0xc208028d00)
            .../Godeps/_workspace/src/github.com/SlyMarbo/spdy/spdy3_conn.go:1256 +0x2aa
    created by .../src/github.com/SlyMarbo/spdy.(*connV3).Run
            .../Godeps/_workspace/src/github.com/SlyMarbo/spdy/spdy3_conn.go:425 +0xb5
    
    goroutine 16 [runnable]:
    net._C2func_getaddrinfo(0x7f45c80008c0, 0x0, 0xc2084d84e0, 0xc20803a0d0, 0x0, 0x0, 0xc2080ac200)
            net/_obj/_cgo_defun.c:52 +0x36
    net.cgoLookupIPCNAME(0x7fff2414a8b8, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
            /usr/local/go/src/pkg/net/cgo_unix.go:96 +0x1e1
    net.cgoLookupIP(0x7fff2414a8b8, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc208001000)
            /usr/local/go/src/pkg/net/cgo_unix.go:148 +0x69
    net.lookupIP(0x7fff2414a8b8, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0)
            /usr/local/go/src/pkg/net/lookup_unix.go:64 +0x66
    net.func·024(0x0, 0x0, 0x0, 0x0)
            /usr/local/go/src/pkg/net/lookup.go:41 +0x54
    net.(*singleflight).Do(0x97b280, 0x7fff2414a8b8, 0x15, 0xc208441530, 0x0, 0x0, 0x0, 0x0, 0x0)
            /usr/local/go/src/pkg/net/singleflight.go:45 +0x232
    net.lookupIPMerge(0x7fff2414a8b8, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0)
            /usr/local/go/src/pkg/net/lookup.go:42 +0xae
    net.lookupIPDeadline(0x7fff2414a8b8, 0x15, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
            /usr/local/go/src/pkg/net/lookup.go:57 +0x12b
    net.resolveInternetAddr(0x79d8b0, 0x3, 0x7fff2414a8b8, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
            /usr/local/go/src/pkg/net/ipsock.go:285 +0x49a
    net.resolveAddr(0x798950, 0x4, 0x79d8b0, 0x3, 0x7fff2414a8b8, 0x19, 0x0, 0x0, 0x0, 0x0, ...)
            /usr/local/go/src/pkg/net/dial.go:110 +0x385
    net.(*Dialer).Dial(0xc20804e280, 0x79d8b0, 0x3, 0x7fff2414a8b8, 0x19, 0x0, 0x0, 0x0, 0x0)
            /usr/local/go/src/pkg/net/dial.go:159 +0xff
    crypto/tls.DialWithDialer(0xc20804e280, 0x79d8b0, 0x3, 0x7fff2414a8b8, 0x19, 0xc208068000, 0x21, 0x0, 0x0)
            /usr/local/go/src/pkg/crypto/tls/tls.go:115 +0x1d1
    crypto/tls.Dial(0x79d8b0, 0x3, 0x7fff2414a8b8, 0x19, 0xc208068000, 0xa, 0x0, 0x0)
            /usr/local/go/src/pkg/crypto/tls/tls.go:165 +0x83
    .../src/github.com/SlyMarbo/spdy.Connect(0x7fff2414a8b0, 0x21, 0xc208068000, 0xc2084bc180, 0x0, 0x0, 0x0, 0x0)
            .../Godeps/_workspace/src/github.com/SlyMarbo/spdy/proxy.go:34 +0x202
    ....(*ServerHandler).connect(0xc20803e780, 0xc208068000, 0x0, 0x0, 0x0, 0x0)
            .../serverhandler.go:138 +0x142
    ....(*ServerHandler).Run(0xc20803e780, 0x0, 0x0)
            .../serverhandler.go:61 +0x481
    main.main()
            ...:73 +0x324
    
    goroutine 19 [finalizer wait, 13 minutes]:
    runtime.park(0x413a00, 0x97a230, 0x9784c9)
            /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
    runtime.parkunlock(0x97a230, 0x9784c9)
            /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
    runfinq()
            /usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
    runtime.goexit()
            /usr/local/go/src/pkg/runtime/proc.c:1445
    
    goroutine 17 [syscall, 1317 minutes]:
    runtime.goexit()
            /usr/local/go/src/pkg/runtime/proc.c:1445
    
    goroutine 7503 [chan receive, 7 minutes]:
    .../src/github.com/SlyMarbo/spdy.(*serverStreamV3).Run(0xc2083c2750, 0x0, 0x0)
            .../Godeps/_workspace/src/github.com/SlyMarbo/spdy/spdy3_server_stream.go:248 +0xd9
    created by .../src/github.com/SlyMarbo/spdy.(*connV3).handleRequest
            .../Godeps/_workspace/src/github.com/SlyMarbo/spdy/spdy3_conn.go:745 +0x7fb
    
    opened by rnapier 4
  • [Enhancement] http.Flusher

    [Enhancement] http.Flusher

    (I'm just putting it here at the Issues-tab, because I'm not 100% sure it's even possible with the SPDY-implementation. )

    The following code works with the http.ListenAndServeTLS, but not with spdy.ListenAndServeTLS:

    func handle(w http.ResponseWriter, r *http.Request) {
        if f, ok := w.(http.Flusher); ok {
            f.Flush()
            // Flushed to client - http package
        } else {
            // Flush not supported - spdy package
        } 
    }
    
    opened by EtienneBruines 3
  • Errors (Received SYN_REPLY with unopened or closed Stream...) during high traffic.

    Errors (Received SYN_REPLY with unopened or closed Stream...) during high traffic.

    Hi,

    I made just a slight modification on the example server and client. The client starts number of concurrent function each keep sending requests at given rate, the server replies with one 1KB of random data.

    At 10x concurrent functions each making 1000 request /s there are errors reported:

    (spdy) 2016/02/01 22:16:58 error_handling.go:18: Error: Received SYN_REPLY with unopened or closed Stream ID 15065. (spdy) 2016/02/01 22:16:58 io.go:19: Encountered receive error: runtime error: invalid memory address or nil pointer dereference (runtime.errorString)

    Not sure, if it is a bug in the protocol implementation or my client/server share some data between concurrent functions that should not be shared.

    Attaching full log and my updated code.

    git log -1 commit 431b911c621782fff951b92948f9ce38633188f0

    (github, didn't accept .zip file so I rename to txt) spdy_go_test.zip.txt

    opened by przemekr 3
  • ResponseStream should NOT wait until the full request has been received

    ResponseStream should NOT wait until the full request has been received

    ResponseStream do not ServeHTTP() until the full request has been received. For a SPDY proxy, the disadvantage of current design is obvious:

    1. unnecessary memory consumption (Post request will not be forwarded to backend server until the full request has been received)
    2. unnecessary process delay (We may do ServeHTTP() if the request header has been received)
    enhancement low priority 
    opened by iyangsj 1
  • Crashes and lockups on network error

    Crashes and lockups on network error

    If there is a network error (corrupted data at a minimum, but I believe I'm also encountering it when servers suddenly become unavailable), there are data races in shutdown(). This can lead to panics or lockup in the client. In some cases, it panics with:

    panic: assignment to entry in nil map
    
    goroutine 1598 [running]:
    github.com/SlyMarbo/spdy/spdy3.(*Conn).Request(0xc823460480, 0xc8228c41c0, 0x3001630, 0xc82347c240, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/rnapier/work/agent/src/github.com/SlyMarbo/spdy/spdy3/requests.go:129 +0x18d9
    github.com/SlyMarbo/spdy/spdy3.(*Conn).RequestResponse(0xc823460480, 0xc8228c41c0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0)
        /Users/rnapier/work/agent/src/github.com/SlyMarbo/spdy/spdy3/requests.go:139 +0x294
    github.com/SlyMarbo/spdy.(*Transport).RoundTrip(0xc8228cf080, 0xc8228c41c0, 0x61e635, 0x0, 0x0)
        /Users/rnapier/work/agent/src/github.com/SlyMarbo/spdy/transport.go:190 +0x539
    net/http.send(0xc8228c41c0, 0x1564738, 0xc8228cf080, 0xf, 0x0, 0x0)
        /usr/local/Cellar/go/1.5/libexec/src/net/http/client.go:220 +0x73e
    net/http.(*Client).send(0xc8232285a0, 0xc8228c41c0, 0xf, 0x0, 0x0)
        /usr/local/Cellar/go/1.5/libexec/src/net/http/client.go:143 +0x1f8
    net/http.(*Client).doFollowingRedirects(0xc8232285a0, 0xc8228c41c0, 0x621a88, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5/libexec/src/net/http/client.go:380 +0x105a
    net/http.(*Client).Get(0xc8232285a0, 0x56b340, 0xf, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5/libexec/src/net/http/client.go:306 +0xc9
    main.main.func2()
        /Users/rnapier/work/agent/src/kace/konea/cmd/kontap/client.go:29 +0x5c
    created by main.main
        /Users/rnapier/work/agent/src/kace/konea/cmd/kontap/client.go:42 +0x142
    

    In other cases, the panic is caught by a recover, but this leads to a deadlock because the send goroutine is no longer processing packets. So when the error handler tries to send a RST, the whole connection locks up. This can lead to further symptoms like Ping() never returning (which is how I discovered this issue originally).

    This block in shutdown seems the primary cause of data races:

    s.output = nil
    s.Request = nil
    s.Receiver = nil
    s.header = nil
    s.stop = nil
    

    Here are a few of the conflicts that popped up in my tests:

      github.com/SlyMarbo/spdy/spdy3.(*Conn).send()
        Attempt 26:   /Users/rnapier/work/agent/src/github.com/SlyMarbo/spdy/spdy3/io.go:129 +0x5cc
    
      github.com/SlyMarbo/spdy/spdy3.(*RequestStream).ReceiveFrame.func2()
          /Users/rnapier/work/agent/src/github.com/SlyMarbo/spdy/spdy3/request_stream.go:193 +0x44
    
      github.com/SlyMarbo/spdy/spdy3.(*Conn).send()
          /Users/rnapier/work/agent/src/github.com/SlyMarbo/spdy/spdy3/io.go:142 +0xac6
    
      github.com/SlyMarbo/spdy/spdy3.(*RequestStream).ReceiveFrame.func1()
          /Users/rnapier/work/agent/src/github.com/SlyMarbo/spdy/spdy3/request_stream.go:183 +0x7a
    

    To demonstrate the problem, I use the following programs:

    • server - A spdy server that returns "SUCCESS"
    • client - A spdy client that connects very rapidly via a proxy and prints the output
    • proxy - An unreliable proxy. It corrupts 1% of packets

    The client and servers generally show data races immediately, and the client will generally crash within a few seconds. If you try to reuse the client for every iteration (which better matches my real use case), the Get call will hang.

    // server.go
    // openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes -batch
    
    package main
    
    import (
        "fmt"
        "log"
        "net/http"
    
        "github.com/SlyMarbo/spdy"
    )
    
    func httpHandler(w http.ResponseWriter, req *http.Request) {
        fmt.Fprintf(w, "SUCCESS")
    }
    
    func main() {
        http.HandleFunc("/", httpHandler)
        log.Printf("About to listen on 10443. Go to https://127.0.0.1:34567/")
        err := spdy.ListenAndServeTLS(":10443", "cert.pem", "key.pem", nil)
        if err != nil {
            log.Fatal(err)
        }
    }
    
    // client.go
    package main
    
    import (
        "fmt"
        "io/ioutil"
        "time"
    
        "github.com/SlyMarbo/spdy" // This adds SPDY support to net/http
    )
    
    func main() {
        n := 0
        client := spdy.NewClient(true) // In this configuration, the Get hangs w/o providing an error
    
        for {
            n++
            go func(iter int) {
                // client := spdy.NewClient(true) // Using a new client every time avoids the hang in Get, making crashes more common
                fmt.Printf("Attempt %d: ", iter)
                res, err := client.Get("https://:34567/") // 10443 for direct
                fmt.Printf("...")
                if err != nil {
                    fmt.Println(err)
                    return
                }
    
                body, err := ioutil.ReadAll(res.Body)
                if err != nil {
                    fmt.Println("ERROR")
                    return
                }
                res.Body.Close()
                fmt.Println(string(body))
            }(n)
            time.Sleep(10 * time.Millisecond)
        }
    }
    
    // proxy.go
    
    package main
    
    import (
        "fmt"
        "io"
        "log"
        "math/rand"
        "net"
        "os"
    )
    
    func main() {
        port := 34567
        dest := 10443
        rate := 0.01
    
        l, err := net.Listen("tcp", fmt.Sprintf(":%d", port))
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
        fmt.Printf("http://%v -> :%d\n", l.Addr(), dest)
    
        for {
            conn, err := l.Accept()
            if err != nil {
                log.Fatal(err)
            }
    
            fmt.Printf("Connection from %s\n", conn.RemoteAddr())
            go proxy(conn, dest, rate)
        }
    }
    
    func proxy(in net.Conn, dest int, rate float64) {
        out, err := net.Dial("tcp", fmt.Sprintf(":%d", dest))
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
    
        go unreliableCopy(out, in, rate)
        unreliableCopy(in, out, rate)
        fmt.Printf("Disconnect from %s\n", in.RemoteAddr())
    }
    
    func unreliableCopy(dst io.Writer, src io.Reader, rate float64) {
        buf := make([]byte, 32*1024)
        for {
            nr, er := src.Read(buf)
            if nr > 0 {
                maybeCorrupt(buf[0:nr], rate)
                nw, ew := dst.Write(buf[0:nr])
                if ew != nil {
                    break
                }
                if nr != nw {
                    break
                }
            }
            if er == io.EOF {
                break
            }
            if er != nil {
                break
            }
        }
    }
    
    func maybeCorrupt(buf []byte, rate float64) {
        if rand.Float64() < rate {
            fmt.Println("Corrupting")
            loc := rand.Intn(len(buf))
            val := byte(rand.Intn(0x100))
            buf[loc] = val
        }
    }
    

    I'm currently seeing this very often in the field, and my infrastructure is locking up usually within a few hours of usage. I have one client, and many servers that come and go, so network errors are reasonably common.

    opened by rnapier 4
  • Is this a bug?

    Is this a bug?

    When I loop sending queries to get a small(this is key,130 Bytes) html document long time,I can get one err like this: (spdy) 2015/07/14 17:52:08 error_handling.go:17: Error: Received PROTOCOL_ERROR on stream 0. Closing connection. RoundTripper returned a response & error; ignoring response

    And if I get two errors above continuous,the client will block cause the DefaultMaxIdleConnsPerHost is 2,and when occur this error,not produce t.connLimit[req.URL.Host].And it will block at dial() function at line 108:<-t.connLimit[u.Host] cause it wanna consume but nobody produce.

    And why the server will response this error?Does some error occured in the client?

    opened by albus01 2
  • ServeHTTP panic does not close stream

    ServeHTTP panic does not close stream

    If a response ServeHTTP panics during RequestResponse, the stream is not closed, causing the requester to hang. The equivalent http method does close the stream:

    If ServeHTTP panics, the server (the caller of ServeHTTP) assumes that the effect of the panic was isolated to the active request. It recovers the panic, logs a stack trace to the server error log, and hangs up the connection.

    I've tried adding a s.shutdown() to the recovery function in ResponseStream.Run(), but that doesn't seem to help. s.conn.Close() is too much (we don't want to kill the whole connection).

    proxy_server.go

    package main
    
    import (
        "bytes"
        "fmt"
        "io"
        "net/http"
    
        "github.com/SlyMarbo/spdy"
    )
    
    func handle(err error) {
        if err != nil {
            panic(err)
        }
    }
    
    func fetch(conn spdy.Conn, path string) {
        fmt.Printf("Fetching %v\n", path)
        url := "http://" + conn.Conn().RemoteAddr().String() + path
    
        req, err := http.NewRequest("GET", url, nil)
        if err != nil {
            println("NEWREQ: " + err.Error())
            return
        }
    
        fmt.Printf("REQRES: %v\n", path)
        res, err := conn.RequestResponse(req, nil, 1)
        if err != nil {
            println("REQRES: " + err.Error())
            return
        }
    
        fmt.Printf("COPY: %v\n", path)
        buf := new(bytes.Buffer)
        _, err = io.Copy(buf, res.Body)
        if err != nil {
            println("COPY: " + err.Error())
            return
        }
    
        res.Body.Close()
    
        fmt.Printf("RECV: %v (%d) %v\n", path, res.StatusCode, buf.String())
    }
    
    func handleProxy(conn spdy.Conn) {
        fetch(conn, "/crash")
        fetch(conn, "/") // Never runs
    
        wait := make(chan interface{})
        <-wait
    }
    
    func main() {
        // spdy.EnableDebugOutput()
        handler := spdy.ProxyConnHandlerFunc(handleProxy)
        http.Handle("/", spdy.ProxyConnections(handler))
        handle(http.ListenAndServeTLS(":8080", "cert.pem", "key.pem", nil))
    }
    

    proxy_client.go

    package main
    
    import (
        "crypto/tls"
        "fmt"
        "net/http"
    
        "github.com/SlyMarbo/spdy"
    )
    
    func handle(err error) {
        if err != nil {
            panic(err)
        }
    }
    
    func serveHTTP(w http.ResponseWriter, r *http.Request) {
        fmt.Printf("Serving %v\n", r.URL.Path)
        if r.URL.Path == "/crash" {
            panic("CRASH")
        }
        w.Write([]byte("Testing, testing, 1, 2, 3."))
    }
    
    func main() {
        // spdy.EnableDebugOutput()
        http.HandleFunc("/", serveHTTP)
        handle(spdy.ConnectAndServe("http://localhost:8080/", &tls.Config{InsecureSkipVerify: true}, nil))
    
        wait := make(chan interface{})
        <-wait
    }
    
    opened by rnapier 1
  • Data is not sent if message is near to or larger than window

    Data is not sent if message is near to or larger than window

    If the size of the data sent by an HTTP Handler is within 15 bytes of the window size, then the data will be sent, but no FIN, which will cause the client to hang.

    If the size of the data sent by an HTTP Handler is greater than the window size, then no data will be sent.

    client.go

    package main
    
    import (
        "fmt"
        "io/ioutil"
    
        "github.com/SlyMarbo/spdy" // This adds SPDY support to net/http
    )
    
    func main() {
        spdy.EnableDebugOutput()
        client := spdy.NewClient(true)
        res, err := client.Get("https://localhost:10443/")
        if err != nil {
            panic(err)
        }
        defer res.Body.Close()
    
        bytes, err := ioutil.ReadAll(res.Body)
        if err != nil {
            panic(err)
        }
    
        fmt.Printf("Received: %s\n", bytes)
    }
    

    server.go

    // openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes -batch
    
    package main
    
    import (
        "bytes"
        "log"
        "net/http"
    
        "github.com/SlyMarbo/spdy"
        "github.com/SlyMarbo/spdy/common"
    )
    
    func httpHandler(w http.ResponseWriter, req *http.Request) {
        w.Header().Set("Content-Type", "text/plain")
        // w.Write(bytes.Repeat([]byte("X"), common.DEFAULT_INITIAL_CLIENT_WINDOW_SIZE-16)) // Works
        w.Write(bytes.Repeat([]byte("X"), common.DEFAULT_INITIAL_CLIENT_WINDOW_SIZE-15)) // Hangs (no FIN is sent)
        // w.Write(bytes.Repeat([]byte("X"), common.DEFAULT_INITIAL_CLIENT_WINDOW_SIZE)) // Sends FIN, but no data
    }
    
    func main() {
        spdy.EnableDebugOutput()
        http.HandleFunc("/", httpHandler)
        log.Printf("About to listen on 10443. Go to https://127.0.0.1:10443/")
        err := spdy.ListenAndServeTLS(":10443", "cert.pem", "key.pem", nil)
        if err != nil {
            log.Fatal(err)
        }
    }
    
    opened by rnapier 2
Releases(v7.0)
Owner
Jamie Hall
Jamie Hall
gqlgenc is a fully featured go gql client, powered by codegen

gqlgenc Note: ⚠️ This is a WIP, backward-compatibility cannot be guaranteed yet, use at your own risk gqlgenc is a fully featured go gql client, power

Infiot Inc 29 Sep 17, 2022
DeSo is a blockchain built from the ground up to support a fully-featured social network

DeSo is a blockchain built from the ground up to support a fully-featured social network. Its architecture is similar to Bitcoin, only it supports complex social network data like profiles, posts, follows, creator coin transactions, and more.

DeSo Protocol 284 Aug 30, 2022
Simple hosts file management in Golang (deprecated).

Goodhosts (deprecated) This library is now deprecated. See the goodhosts organisation for the current maintained version. Simple hosts file (/etc/host

Lex T 69 Mar 17, 2022
Hprose 1.0 for Golang (Deprecated). Hprose 2.0 for Golang is here:

Hprose for Golang Introduction Installation Usage Http Server Http Client Synchronous Invoking Synchronous Exception Handling Asynchronous Invoking As

Hprose 137 Sep 22, 2022
SOCKS Protocol Version 5 Library in Go. Full TCP/UDP and IPv4/IPv6 support

socks5 中文 SOCKS Protocol Version 5 Library. Full TCP/UDP and IPv4/IPv6 support. Goals: KISS, less is more, small API, code is like the original protoc

TxThinking 485 Sep 22, 2022
A C/S Tool to Download Torrent Remotely and Retrieve Files Back Over HTTP at Full Speed without ISP Torrent Limitation.

remote-torrent Download Torrent Remotely and Retrieve Files Over HTTP at Full Speed without ISP Torrent Limitation. This repository is an extension to

Bruce Wang 58 Aug 9, 2022
Inspired by go-socks5,This package provides full functionality of socks5 protocol.

The protocol described here is designed to provide a framework for client-server applications in both the TCP and UDP domains to conveniently and securely use the services of a network firewall.

Zhangliu 69 Aug 15, 2022
Baseledger core consensus for running validator, full and seed nodes

baseledger-core Baseledger core consensus client for running a validator, full or seed node. ⚠️ WARNING: this code has not been audited and is not rea

Baseledger 0 Jan 13, 2022
Server and client implementation of the grpc go libraries to perform unary, client streaming, server streaming and full duplex RPCs from gRPC go introduction

Description This is an implementation of a gRPC client and server that provides route guidance from gRPC Basics: Go tutorial. It demonstrates how to u

Joram Wambugu 0 Nov 24, 2021
proxylogon, proxyshell, proxyoracle full chain exploit tool

Proxy-Attackchain proxylogon, proxyshell, proxyoracle full chain exploit tool ProxyLogon: The most well-known and impactful Exchange exploit chain Pro

lUc1f3r11 192 Sep 19, 2022
Imersão Full Cycle 5

Imersão Full Cycle 5 Dinâmica do sistema Tecnologias Frontend Painel: Next.js Backend Painel: Nest.js Microsserviço processamento: Golang Sistema de m

null 1 Jan 10, 2022
A Go package for creating contributor list by release, Help full for those organization that use one repository for platform release

This is a Go package which create contributors list by release by scanning across all repository that exist in organisation, Only helpful for those or

Yuvraj 0 Dec 26, 2021
Wrapper around bufcli to make it do cross-repo compiles for private repos and use full paths.

Bufme A tool for compiling protos with full directory paths and cross repo compiles. Introduction Protocol buffers rock, but protoc should die in a fi

John Doak 0 Feb 5, 2022
A Gradle Plugin Providing Full Support for Go

Gogradle - a Full-featured Build Tool for Golang 中文文档 Gogradle is a gradle plugin which provides support for building golang. 2017-06-23 Gogradle is a

null 770 Sep 22, 2022
CoAP Client/Server implementing RFC 7252 for the Go Language

Canopus Canopus is a client/server implementation of the Constrained Application Protocol (CoAP) Updates 25.11.2016 I've added basic dTLS Support base

Zubair Hamed 148 Jan 23, 2022
BGP implemented in the Go Programming Language

GoBGP: BGP implementation in Go GoBGP is an open source BGP implementation designed from scratch for modern environment and implemented in a modern pr

null 3k Sep 26, 2022
A simple wrapper around libpcap for the Go programming language

PCAP This is a simple wrapper around libpcap for Go. Originally written by Andreas Krennmair [email protected] and only minorly touched up by Mark Smith

Andreas Krennmair 456 Sep 27, 2022
Go language interface to the Libcircle distributed-queue API

Circle Description The Circle package provides a Go interface to the Libcircle distributed-queue API. Despite the name, Circle has nothing to do with

Los Alamos National Laboratory 24 Nov 1, 2020