Kademlia/Mainline DHT node in Go.

The automatically registered flags clutter and possibly conflict with applications using this package. They also don't necessarily match the wrapping application's argument style (i.e. two-words vs twoWords).

This patch also enables configuration by the wrapping application, previously configuration could only be done by the person running the binary.

It's not clear about possibility to build a DHT router on this library. It seems that this is possible, because in dht_test.go there is a test TestDHTLocal that seems testing exact same case. But it also looks like test itself is incorrect. If add little bit more logging to it, then output will be

n1 started at port 52361
n2 started at port 52604
n3 started at port 57136
Node n3 found a peer 0: 127.0.0.1:57136
Node n2 found a peer 0: 127.0.0.1:52604

It looks like node found itself as a peer instead of finding at least one more node. Do I understand meaning of the test correctly? Is it possible after all to use this library for building DHT router?

This adds the ability to maintain a fresh list of nodes in the peerstore, other additions include:

• if we don't have enough nodes, try to bootstrap from the router
• respect maxnodependingqueries
• if a routingtable.lookupfiltered fails to return nodes, try routingtable.lookup
• don't try to be smarter than the bittorrent client: if it asks for more nodes, forward the request to the dht (this helps if all the nodes we know are stuck)

One of the harder parts of P2P is dealing with IP Masq/NAT'd peers. Fortunately there's an awesome fix, IPv6. Google claims 9.85% of Google's traffic in the USA is from IPv6 users, far from a trivial number. I've managed to get IPv6 connectivity working at 2 locations so I can tinker. Comcast is giving out /60's to regular home users!

I'm going to take a shot at it. I've not written any IPv6 code before though.

The resources I'm referencing are https://github.com/jech/dht and http://www.bittorrent.org/beps/bep_0032.html.

Just figured I'd post here in case anyone else is interested or planning similar.

This is not reliably reproduceable.

$ go test -v
=== RUN TestDHTLocal
Found peer 0: 127.0.0.1:52407
Found peer 0: 127.0.0.1:42450
--- PASS: TestDHTLocal (1.41 seconds)
=== RUN TestDHTLarge
panic: routingTable.insert() got a node with a nil address

goroutine 41 [running]:
runtime.panic(0x67d840, 0xc20815e100)
        /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/nictuku/dht.(*routingTable).insert(0xc2080a4ba0, 0xc20815ac00, 0x0, 0x0)
        /home/nictuku/src/github.com/nictuku/dht/routing_table.go:122 +0x391
github.com/nictuku/dht.(*routingTable).getOrCreateNode(0xc2080a4ba0, 0xc208152000, 0x14, 0xc208169fe0, 0x10, 0xc20815ac0
0, 0x0, 0x0)
        /home/nictuku/src/github.com/nictuku/dht/routing_table.go:162 +0x180
github.com/nictuku/dht.(*DHT).processFindNodeResults(0xc2080b2fc0, 0xc2080a2240, 0xc20816962a, 0x1, 0xc20816970a, 0x1, 0
x0, 0x0, 0x0, 0x0, ...)
        /home/nictuku/src/github.com/nictuku/dht/dht.go:908 +0xa86
github.com/nictuku/dht.(*DHT).processPacket(0xc2080b2fc0, 0xc208121000, 0x11c, 0x1000, 0xc208141e88, 0x4, 0x4, 0x1ae1, 0
x0, 0x0)
        /home/nictuku/src/github.com/nictuku/dht/dht.go:518 +0xec7
github.com/nictuku/dht.(*DHT).Run(0xc2080b2fc0, 0x0, 0x0)
        /home/nictuku/src/github.com/nictuku/dht/dht.go:385 +0xca4
created by github.com/nictuku/dht.TestDHTLarge 
        /home/nictuku/src/github.com/nictuku/dht/dht_test.go:178 +0x2cf

goroutine 16 [chan receive]:
testing.RunTests(0x7ff570, 0x91ff60, 0xa, 0xa, 0x1)
        /usr/local/go/src/pkg/testing/testing.go:505 +0x923
testing.Main(0x7ff570, 0x91ff60, 0xa, 0xa, 0x91e720, 0x3, 0x3, 0x91b0a0, 0x1, 0x1)
        /usr/local/go/src/pkg/testing/testing.go:435 +0x84
main.main()
        github.com/nictuku/dht/_test/_testmain.go:73 +0x9c

Open to suggestions on how to improve the code or organize it better welcome.

It's working well for me. Here's a run: $ go run main6.go -v6 [2601:c:a200:1228:76d0:2bff:fe90:8b90] 12b9b08b325e877d08cc63a75e8fd24699c3dc6b Tring to bind to IPv6=[2601:c:a200:1228:76d0:2bff:fe90:8b90] =========================== DHT Note that there are many bad nodes that reply to anything you ask. Peers found: =========================== DHT Note that there are many bad nodes that reply to anything you ask. Peers found: 0: [2505:20f6:19c2:2a:9c01:fe:00ca:dd11]:8445 0: 114.40.158.84:19944 1: 220.135.171.68:11183 2: 125.81.132.91:8978 3: 154.45.216.164:1073 4: 185.6.124.193:49572 1: [2601:c:a200:1228:76d0:2bff:fe90:8b90]:8445

Ref #61

@nictuku This is the smallest interface that I can think of: https://github.com/cenkalti/dht/blob/29206eba2e98f2d7fab5affe38d5655a78b45db7/logging.go#L3-L7

By implementing this interface people can still use dht with glog or any logger they want.

I have named it as Logger2 because there is already an interface named as Logger: https://github.com/cenkalti/dht/blob/29206eba2e98f2d7fab5affe38d5655a78b45db7/dht.go#L223-L227

Of course this name is unacceptable :)

In fact, existing Logger interface does not represent a logger actually. I suggest renaming it as Hooks or something else and make Logger2 -> Logger. However, this is not possible without breaking backwards compatibility. I think it is best to tag the current commit as v1 and include this change in v2.

jackpal/bencode-go currently uses a lot of ressources..

Is it maybe easy to use IncSW/go-bencode instead?

I've tried for a while but couldn't get it into the right format/syntax, to get it to work.

Performance

Marshal

| Library | Time | Bytes Allocated | Objects Allocated | | :--- | :---: | :---: | :---: | | IncSW/go-bencode | 1493 ns/op | 554 B/op | 15 allocs/op | | jackpal/bencode-go | 8497 ns/op | 2289 B/op | 66 allocs/op |

Unmarshal

| Library | Time | Bytes Allocated | Objects Allocated | | :--- | :---: | :---: | :---: | | IncSW/go-bencode | 3151 ns/op | 1360 B/op | 46 allocs/op | | jackpal/bencode-go | 6850 ns/op | 3073 B/op | 102 allocs/op |

following #44, I also applied the modification to the tests & examples, and included the minifix I mentionned about closing the d.portRequest chan

I think there's a small issue regarding the dht.Run() function, it both contains an initialization phase, prone to error (listen fails), and the main loop which I think suggests that it should be run in a go routine, since the only other way out is dht.Stop() from another routine.

It makes the handling of the first error a bit tricky. It also leads to another trickiness : a call to dht.Port() method is prone to hanging if listen fails and we did not check Run()'s return value - at least a close(dht.portRequest) would be necessary I think

I'm not sure about the best way to handle this, or if you wish to change anything at all, ideally I think we should accept the main loop will be running in a routine, and go dht.loop() ourselves in there as a private function if initialization (listen) was successful - loop not giving any error. It would allow the main entry point to return everytime, so we know the dht is running, or not. A downside to this is that it will induce retro-compatibilities issues, since we cannot expect dht.Run() to block - maybe something like an alternate dht.Start(), with depreciation on Run ?

I have recently been using Popcorn Time a lot, and I like the features but dislike the build quality.

So I started thinking about how one could build something similar, but more robust. The first improvement, of course, would be to build it in Go instead of JavaScript.

Another would be to use some other search/browse database than whatever Popcorn Time uses (which isn't very robust at all).

Finally I found https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf, which describes how to seed the network with pure forwarding nodes that only tap the infohashes it sees in those nodes, and then fetch the metadata from the hosting nodes to put them in a local searchable database.

This made me start looking at your code. Unfortunately I get confused by it - I am used to chord-style networks, and kademlia is like greek to me :)

Where should I add code to enable some kind of callback function whenever the node gets queried about an infohash?

https://github.com/vitessio/vitess/blob/master/go/cache/lru_cache.go they have added a cost

type LRUCache struct { mu sync.Mutex // list & table contain *entry objects. list *list.List table map[string]*list.Element cost func(interface{}) int64 size int64 capacity int64 evictions int64 }

When I looked at the code of peerContactsSet, this function seems to be wrong.

func (p *peerContactsSet) next() []string {
	count := kNodes
	if count > len(p.set) {
		count = len(p.set)
	}
	x := make([]string, 0, count)
	xx := make(map[string]bool) //maps are easier to dedupe
	for range p.set {
		nid := p.ring.Move(1).Value.(string)  <----this line will always return the first next item since p.ring never change its pos.
		if _, ok := xx[nid]; p.set[nid] && !ok {
			xx[nid] = true
		}
		if len(xx) >= count {
			break
		}
	}
...

The move function will return a new pointer rather than change the original pointer as demonstrated in this playground.

func (r *Ring) Move(n int) *Ring {
	if r.next == nil {
		return r.init()
	}
	switch {
	case n < 0:
		for ; n < 0; n++ {
			r = r.prev
		}
	case n > 0:
		for ; n > 0; n-- {
			r = r.next
		}
	}
	return r
}

Need a PR to fix it?

// Read from UDP socket, writes slice of byte into channel.
func readFromSocket(socket *net.UDPConn, conChan chan packetType, bytesArena arena, stop chan bool, log DebugLogger) {
	for {
		b := bytesArena.Pop()
		n, addr, err := socket.ReadFromUDP(b)
		if err != nil {
			log.Debugf("DHT: readResponse error:%s\n", err)
		}
		b = b[0:n]
		if n == maxUDPPacketSize {
			log.Debugf("DHT: Warning. Received packet with len >= %d, some data may have been discarded.\n", maxUDPPacketSize)
		}
		totalReadBytes.Add(int64(n))
		if n > 0 && err == nil {
			p := packetType{b, *addr}
			select {
			case conChan <- p:
				continue
			case <-stop:
				return
			}
		}
		// Do a non-blocking read of the stop channel and stop this goroutine if the channel
		// has been closed.
		select {
		case <-stop:
			return
		default:
		}
	}
}

If I get that right you do a bytesArena.Pop() to get a pre-allocated buffer. but in case of a 0 byte package or an error you never return that buffer back, since that happens only for packets in conChan aka socketChan (dht.go:499).

my dht client recently began to randomly stopping to receiving/process any incoming packets until I restart.. so I began digging.. and that might be the reason?

This makes the code a bit better, as it makes these odd indirections (pointer to interface) unnecessary, and fixes one logger call that happens inside New() before caller has any chance to initialize DebugLogger.

Unfortunately, this change is backward incompatible from API standpoint.

• added few more well known DHTRouters

• added optional PassiveMode parameter, so the server doesn't responde to any incoming queries (get_peers, ping, etc)

• added optional MaxSearchQueries parameter to limit the number of generated get_peers queries per hash. If NumTargetPeers is set to ie 5000 the server will infinityfly create new get_peers requests, till NumTargetPeers is reached - which may create a lot of packets and traffic that can't be handled any more. Especial if the hash has only 5 peers out there, the server will keep sending new requests. With MaxSearchQueries set, this will hard limit the number of new get_peers queries per hash. The counter will reset (per hash) after SearchCntExpire (default 10 minutes).

• added an expiration timer for nodes. This will only take effect if MaxNodes limit is reached, and purge old (> CleanupPeriod) nodes without pendingQueries, till MaxNodes is reached again.

• replaced the old Ubuntu Iso fileHash in tests with a current one