Skynet is a framework for distributed services in Go.

Overview

logo

##Introduction Skynet is a communication protocol for building massively distributed apps in Go. It is not constrained to Go, so it will lend itself nicely to polyglot environments. The first planned language addition is Ruby.

Skynet is currently undergoing a large refactoring. What's represented here used to exist under the skynet2 repo, and leverages zookeeper. We are currently refactoring to use the new SkyDNS and HTTP/JSON. This readme as well as extensive documentation will be released when the refactoring is completed.

##Open Source - MIT Software License Copyright (c) 2013 Brian Ketelsen

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Comments
  • Dashboard

    Dashboard

    Refactoring of InstanceMonitor to use a single notification channel and a callback method to keep the exposed interface from having to know anything about the InstanceListener object, and monitoring many channels.

    Also refactored instancesocket.go to support cleaning up after timeouts, and stale web sockets that failed to read/write, this refactoring also includes modifications so that it uses the new InstanceMonitor implementation

    Several changes to the logging inside dashboard that were merged from andrey's branch

    opened by erikstmartin 16
  • Pluggable configuration storage

    Pluggable configuration storage

    Doozer is primary and makes most sense - but we should support some other options - via interface, selectable at generation time. Redis is a good choice because it supports push on changes. Others should be supported by implementing the interface.

    opened by bketelsen 13
  • MongoLogger as a skynet service

    MongoLogger as a skynet service

    to remove hard mongodb dependency from skynet, and to use skynet to keep the logger up itself, let's make the mongologger a wrapper to a skynet client, that sends it's requests through skynet rpc to a distributed mongo logger service that does the actual interaction with mongodb

    opened by erikstmartin 12
  • Allow skydaemon to take an optional flag for which mongodb server to connect to

    Allow skydaemon to take an optional flag for which mongodb server to connect to

    This is my first pull request so feel free to let me know what I've done wrong :) I tried to keep the code duplication down by splitting GetServiceConfigFromFlags into two.

    I was having issues with the default Vagrantfile setup with all the non-main VMs trying to connect to a mongo on localhost that was hardcoded---the mongo server is only running on one of the 5 vms.

    cheers, leo

    opened by lfranchi 8
  • Generator

    Generator

    Change the generator to be interactive. Instead of a bunch of ugly and hard to remember command line flags, let's use an interactive generator that builds up enough information to make a skynet mesh. This might be the highest bang for the buck right now.

    opened by bketelsen 7
  • check GOBIN first

    check GOBIN first

    According to http://golang.org/cmd/go/#GOPATH_environment_variable :

    If the GOBIN environment variable is set, commands are installed to the directory it names instead of DIR/bin.

    The command sky deploy should check the environment variable GOBIN first.

    opened by mikespook 6
  • mango initiator example

    mango initiator example

    I added a quick mango initiator example. It's basically quick copy/paste hack-job of the other one. Figured it was the least I could do to contribute. :)

    opened by paulbellamy 6
  • Install fails as Doozer can't compile

    Install fails as Doozer can't compile

    On executing the following (as listed in the wiki)

    go get github.com/4ad/doozer go get github.com/4ad/doozerd

    Go throws an error stating

    go/src/pkg/github.com/4ad/doozer/msg.pb.go:10: undefined: proto.GetString

    Without Doozer, I can't run SkyNet :-(

    opened by Lazarus404 5
  • Routing methodologies

    Routing methodologies

    Right now, Skynet chooses a simple route randomly. May be very useful to choose routes other ways... perhaps by using a data center metric that gives affinity to closer nodes? More thought required on this.

    opened by bketelsen 5
  • skynet can't send non-small messages

    skynet can't send non-small messages

    I believe I've found an issue that shows up when trying to send megabyte-scale messages in Skynet.

    Skynet version: 2cf329fcc Environment: OSX Lion 10.8.2 64-bit Intel Go: go1.0.3

    The minimum case is:

    1. create a service which accepts some payload containing an array of bytes
    2. create a client which makes an RPC call sending a payloads of around 6MB of binary data
    3. the service will receive and process the payload correctly, but a following message will be sent to the service with a bogus (and in most cases, gigabyte) packet size

    I modified the bsoncoder.go file to print out the byte-shifted data length, and it is getting a phantom second message with an very large packet size:

    skynet: 2013/02/02 20:42:21 trace: Service "Art" registered
    BSON LENGTH = 5
    req = &{ServiceMethod: Seq:0 next:<nil>}
    BSON LENGTH = 49
    BSON LENGTH = 6656246
    req = &{ServiceMethod: Seq:0 next:<nil>}
    skynet: 2013/02/02 20:42:26 trace: Method "Put" called with RequestInfo &{127.0.0.1:49159 127.0.0.1:49159 c96ea8d1-187d-4b87-9d40-9247e675cb20 0}
    BSON LENGTH = 3242877185
    panic: runtime error: makeslice: len out of range
    

    The 6656246 value is the valid RPC call; the next BSON message being decoded is the spurious call. I also looked at the BSON length array; the 4th array element is 255 which gets left-bit-shifted by 24 bits and which creats this erroneous packet. I don't know where this message is coming from; the client is making a single RPC call.

    I have a (possibly) minimum sample program, which I will attach to this ticket. Instructions for use:

    1. Create a binary data file called bigfile.bin; I did this with: dd if=/dev/random of=bigfile.bin bs=1024 count=6500
    2. Compile the client: go build -o client.bin ./client
    3. Compile the service: go build -o artd.bin ./artd
    4. Ensure that Skynet is up and running (on localhost)
    5. Start the service: ./artd.bin
    6. Execute the client: ./client.bin

    The service will crash. So will the client, but that's because of a panic being thrown when the service connection errors out.

    Source code for minimum case demonstrating the problem can be fetched from Google Drive: http://goo.gl/sorXf

    opened by ghost 4
  • Logging

    Logging

    Create interface for remote logging, and single implementation using mongodb capped collection. Interface should allow any implementation, first impl will be MongoDB capped collection.

    Build Log viewer with searching into web UI.

    opened by bketelsen 4
  • about wiki

    about wiki "Create a simple service"

    skynet service hadn't GetServiceConfig function, it change to NewServiceInfo.

    func main() {
        tutorial := &TutorialService{}
        config := skynet.NewServiceInfo("TutorialService", "1")
        config.Region = "Development"
        service := service.CreateService(tutorial, config)
        defer func() {
            service.Shutdown()
        }()
        waiter := service.Start()
        waiter.Wait()
    }
    

    However, I found the exception , panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x0 pc=0x4f0a9e]

    opened by zweite 2
  • Looking to learn skynet

    Looking to learn skynet

    I want to really start looking at skynet; (should I be using skynet2).

    however I cannot see any clear docs on Getting Started; (Either golang or the Ruby)

    In a nut shell I want to run on my local machine create some Services and see what I can come up with.

    what are the easiest coolest steps to get this going? I think this would really helpful for people that want to get to grip oon this beast

    Thanks

    opened by hookercookerman 4
  • Add custom sky commands to forward to services?

    Add custom sky commands to forward to services?

    Add functionality so we could possibly do "sky command FOO" and because it's not a recognized command it would be forwarded along to the service.

    This would allow implementors to leverage sky for custom things they may want to enable or disable etc.

    opened by erikstmartin 0
  • Smarter service versioning

    Smarter service versioning

    Rather than a direct string comparison we should investigate doing something like

    https://github.com/jm/go-semver

    so that we can have more loose version requirements

    opened by erikstmartin 0
Owner
null
A distributed, proof of stake blockchain designed for the financial services industry.

Provenance Blockchain Provenance is a distributed, proof of stake blockchain designed for the financial services industry.

Provenance Blockchain, Inc. 64 Nov 6, 2022
This library contains utilities that are useful for building distributed services.

Grafana Dskit This library contains utilities that are useful for building distributed services. Current state This library is still in development. D

Grafana Labs 258 Nov 22, 2022
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Sergey Shepelev 25 Dec 24, 2019
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

etcd-io 41.9k Nov 27, 2022
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 19.7k Nov 26, 2022
a dynamic configuration framework used in distributed system

go-archaius This is a light weight configuration management framework which helps to manage configurations in distributed system The main objective of

null 203 Nov 1, 2022
Go Micro is a standalone framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 19.7k Nov 29, 2022
Tarmac is a unique framework designed for the next generation of distributed systems

Framework for building distributed services with Web Assembly

Benjamin Cane 225 Nov 25, 2022
A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

Tal Derei 4 Nov 4, 2022
AppsFlyer 500 Dec 1, 2022
A Go library for master-less peer-to-peer autodiscovery and RPC between HTTP services

sleuth sleuth is a Go library that provides master-less peer-to-peer autodiscovery and RPC between HTTP services that reside on the same network. It w

null 360 Nov 17, 2022
An experimental library for building clustered services in Go

Donut is a library for building clustered applications in Go. Example package main import ( "context" "log" "os" // Wait for etcd client v3.4, t

David Forsythe 97 Nov 17, 2022
distributed data sync with operational transformation/transforms

DOT The DOT project is a blend of operational transformation, CmRDT, persistent/immutable datastructures and reactive stream processing. This is an im

DOT & Chain 74 Nov 9, 2022
High performance, distributed and low latency publish-subscribe platform.

Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto

emitter 3.4k Nov 25, 2022
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Chris Lu 3.1k Nov 27, 2022
Simplified distributed locking implementation using Redis

redislock Simplified distributed locking implementation using Redis. For more information, please see examples. Examples import ( "fmt" "time"

Black Square Media 858 Nov 22, 2022
A distributed lock service in Go using etcd

locker A distributed lock service client for etcd. What? Why? A distributed lock service is somewhat self-explanatory. Locking (mutexes) as a service

James Gregory 49 Sep 27, 2022
Go Open Source, Distributed, Simple and efficient Search Engine

Go Open Source, Distributed, Simple and efficient full text search engine.

ego 6.1k Nov 28, 2022