A serverless cluster computing system for the Go programming language

Overview

Bigslice

Bigslice is a serverless cluster data processing system for Go. Bigslice exposes composable API that lets the user express data processing tasks in terms of a series of data transformations that invoke user code. The Bigslice runtime then transparently parallelizes and distributes the work, using the Bigmachine library to create an ad hoc cluster on a cloud provider.

Developing Bigslice

Bigslice uses Go modules to capture its dependencies; no tooling other than the base Go install is required.

$ git clone https://github.com/grailbio/bigslice
$ cd bigslice
$ GO111MODULE=on go test

If tests fail with socket: too many open files errors, try increasing the maximum number of open files.

$ ulimit -n 2000
Comments
  • exec: add some form of log.Flush.Sync mechanism

    exec: add some form of log.Flush.Sync mechanism

    As called out in PR #21, it appears that the last few lines of log output can be lost when an error is reported to the master. The exact reason is not clear, but adding a sleep worker.Init helps. Investigate this and provide a more robust mechanism to ensure that log lines are not lost.

    opened by cosnicolaou 15
  • Filter and counts, maybe consider mapreduce counters

    Filter and counts, maybe consider mapreduce counters

    I've attempted to use Filter in two real world examples now but for one I had compromise on my stats reporting and for the other I had to back out of using it. I need to report the # of original inputs and the # post filtering which doesn't seem to be easily possible. I can use Scan or some other side-effect support operation, but then I need to make the side effect work across multiple machines which is annoying and expensive for something as simple as a count. Google's mapreduce offered counters which could be used for this task as this; this mechanism though seems overly general and was extensively abused so I'm not necessarily advocating for it. Given bigslice's ability to carry results through the graph via the slices themselves, maybe it makes sense to add stats to the core slice structure to report on progress through the graph - the simplest being # invocations per operation, but more detailed/extensive ones also being possible.

    opened by cosnicolaou 12
  • Checksum mismatch on github.com/grailbio/testutil@v0.0.1

    Checksum mismatch on github.com/grailbio/[email protected]

    Steps to reproduce:

    $ git clone https://github.com/grailbio/bigslice $ cd bigslice $ go test

    Output of go test::

    go: downloading github.com/grailbio/testutil v0.0.1
    verifying github.com/grailbio/[email protected]: checksum mismatch
    	downloaded: h1:s6IeIZsZHQZXcUnmEKqz22cSn05QsTH5AwHnrxMRKEs=
    	go.sum:     h1:RzGxJO5krJooQGu7pOOgA7RdrwF9L+PTGEIuO3O/M0g=
    
    SECURITY ERROR
    This download does NOT match an earlier download recorded in go.sum.
    The bits may have been replaced on the origin server, or an attacker may
    have intercepted the download attempt.
    
    For more information, see 'go help module-auth'.
    

    Go version:

    go version go1.13 linux/amd64
    

    go env:

    GO111MODULE=""
    GOARCH="amd64"
    GOBIN=""
    GOCACHE="/home/ps/.cache/go-build"
    GOENV="/home/ps/.config/go/env"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="linux"
    GOPATH="/home/ps/go"
    GOPRIVATE=""
    GOPROXY="https://proxy.golang.org,direct"
    GOROOT="/usr/local/go"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
    GCCGO="gccgo"
    AR="ar"
    CC="gcc"
    CXX="g++"
    CGO_ENABLED="1"
    GOMOD="/home/ps/Code/bigslice/go.mod"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build504681712=/tmp/go-build -gno-record-gcc-switches"
    
    opened by psampaz 12
  • Track dependencies correctly given task loss

    Track dependencies correctly given task loss

    Track dependencies correctly given task loss. Fixes the following scenario:

    1. Task A depends on B and C. Nothing has yet been evaluated. A's dependency count is 2.
    2. B is successfully evaluated. A's dependency count goes to 1.
    3. C is successfully evaluated. B is concurrently lost. A's dependency count goes to 0.
    4. A is enqueued, sees that B is lost. Because it is still in the dependency set, A's count is not updated.
    5. B completes. A's count goes to -1, so it is not enqueued.
    6. There is nothing to do, and nothing pending. Evaluation returns without A, the root, ever having been evaluated successfully.

    Fix this by clearing dependency information before reconstructing it in Enqueue.

    Add a stress test that abuses evaluation with lost tasks. The test reliably produces the problem scenario.

    opened by jcharum 10
  • a question on sharding..

    a question on sharding..

    I have a large input set in a dynamodb table that I'd like to dump it to a sharded set of output files that support lookups (grailbio/recordio is the underlying format). I'd like to mod shard the original keys so that given a key I know which file to look the key up in. I have a writerfuncs set up to match the number of input shards and ideally I'd like bigslice to invoke the write func for shard n with all and only the mod-sharded values of my input keys. Is there any way to do this? I don't see any guaranteed way of doing so? My alternative of course is to have each writerfunc be able to write all of the sharded output files, but that seems contrary to the writerfunc model.

    I should say that Reshuffle doesn't appear to work as I would expect it to, which would be drive all keys with the same value to the same shard.

    Thanks!

    opened by cosnicolaou 8
  • cmd/bigslice: extract build and run into a package to allow for reuse.

    cmd/bigslice: extract build and run into a package to allow for reuse.

    This is a protoype for one means of more easily integrating bigslice into other user specific environments.

    This PR extracts the bigslice commands 'build' and 'run' functions into a separate package to allow them to be reused in other user environments. The intent being to allow user sites to customize their setup by providing their own bigslice command, with a site specific setup-ec2 command whilst still reusing the rest of the bigslice commands functionality.

    opened by cosnicolaou 8
  • support for user-defined metrics

    support for user-defined metrics

    This change implements user-defined metrics, following the design proposed in #18.

    It introduces a new package, github.com/grailbio/bigslice/metrics, with which users can declare metrics. These metrics can then be used during evaluation, and inspected upon completion.

    The code anticipates other modes of aggregation: e.g., live sampling, and aggregating by task/phase, etc..

    Example:

    var filtered = metrics.NewCounter()
    
    var myBigsliceFunc = bigslice.Func(func(...) bigslice.Slice {
        slice = ...
        return bigslice.Filter(slice, func(ctx context.Context, data myDataType) bool {
            if keep(data) {
                return true
            }
            scope := metrics.ContextScope(ctx)
            filtered.Incr(scope, 1)
            return false
        })
    })
    
    func main() {
        sess = ...
        res, err := sess.Run(cox, myBigsliceFunc, ...)
        ...
        log.Print("filtered items: ", filtered.Value(res.Scope()))
    }
    
    opened by mariusae 6
  • ReaderFunc for multiple files..

    ReaderFunc for multiple files..

    I would expect a ReaderFunc created with 2 shards, one for each of two files, to read the two files concurrently. However, it seems that they are read sequentially, at least when run in local mode.

    slice = bigslice.ReaderFunc(len(files), NewReaderFunc(files))
    ....
    

    where NewReaderFunc looks like:

    func NewReaderFunc(filenames []string) interface{} {
    	type state struct {}
    	return func(shard int, state *state, entities []wikientities.Entity) (n int, err error) {
                    ....
    		if state.Scanner == nil {
    		....
    			fmt.Printf("processing: shard: %v file: %v\n", shard, filenames[shard])
    
    		}
    .....
    

    is this expected?

    opened by cosnicolaou 6
  • Fix go.sum

    Fix go.sum

    fixes #2

    What I did was the following:

    git clone https://github.com/grailbio/bigslice.git
    cd bigslice
    rm go.sum
    go mod tidy
    

    using

    go version

    go version go1.13 linux/amd64

    and

    GOPROXY="https://proxy.golang.org,direct" GOSUMDB="sum.golang.org"

    opened by psampaz 6
  • Provide user-controlled sharding

    Provide user-controlled sharding

    As outlined in #16, it's often useful to extend fine-grained control of sharding to the user. It can be solved by wrapping integers with an identity hash function, but that seems less than ideal. It might be useful to provide this functionality as part of bigslice.Reshuffle.

    opened by mariusae 5
  • Free ephemeral resources used for scanning

    Free ephemeral resources used for scanning

    When we scan results, we get a slice of *openerAtReaders, one for each result task. We read from each reader sequentially. When we are done with a given reader, we retain some of the resources used when reading from it, most notably gob decode buffers. As we scan, we accumulate these defunct buffers, and our memory footprint grows.

    This happens for two reasons:

    1. We pop off the slice of readers to iterate, i.e. q = q[1:]. However, we do not clear the backing array reference to the reader.
    2. When we close the reader, we don't clear the sliceioReader, which in turn holds the gob decoder.

    Fixing either would eliminate the specific scan leak. Fix both, as I think it's the correct behavior.

    opened by jcharum 3
  • build(deps): bump nokogiri from 1.11.2 to 1.13.9 in /docs

    build(deps): bump nokogiri from 1.11.2 to 1.13.9 in /docs

    Bumps nokogiri from 1.11.2 to 1.13.9.

    Release notes

    Sourced from nokogiri's releases.

    1.13.9 / 2022-10-18

    Security

    Dependencies

    • [CRuby] Vendored libxml2 is updated to v2.10.3 from v2.9.14.
    • [CRuby] Vendored libxslt is updated to v1.1.37 from v1.1.35.
    • [CRuby] Vendored zlib is updated from 1.2.12 to 1.2.13. (See LICENSE-DEPENDENCIES.md for details on which packages redistribute this library.)

    Fixed

    • [CRuby] Nokogiri::XML::Namespace objects, when compacted, update their internal struct's reference to the Ruby object wrapper. Previously, with GC compaction enabled, a segmentation fault was possible after compaction was triggered. [#2658] (Thanks, @​eightbitraptor and @​peterzhu2118!)
    • [CRuby] Document#remove_namespaces! now defers freeing the underlying xmlNs struct until the Document is GCed. Previously, maintaining a reference to a Namespace object that was removed in this way could lead to a segfault. [#2658]

    sha256 checksums:

    9b69829561d30c4461ea803baeaf3460e8b145cff7a26ce397119577a4083a02  nokogiri-1.13.9-aarch64-linux.gem
    e76ebb4b7b2e02c72b2d1541289f8b0679fb5984867cf199d89b8ef485764956  nokogiri-1.13.9-arm64-darwin.gem
    15bae7d08bddeaa898d8e3f558723300137c26a2dc2632a1f89c8574c4467165  nokogiri-1.13.9-java.gem
    f6a1dbc7229184357f3129503530af73cc59ceba4932c700a458a561edbe04b9  nokogiri-1.13.9-x64-mingw-ucrt.gem
    36d935d799baa4dc488024f71881ff0bc8b172cecdfc54781169c40ec02cbdb3  nokogiri-1.13.9-x64-mingw32.gem
    ebaf82aa9a11b8fafb67873d19ee48efb565040f04c898cdce8ca0cd53ff1a12  nokogiri-1.13.9-x86-linux.gem
    11789a2a11b28bc028ee111f23311461104d8c4468d5b901ab7536b282504154  nokogiri-1.13.9-x86-mingw32.gem
    01830e1646803ff91c0fe94bc768ff40082c6de8cfa563dafd01b3f7d5f9d795  nokogiri-1.13.9-x86_64-darwin.gem
    8e93b8adec22958013799c8690d81c2cdf8a90b6f6e8150ab22e11895844d781  nokogiri-1.13.9-x86_64-linux.gem
    96f37c1baf0234d3ae54c2c89aef7220d4a8a1b03d2675ff7723565b0a095531  nokogiri-1.13.9.gem
    

    1.13.8 / 2022-07-23

    Deprecated

    • XML::Reader#attribute_nodes is deprecated due to incompatibility between libxml2's xmlReader memory semantics and Ruby's garbage collector. Although this method continues to exist for backwards compatibility, it is unsafe to call and may segfault. This method will be removed in a future version of Nokogiri, and callers should use #attribute_hash instead. [#2598]

    Improvements

    • XML::Reader#attribute_hash is a new method to safely retrieve the attributes of a node from XML::Reader. [#2598, #2599]

    Fixed

    ... (truncated)

    Changelog

    Sourced from nokogiri's changelog.

    1.13.9 / 2022-10-18

    Security

    Dependencies

    • [CRuby] Vendored libxml2 is updated to v2.10.3 from v2.9.14.
    • [CRuby] Vendored libxslt is updated to v1.1.37 from v1.1.35.
    • [CRuby] Vendored zlib is updated from 1.2.12 to 1.2.13. (See LICENSE-DEPENDENCIES.md for details on which packages redistribute this library.)

    Fixed

    • [CRuby] Nokogiri::XML::Namespace objects, when compacted, update their internal struct's reference to the Ruby object wrapper. Previously, with GC compaction enabled, a segmentation fault was possible after compaction was triggered. [#2658] (Thanks, @​eightbitraptor and @​peterzhu2118!)
    • [CRuby] Document#remove_namespaces! now defers freeing the underlying xmlNs struct until the Document is GCed. Previously, maintaining a reference to a Namespace object that was removed in this way could lead to a segfault. [#2658]

    1.13.8 / 2022-07-23

    Deprecated

    • XML::Reader#attribute_nodes is deprecated due to incompatibility between libxml2's xmlReader memory semantics and Ruby's garbage collector. Although this method continues to exist for backwards compatibility, it is unsafe to call and may segfault. This method will be removed in a future version of Nokogiri, and callers should use #attribute_hash instead. [#2598]

    Improvements

    • XML::Reader#attribute_hash is a new method to safely retrieve the attributes of a node from XML::Reader. [#2598, #2599]

    Fixed

    • [CRuby] Calling XML::Reader#attributes is now safe to call. In Nokogiri <= 1.13.7 this method may segfault. [#2598, #2599]

    1.13.7 / 2022-07-12

    Fixed

    XML::Node objects, when compacted, update their internal struct's reference to the Ruby object wrapper. Previously, with GC compaction enabled, a segmentation fault was possible after compaction was triggered. [#2578] (Thanks, @​eightbitraptor!)

    1.13.6 / 2022-05-08

    Security

    • [CRuby] Address CVE-2022-29181, improper handling of unexpected data types, related to untrusted inputs to the SAX parsers. See GHSA-xh29-r2w5-wx8m for more information.

    ... (truncated)

    Commits
    • 897759c version bump to v1.13.9
    • aeb1ac3 doc: update CHANGELOG
    • c663e49 Merge pull request #2671 from sparklemotion/flavorjones-update-zlib-1.2.13_v1...
    • 212e07d ext: hack to cross-compile zlib v1.2.13 on darwin
    • 76dbc8c dep: update zlib to v1.2.13
    • 24e3a9c doc: update CHANGELOG
    • 4db3b4d Merge pull request #2668 from sparklemotion/flavorjones-namespace-scopes-comp...
    • 73d73d6 fix: Document#remove_namespaces! use-after-free bug
    • 5f58b34 fix: namespace nodes behave properly when compacted
    • b08a858 test: repro namespace_scopes compaction issue
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies ruby 
    opened by dependabot[bot] 0
  • build(deps): bump addressable from 2.7.0 to 2.8.1 in /docs

    build(deps): bump addressable from 2.7.0 to 2.8.1 in /docs

    Bumps addressable from 2.7.0 to 2.8.1.

    Changelog

    Sourced from addressable's changelog.

    Addressable 2.8.1

    • refactor Addressable::URI.normalize_path to address linter offenses (#430)
    • remove redundant colon in Addressable::URI::CharacterClasses::AUTHORITY regex (#438)
    • update gemspec to reflect supported Ruby versions (#466, #464, #463)
    • compatibility w/ public_suffix 5.x (#466, #465, #460)
    • fixes "invalid byte sequence in UTF-8" exception when unencoding URLs containing non UTF-8 characters (#459)
    • Ractor compatibility (#449)
    • use the whole string instead of a single line for template match (#431)
    • force UTF-8 encoding only if needed (#341)

    #460: sporkmonger/addressable#460 #463: sporkmonger/addressable#463 #464: sporkmonger/addressable#464 #465: sporkmonger/addressable#465 #466: sporkmonger/addressable#466

    Addressable 2.8.0

    • fixes ReDoS vulnerability in Addressable::Template#match
    • no longer replaces + with spaces in queries for non-http(s) schemes
    • fixed encoding ipv6 literals
    • the :compacted flag for normalized_query now dedupes parameters
    • fix broken escape_component alias
    • dropping support for Ruby 2.0 and 2.1
    • adding Ruby 3.0 compatibility for development tasks
    • drop support for rack-mount and remove Addressable::Template#generate
    • performance improvements
    • switch CI/CD to GitHub Actions
    Commits
    • 8657465 Update version, gemspec, and CHANGELOG for 2.8.1 (#474)
    • 4fc5bb6 CI: remove Ubuntu 18.04 job (#473)
    • 860fede Force UTF-8 encoding only if needed (#341)
    • 99810af Merge pull request #431 from ojab/ct-_do_not_parse_multiline_strings
    • 7ce0f48 Merge branch 'main' into ct-_do_not_parse_multiline_strings
    • 7ecf751 Merge pull request #449 from okeeblow/freeze_concatenated_strings
    • 41f12dd Merge branch 'main' into freeze_concatenated_strings
    • 068f673 Merge pull request #459 from jarthod/iso-encoding-problem
    • b4c9882 Merge branch 'main' into iso-encoding-problem
    • 08d27e8 Merge pull request #471 from sporkmonger/sporkmonger-enable-codeql
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies ruby 
    opened by dependabot[bot] 0
  • build(deps): bump tzinfo from 1.2.9 to 1.2.10 in /docs

    build(deps): bump tzinfo from 1.2.9 to 1.2.10 in /docs

    Bumps tzinfo from 1.2.9 to 1.2.10.

    Release notes

    Sourced from tzinfo's releases.

    v1.2.10

    TZInfo v1.2.10 on RubyGems.org

    Changelog

    Sourced from tzinfo's changelog.

    Version 1.2.10 - 19-Jul-2022

    Commits
    • 0814dcd Fix the release date.
    • fd05e2a Preparing v1.2.10.
    • b98c32e Merge branch 'fix-directory-traversal-1.2' into 1.2
    • ac3ee68 Remove unnecessary escaping of + within regex character classes.
    • 9d49bf9 Fix relative path loading tests.
    • 394c381 Remove private_constant for consistency and compatibility.
    • 5e9f990 Exclude Arch Linux's SECURITY file from the time zone index.
    • 17fc9e1 Workaround for 'Permission denied - NUL' errors with JRuby on Windows.
    • 6bd7a51 Update copyright years.
    • 9905ca9 Fix directory traversal in Timezone.get when using Ruby data source
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies ruby 
    opened by dependabot[bot] 0
  • Fail typechecking for functions passed to `bigslice.Func` that take `func` and channel arguments

    Fail typechecking for functions passed to `bigslice.Func` that take `func` and channel arguments

    Right now, these aren't caught until we try to gob-encode. Consider failing faster in type-checking to avoid too much confusion/loss when it works with local execution.

    good first issue 
    opened by jcharum 1
  • Proposal (WIP): PushReader, a simpler reader API

    Proposal (WIP): PushReader, a simpler reader API

    This is a concept for making data reading simpler than ReaderFunc by allowing "normal" Go state (in a closure).

    slice := bigslice.PushReader(Nshard, func(shard int, push func(string, int)) error {
    	fuzzer := fuzz.NewWithSeed(1)
    	var row struct {
    		string
    		int
    	}
    	for i := 0; i < N; i++ {
    		fuzzer.Fuzz(&row)
    		push(row.string, row.int)
    	}
    	return nil
    })
    

    The performance cost of this may be significant; I haven't measured yet. I wanted to start by having a concrete example of what user code will look like.

    opened by josh-newman 8
Owner
GRAIL
Source code created or maintained by GRAIL, Inc.
GRAIL
Edge Orchestration project is to implement distributed computing between Docker Container enabled devices.

Edge Orchestration Introduction The main purpose of Edge Orchestration project is to implement distributed computing between Docker Container enabled

null 1 Dec 17, 2021
kubetnl tunnels TCP connections from within a Kubernetes cluster to a cluster-external endpoint, e.g. to your local machine. (the perfect complement to kubectl port-forward)

kubetnl kubetnl (kube tunnel) is a command line utility to tunnel TCP connections from within a Kubernetes to a cluster-external endpoint, e.g. to you

null 4 Sep 18, 2022
A pod scaler golang app that can scale replicas either inside of cluster or out of the cluster

pod-scaler A simple pod scaler golang application that can scale replicas via manipulating the deployment Technologies The project has been created us

Mert Doğan 0 Oct 24, 2021
cluster-api-state-metrics (CASM) is a service that listens to the Kubernetes API server and generates metrics about the state of custom resource objects related of Kubernetes Cluster API.

Overview cluster-api-state-metrics (CASM) is a service that listens to the Kubernetes API server and generates metrics about the state of custom resou

Daimler Group 61 Oct 27, 2022
Go-gke-pulumi - A simple example that deploys a GKE cluster and an application to the cluster using pulumi

This example deploys a Google Cloud Platform (GCP) Google Kubernetes Engine (GKE) cluster and an application to it

Snigdha Sambit Aryakumar 1 Jan 25, 2022
Influxdb-cluster - InfluxDB Cluster for replacing InfluxDB Enterprise

InfluxDB ATTENTION: Around January 11th, 2019, master on this repository will be

Shiwen Cheng 500 Nov 17, 2022
A Terraform module to manage cluster authentication (aws-auth) for an Elastic Kubernetes (EKS) cluster on AWS.

Archive Notice The terraform-aws-modules/eks/aws v.18.20.0 release has brought back support aws-auth configmap! For this reason, I highly encourage us

Aidan Melen 27 Sep 14, 2022
The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability

metrics-agent The metrics-agent collects allocation metrics from a Kubernetes cluster system and sends the metrics to cloudability to help you gain vi

null 0 Jan 14, 2022
Continuous Delivery for Declarative Kubernetes, Serverless and Infrastructure Applications

Continuous Delivery for Declarative Kubernetes, Serverless and Infrastructure Applications Explore PipeCD docs » Overview PipeCD provides a unified co

PipeCD 634 Nov 28, 2022
Putting serverless on your server

Matterless: putting serverless on your server Serverless computing enables you to build applications that automatically scale with demand, and your wa

Zef Hemel 26 Sep 12, 2022
Kubernetes Native Serverless Framework

kubeless is a Kubernetes-native serverless framework that lets you deploy small bits of code without having to worry about the underlying infrastructu

Kubeless 6.9k Nov 24, 2022
Serviço de consulta de CEP Serverless usando Lambda function em Golang

Consulta CEP Serverless Consulta CEP foi desenvolvido com o objetivo de facilitar a vida do desenvolvedor que precisa de um serviço de consulta de CEP

Otavio Baldan 1 Oct 26, 2021
GCP Serverless API With Golang

GCP SERVERLESS API TECH STACK API Gateway Golang Google Cloud Firestore (Native Mode) Google Cloud Functions Google Cloud Storage LOCAL SETUP git clon

Emmanuel Onyekaba 2 Nov 8, 2021
Koyeb is a developer-friendly serverless platform to deploy apps globally.

Koyeb Serverless Platform Deploy a Go Gin application on Koyeb Learn more about Koyeb · Explore the documentation · Discover our tutorials About Koyeb

Koyeb 4 Nov 14, 2022
FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Compute (USENIX ATC'21)

FaaSNet FaaSNet is the first system that provides an end-to-end, integrated solution for FaaS-optimized container runtime provisioning. FaaSNet uses l

LeapLab @ CS_GMU 39 Nov 17, 2022
Go serverless functions examples with most popular Cloud Providers

go-serverless Go serverless functions examples with most popular Cloud Providers Creating zip archive go mod download go build ./cmd/<aws|gcp> zip -

Vlad Morzhanov 0 Nov 16, 2021
Docker-NodeJS - Creating a CI/CD Environment for Serverless Containers on Google Cloud Run

Creating a CI/CD Environment for Serverless Containers on Google Cloud Run Archi

David 1 Jan 8, 2022
Deploy 2 golang aws lambda functions using serverless framework.

Deploy 2 golang aws lambda functions using serverless framework.

NguyenTheAnh 0 Jan 20, 2022
Build powerful pipelines in any programming language.

Gaia is an open source automation platform which makes it easy and fun to build powerful pipelines in any programming language. Based on HashiCorp's g

Gaia 4.9k Nov 18, 2022