Next generation distributed, event-driven, parallel config management!

Overview

mgmt: next generation config management!

mgmt!

Go Report Card Build Status GoDoc IRC Patreon Liberapay

About:

Mgmt is a real-time automation tool. It is familiar to existing configuration management software, but is drastically more powerful as it can allow you to build real-time, closed-loop feedback systems, in a very safe way, and with a surprisingly small amout of our mcl code. For example, the following code will ensure that your file server is set to read-only when it's friday.

import "datetime"
$is_friday = datetime.weekday(datetime.now()) == "friday"
file "/srv/files/" {
	state => $const.res.file.state.exists,
	mode => if $is_friday { # this updates the mode, the instant it changes!
		"0550"
	} else {
		"0770"
	},
}

It can run continuously, intermittently, or on-demand, and in the first case, it will guarantee that your system is always in the desired state for that instant! In this mode it can run as a decentralized cluster of agents across your network, each exchanging information with the others in real-time, to respond to your changing needs. For example, if you want to ensure that some resource runs on a maximum of two hosts in your cluster, you can specify that as well:

import "sys"
import "world"

# we'll set a few scheduling options:
$opts = struct{strategy => "rr", max => 2, ttl => 10,}

# schedule in a particular namespace with options:
$set = world.schedule("xsched", $opts)

if sys.hostname() in $set {
	# use your imagination to put something more complex right here...
	print "i got scheduled" {} # this will run on the chosen machines
}

As you add and remove hosts from the cluster, the real-time schedule function will dynamically pick up to two hosts from the available pool. These specific functions aren't intrinsic to the core design, and new ones can be easily added.

Please read on if you'd like to learn more...

Community:

Come join us in the mgmt community!

Medium Link
IRC #mgmtconfig on Freenode
Twitter @mgmtconfig & #mgmtconfig
Mailing list [email protected]
Patreon purpleidea on Patreon
Liberapay purpleidea on Liberapay

Status:

Mgmt is a next generation automation tool. It has similarities to other tools in the configuration management space, but has a fast, modern, distributed systems approach. The project contains an engine and a language. Please have a look at an introductory video or blog post.

Mgmt is a fairly new project. It is usable today, but not yet feature complete. With your help you'll be able to influence our design and get us to 1.0 sooner! Interested users should read the quick start guide.

Documentation:

Please read, enjoy and help improve our documentation!

Documentation Additional Notes
quick start guide for everyone
frequently asked questions for everyone
general documentation for everyone
language guide for everyone
function guide for mgmt developers
resource guide for mgmt developers
style guide for mgmt developers
godoc API reference for mgmt developers
prometheus guide for everyone
puppet guide for puppet sysadmins
development for mgmt developers

Questions:

Please ask in the community! If you have a well phrased question that might benefit others, consider asking it by sending a patch to the FAQ section. I'll merge your question, and a patch with the answer!

Get involved:

Feel free to grab one of the straightforward #mgmtlove issues if you're a first time contributor to the project or if you're unsure about what to hack on! Please get involved by working on one of these items or by suggesting something else! There are some lower priority issues and harder issues available in our TODO file. Please have a look.

Bugs:

Please set the DEBUG constant in main.go to true, and post the logs when you report the issue. Feel free to read my article on debugging golang programs.

Patches:

We'd love to have your patches! Please send them by email, or as a pull request.

On the web:

Read what people are saying and publishing about mgmt!

Happy hacking!

Issues
  • Declarative DSL

    Declarative DSL

    Consider a DSL that is similar to make with explicitly set defaults to map to fqdn's (via import/whatever). This way targets could be handled by an external program of some sort that 'schedules' hosts (e.g host with least latency to $someplace start http server).

    design 
    opened by nwmcsween 49
  • legal: Re-license to GPLv3

    legal: Re-license to GPLv3

    null

    opened by purpleidea 35
  • file resource needs improvements

    file resource needs improvements

    The file resource can (currently) only set file contents for a single file. It can't make directories or do anything fancy yet. That's okay, because the remaining parts are left as an introductory patch for a new golang hacker. Here's the design, people working on this code are welcome to suggest alternatives:

    All the work should probably happen in Apply() and StateOK(): https://github.com/purpleidea/mgmt/blob/master/file.go#L339 NOTE: i'm considering changing the API to merge the two functions-- comments welcome. It might be preferable to delay this patch if you're not willing to rebase your work. Such is a pre 1.x release.

    The resource offers up a list of file properties, and does "the right thing": $name - if ends in / then it's a directory, otherwise it's a file $content - for files, this is a string with the contents of the file $content - for directories, this is the path to an existing directory, the structure/contents of which we'll want to copy in $content - for directories, if this is an empty string, it means an empty directory $recursive - if resource would require changing more than a single thing, this must be true, otherwise error. we'll have to reconsider if this is necessary or not. only applies to directories. $force - if resource would require a switch from file to directory or vice versa, then force must be true. otherwise it's an error. it can probably be thought of as the -f flag on rm.

    Lastly (and can be a separate patch) the Watch() function will need support for watching recursively into the directories. Adding fanotify support ( https://github.com/go-fsnotify/fsnotify/issues/114 ) would also be welcome in addition.

    Comments welcome by anyone who is going to hack on this!

    mgmtlove 
    opened by purpleidea 32
  • File: implement file attributes

    File: implement file attributes

    Add owner which must be username or uid of the file owner, group which is the group name or gid of the file, and mode which is the octal unix file permissions.

    opened by mildred 23
  • Allow mgmt to run from Puppet manifests

    Allow mgmt to run from Puppet manifests

    Still on that, but no usable results yet. I had used my first PoC Ruby script to reproduce issue #4.

    My initial approach of using the output of puppet master --compile cannot work. Not only does it not include actual dependency edges, it's also much too raw. I will blog about this with details.

    What I will try to do instead is to rely on the agent side graph generating code, to let Puppet help with validation, munging and the computation of actual edges.

    opened by ffrank 23
  • Ctrl+C doesn't quit mgmt

    Ctrl+C doesn't quit mgmt

    [email protected] ~/s/p/mgmt (master|✔)> ./mgmt run -f examples/graph1b.yaml 
    14:50:16 main.go:65: This is: mgmt, version: 0.0.1-70-g3cf8c4a
    14:50:16 main.go:66: Main: Start: 1454421016718582027
    14:50:16 main.go:196: Main: Running...
    14:50:16 main.go:106: Etcd: Starting...
    14:50:16 etcd.go:132: Etcd: Watching...
    14:50:16 configwatch.go:54: Watching: examples/graph1b.yaml
    14:50:16 main.go:149: Graph: Vertices(2), Edges(1)
    14:50:16 main.go:152: Graphviz: No filename given!
    14:50:16 main.go:163: State: graphNil -> graphStarting
    14:50:16 etcd.go:159: Etcd: Waiting 1000 ms for connection...
    14:50:16 main.go:165: State: graphStarting -> graphStarted
    14:50:16 file.go:340: File[file2]: Apply
    14:50:16 file.go:340: File[file3]: Apply
    14:50:17 etcd.go:132: Etcd: Watching...
    14:50:17 etcd.go:159: Etcd: Waiting 2000 ms for connection...
    14:50:19 etcd.go:132: Etcd: Watching...
    14:50:19 etcd.go:159: Etcd: Waiting 4000 ms for connection...
    14:50:23 etcd.go:132: Etcd: Watching...
    14:50:23 etcd.go:159: Etcd: Waiting 8000 ms for connection...
    14:50:31 etcd.go:132: Etcd: Watching...
    14:50:31 etcd.go:159: Etcd: Waiting 16000 ms for connection...
    ^C14:50:46 main.go:51: Interrupted by ^C
    14:50:47 etcd.go:132: Etcd: Watching...
    14:50:47 etcd.go:159: Etcd: Waiting 16000 ms for connection...
    ^C14:51:03 etcd.go:132: Etcd: Watching...
    14:51:03 etcd.go:159: Etcd: Waiting 16000 ms for connection...
    
    bug 
    opened by igalic 22
  • [LOVE] Our GAPI needs a tweak

    [LOVE] Our GAPI needs a tweak

    This is intended for a new mgmt contributor or for someone with very few patches. If nobody claims this, but you are really itching to write this, that's okay too!

    In mgmt we have a GAPI (graph API) which any frontend can implement to generate graphs that the mgmt engine will run. Currently we have a raw yaml graph input, and a puppet code input. When the mgmt language is built, it will use this API as well.

    (Links to code are at an arbitrary git commit so that they don't break, but obviously you'll want to do this on git master and some things might have moved slightly if we've patches those files. Ping me if you can't find something.)

    Here is the interface: https://github.com/purpleidea/mgmt/blob/73ccbb69ea1d74ab5fd96b2096d5902313588a59/gapi/gapi.go#L44

    The Next() function spits out events when there are new graphs available. In particular, when new "exported resources" are available (from other machines) we refresh our local graphs.

    That stream of events is currently located here: https://github.com/purpleidea/mgmt/blob/73ccbb69ea1d74ab5fd96b2096d5902313588a59/lib/main.go#L377

    But it should actually be hidden inside the GAPI itself, since not every GAPI might necessarily want to Watch for exported resources.

    So, that function should be moved out of there, and into the GAPI, eg here: https://github.com/purpleidea/mgmt/blob/73ccbb69ea1d74ab5fd96b2096d5902313588a59/yamlgraph/gapi.go#L79

    but to wrap it nicely, we'd like to actual have a generic interface for that watch called ResWatch which should go here:

    https://github.com/purpleidea/mgmt/blob/73ccbb69ea1d74ab5fd96b2096d5902313588a59/gapi/gapi.go#L29

    The implementation of that function will probably USE the WatchAll method, and go in here:

    https://github.com/purpleidea/mgmt/blob/73ccbb69ea1d74ab5fd96b2096d5902313588a59/etcd/world.go#L30

    Hopefully this is clear enough, if you have questions please let me know!

    Happy hacking!

    help wanted mgmtlove 
    opened by purpleidea 21
  • Automatic build

    Automatic build

    Would it be possible to publish automated builds generated by travis on GitHub Releases ?

    https://docs.travis-ci.com/user/deployment/releases/

    This would allow us to download the binary for easy deploy on servers.

    opened by mildred 21
  • pippet resource support

    pippet resource support

    The pippet resource implements faster integration of Puppet resources in mgmt at runtime, by piping synchronization commands to a Puppet process that keeps running alongside mgmt. This avoids huge overhead through launching a Puppet process for each operation on a resource that is delegated to Puppet.

    opened by ffrank 19
  • Set up golangci-lint linter based on gometalinter

    Set up golangci-lint linter based on gometalinter

    Creating a new PR because the previous one had a weird amount of strange commit history which went over my head to squash despite the many attempts taken.

    Test HIGHLY based on test-gometalinter, as it is used to lint the same files, except where golangci-lint is used instead of gometalinter.

    Notes:

    1. When golangci-lint uses the --skip-files flag, it still lints the file, but simply doesn't display the linting results.
    2. The build fails because the revive linter has found issues with certain files. This is new as revive is a new linter being used to replace golint, as it runs more quickly. This will be fixed down the line.
    3. Comments and TODOs from the previous gometalinter test file were kept in case new linters are being used.

    What was done:

    1. Updated make deps script to install golangci-lint
    2. Created golangci-lint script based on test-gometalinter
    3. Updated test script to run test-golangci-lint
    opened by ginywiny 3
  • updated fuzzing

    updated fuzzing

    When we get to newer versions of golang, we might want to update our fuzzing. The new format looks roughly like:

    func FuzzParser(f *testing.F) {
       f.Fuzz(func(t *testing.T, data []byte) {
          original fuzz function body goes here
       })
    }
    
    Then
    s/return/t.Skip()/
    s/panic/t.Fatal/
    
    finally
    mv fuzz.go fuzz_test.go
    
    and import "testing"
    

    From notes taken from twitter. TBD for some future hacker to see if all this works with corpus, etc...

    opened by purpleidea 0
  • Feature: Sensitive/secret (mcl) values should be flagged and never logged/exposed

    Feature: Sensitive/secret (mcl) values should be flagged and never logged/exposed

    mgmt should be able to handle secret values. Things like passwords, API keys and other sensitive data. mcl should probably have some notion of "sensitive"- possibly only strings need this? If there is ever a "binary"/"bytes" type, perhaps that would need it too.

    Functional changes of note that will be required:

    • Avoid printing/logging sensitive values anywhere
    • Define at what level/verbosity of logging secrets may be printed (i.e. a "safe" log level)
    • Inheritance of the "sensitive" flag for derivative values to avoid leaking them through data changes.
    opened by frebib 3
  • we should add the SO_REUSEADDR options to etcd

    we should add the SO_REUSEADDR options to etcd

    This is fixed now I think: https://github.com/etcd-io/etcd/issues/6042 We should add the options in and test.

    help wanted mgmtlove 
    opened by purpleidea 2
  • Automatically catch races

    Automatically catch races

    Someone should try this project out and see if it catches and/or fixes any races:

    https://github.com/system-pclub/GCatch

    If it's successful, then we should add it to our test suite.

    FWIW as of this writing, we do have some races we need to fix, but in parallel I'm curious what this tool says.

    help wanted mgmtlove 
    opened by purpleidea 0
  • lang: funcs: Validates mac address strings

    lang: funcs: Validates mac address strings

    Tips:

    • please read the style guide before submitting your patch: docs/style-guide.md

    • commit message titles must be in the form:

    topic: Capitalized message with no trailing period

    or:

    topic, topic2: Capitalized message with no trailing period

    • golang code must be formatted according to the standard, please run:
    make gofmt		# formats the entire project correctly
    

    or format a single golang file correctly:

    gofmt -w yourcode.go
    
    • please rebase your patch against current git master:
    git checkout master
    git pull origin master
    git checkout your-feature
    git rebase master
    git push your-remote your-feature
    hub pull-request	# or submit with the github web ui
    
    • after a patch review, please ping @purpleidea so we know to re-review:
    # make changes based on reviews...
    git add -p		# add new changes
    git commit --amend	# combine with existing commit
    git push your-remote your-feature -f
    # now ping @purpleidea in the github PR since it doesn't notify us automatically
    

    Thanks for contributing to mgmt and welcome to the team!

    opened by S-ign 1
  • possible key buffer overflow when calling fsnotify/fsnotify on Windows

    possible key buffer overflow when calling fsnotify/fsnotify on Windows

    Versions:

    • mgmt version (eg: mgmt --version): Any

    • operating system/distribution (eg: uname -a): Windows

    • golang version (eg: go version): Any

    Description:

    https://github.com/golang/go/issues/44593 See https://github.com/fsnotify/fsnotify/blob/7f4cf4dd2b522a984eaca51d1ccee54101d3414a/windows.go#L381

    opened by AZ-X 0
  • lang: funcs: New simple funcs

    lang: funcs: New simple funcs

    Tips:

    • please read the style guide before submitting your patch: docs/style-guide.md

    • commit message titles must be in the form:

    topic: Capitalized message with no trailing period

    or:

    topic, topic2: Capitalized message with no trailing period

    • golang code must be formatted according to the standard, please run:
    make gofmt		# formats the entire project correctly
    

    or format a single golang file correctly:

    gofmt -w yourcode.go
    
    • please rebase your patch against current git master:
    git checkout master
    git pull origin master
    git checkout your-feature
    git rebase master
    git push your-remote your-feature
    hub pull-request	# or submit with the github web ui
    
    • after a patch review, please ping @purpleidea so we know to re-review:
    # make changes based on reviews...
    git add -p		# add new changes
    git commit --amend	# combine with existing commit
    git push your-remote your-feature -f
    # now ping @purpleidea in the github PR since it doesn't notify us automatically
    

    Thanks for contributing to mgmt and welcome to the team!

    opened by S-ign 2
  • test: Fix commit testing

    test: Fix commit testing

    Currently, the testing of commits is in CI, and is CI dependent.

    This is a problem if we switch to a new CI, or if someone wants to test locally.

    This fixes it by making sure the test/test-commit-message can be triggered locally, and finds the new commits. As a side benefit, test/test-commit-message easier to understand: It compares the HEAD of the branch until reaching a common ancestor in origin, listing all the SHAs, and send those to the testing functions.

    Closes: https://github.com/purpleidea/mgmt/issues/643

    opened by evrardjp 2
  • Properly test commit messages on branch push

    Properly test commit messages on branch push

    Description:

    We should test commit messages on PRs but also on branches pushes. Here it's not doing it right.

    opened by evrardjp 1
Releases(0.0.21)
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload.

Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload. Run tools like masscan, puredns, ffuf, httpx or anything you need and get results quickly!

null 97 Sep 22, 2021
An edge-native container management system for edge computing

SuperEdge is an open source container management system for edge computing to manage compute resources and container applications in multiple edge regions. These resources and applications, in the current approach, are managed as one single Kubernetes cluster. A native Kubernetes cluster can be easily converted to a SuperEdge cluster.

SuperEdge 597 Sep 23, 2021
AWS SDK for the Go programming language.

AWS SDK for Go aws-sdk-go is the official AWS SDK for the Go programming language. Checkout our release notes for information about the latest bug fix

Amazon Web Services 7.1k Sep 20, 2021
Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Bytedance Inc. 798 Sep 16, 2021
The Cloud Posse Terraform Provider for various utilities (E.g. deep merging)

terraform-provider-utils Terraform provider to add additional missing functionality to Terraform This project is part of our comprehensive "SweetOps"

Cloud Posse 38 Aug 18, 2021
AWS Lambda to work around index.html S3/CloudFront mess

No more index.html mess with AWS CloudFront/S3 Problem Consider you have a statically generated site — a bunch of usual resources, including html file

Artyom Pervukhin 28 Sep 17, 2021
Application Server for Go (ASG)

Application Server for GO (ASG) Version: 2.0.0 - Using Object-mode API. ASG is application server for Golang, which makes it possible to process distr

Mavimax, Ltd 30 Jul 29, 2021
☁️🏃 Get up and running with Go on Google Cloud.

Get up and running with Go and gRPC on Google Cloud Platform, with this lightweight, opinionated, batteries-included service SDK.

Einride 5 Sep 7, 2021
TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage.

TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage. By leveraging smart contracts, client-side e

TurtleDev 18 Feb 17, 2021
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is an open-source POSIX file system built on top of Redis and object storage (e.g. Amazon S3), designed and optimized for cloud native environ

Juicedata, Inc 3.7k Sep 26, 2021
Lightweight Cloud Instance Contextualizer

Flamingo Flamingo is a lightweight contextualization tool that aims to handle initialization of cloud instances. It is meant to be a replacement for c

Tamer Tas 36 Apr 8, 2021
Cloud cost estimates for Terraform in your CLI and pull requests 💰📉

Infracost shows cloud cost estimates for Terraform projects. It helps developers, devops and others to quickly see the cost breakdown and compare different options upfront.

Infracost 4.1k Sep 24, 2021
Sinclair Microdrive emulator for Spectrum & QL

OqtaDrive emulates a bank of up to 8 Microdrives for use with a Sinclair Spectrum or QL machine.

Alexander Vollschwitz 15 Sep 16, 2021
Production-Grade Container Scheduling and Management

Kubernetes (K8s) Kubernetes, also known as K8s, is an open source system for managing containerized applications across multiple hosts. It provides ba

Kubernetes 81.2k Sep 23, 2021
Go language interface to Swift / Openstack Object Storage / Rackspace cloud files (golang)

Swift This package provides an easy to use library for interfacing with Swift / Openstack Object Storage / Rackspace cloud files from the Go Language

Nick Craig-Wood 271 Sep 6, 2021
cloud-native local storage management system

Open-Local是由多个组件构成的本地磁盘管理系统,目标是解决当前 Kubernetes 本地存储能力缺失问题。通过Open-Local,使用本地存储会像集中式存储一样简单。

null 63 Sep 15, 2021
Generate a basic IAM policy from AWS client-side monitoring (CSM)

iamlive Generate a basic IAM policy from AWS client-side monitoring (CSM) Installation Pre-built binaries Pre-built binaries for Windows, macOS and Li

Ian Mckay 1.5k Sep 25, 2021
Cloud-native way to provide elastic Jupyter Notebook services on Kubernetes

elastic-jupyter-operator: Elastic Jupyter on Kubernetes Kubernetes 原生的弹性 Jupyter 即服务 介绍 为用户按需提供弹性的 Jupyter Notebook 服务。elastic-jupyter-operator 提供以下特性

TKEStack 41 Sep 18, 2021