Source code for the Djinn CI platform

Overview

Djinn CI

Djinn CI is a continuous integration system that allows for running builds in Docker containers and Linux VMs. Builds can be run on the server, or they can be run offline using the offline runner. Each build is configured via a simple YAML manifest that describes how the build should be run, and what commands should be executed within the build.

Contributing

Before you start contributing read the Architecture document that details the architecture of Djinn CI and how everything fits together.

Building from source

If you cannot get hold of a binary distribution then you can always build Djinn from source. Read through docs/admin/building.md for more information on building from source.

Local development

You can download the djinn-dev image that is used in the build itself, and use this for local development. The image can be downloaded here. You can use the image for local development via qemu, like so,

$ qemu-system-x86_64 -daemonize \
        -enable-kvm \
        -m 8192 \
        -drive file=djinn-dev,media=disk,if=virtio \
        -net nic,model=virtio \
        -smp 2 \
        -net user,hostfwd=tcp::2222-:22,hostfwd=tcp::5432-:5432,hostfwd=tcp::6379-:6379 \
        -display none

once booted you will be able to SSH into it as root on port 2222,

$ ssh -p 2222 [email protected]

the ports for Redis (6379), and PostgreSQL (5432) will also be locally accessible too.

mgrt is used for performing revisions against for Djinn CI. Once the virtual machine is booted, you can run the following mgrt command,

$ mgrt run -c schema -type postgresql -dsn "host=localhost port=5432 dbname=djinn user=djinn password=secret"
$ mgrt run -c perms -type postgresql -dsn "host=localhost port=5432 dbname=djinn user=djinn password=secret"
Comments
  • Feature request - namespace webhooks

    Feature request - namespace webhooks

    Feature overview Webhooks on a per namespace basis. These would be triggered on events that would be of interest to the namespace owner, primarily on the status of builds that are submitted.

    Problem space Adding webhooks would allow users to help integrate Djinn CI with their development flow, and help kick off things on their end when builds are submitted/completed.

    Additional context --

    feature 
    opened by andrewpillar 3
  • Feature request - auto-tag refs on build pushes

    Feature request - auto-tag refs on build pushes

    Feature overview Builds that are submitted to a namespace via a push or pull-request should be automatically tagged with the ref. This ref would be the ref of the repo a commit was pushed to, or the ref of the target repo in a pull-request.

    Problem space It can be handy to organize builds by the branches they were triggered from. Whilst Djinn CI has no concept of tying a build to an individual repo, you can instead tag the builds to categorize them, this however is arduous.

    Additional context This should perhaps be an optional thing that is configured during namespace creation.

    feature 
    opened by andrewpillar 2
  • Provide more graceful handling of orphaned builds

    Provide more graceful handling of orphaned builds

    Version devel +5e91bbe Mon Apr 26 18:47:15 2021 +0100

    Environment Hosted

    Describe the bug Right now, if we deploy a new version of the djinn-worker (which would involve a restart) whilst builds are running this results in builds being re-run. Whilst this isn't exactly a bad thing we want this to be more graceful. Ideally the following should happen when a worker picks up on a build that is being re run:

    • Mark the build as orphaned, or as an automatic re-run
    • Clear down the previous information it has stored on the build (timings, job output, artifact information)
    • Re-run the build as if it were clean

    Reproduction steps Run a build, then restart the djinn-worker.

    Screenshots --

    Logs --

    Additional context --

    bug 
    opened by andrewpillar 2
  • Investigate flaky tests

    Investigate flaky tests

    Version devel +c65c3f5 Sat Apr 24 16:37:30 2021 +0100

    Environment Hosted

    Describe the bug The make.1 stage in the manifest for building Djinn is failing due to the test Test_TagStoreCreate. This has failed twice so far, but passed every other time. This appears to be happening sporadically, needs further investigation.

    Reproduction steps Hard to reproduce due to the sporadic nature.

    Logs See the below snippet from the build output,

    + cd djinn
    + ./make.sh
    go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0
    --- FAIL: Test_TagStoreCreate (0.00s)
        tag_test.go:120: build.(*TagStore).Create - build/tag.go:189: database.Store.Create - database/database.go:451: Query 'INSERT INTO build_tags (build_id, name, user_id) VALUES ($1, $2, $3) RETURNING id', arguments do not match: argument 0 expected [int64 - 3] does not match actual [int64 - 10]
    FAIL
    

    Additional context The builds have been tagged as flaky, and can be seen here: https://djinn-ci.com/n/djinn-ci/djinn?tag=flaky, also listed below:

    • https://djinn-ci.com/b/djinn-ci/5
    • https://djinn-ci.com/b/djinn-ci/7
    bug 
    opened by andrewpillar 2
  • Feature request - Variable masking

    Feature request - Variable masking

    Feature overview

    A masked variable would be one that is not displayed in the output of build logs, and instead masked over with a string such as xxxxxx.

    Problem space

    Some builds may want to make use of API tokens to make a callout to a third-party API during a run. As of now, they can do this by setting these tokens as a variable in either a build manifest, or via the UI, however these secrets would not be protected, and would be displayed in the build logs. It is correct that a user submitting a build could take additional steps to reduce the output of sensitive tokens in the build logs, but it would beneficial if the platform aided here too.

    Additional context

    Variable masking should be supported when creating a variable for use in a build via the UI. When a variable is masked it should be encrypted on the backend, and replaced with a string such as xxxxxx in build logs. Variables that are masked should be of a certain minimum length, this way we can ensure that we don't superfluously strip information from build logs when masking. Once a variable has been masked, its value cannot be viewed by the API, and only be viewed via the UI by the creator of said variable with password verification

    feature 
    opened by andrewpillar 1
  • Feature request - allow image downloads from FTP

    Feature request - allow image downloads from FTP

    Feature overview During custom image creation, there should be an option to download an image via FTP, whereby you specify an FTP URL for Djinn CI to download the image from.

    Problem space QEMU images can be large, this causes upload times to be incredibly long and is further constrained by the user's bandwidth. This doesn't lead to a very pleasurable UX when uploading a custom image whereby you have to sit there for a few minutes before moving on. Allowing the image to be downloaded via an FTP site would allow for this to happen in the background.

    Additional context There should be a sensible timeout for downloading these files, perhaps something like 15 minutes.

    feature 
    opened by andrewpillar 1
  • Support webhooks

    Support webhooks

    This adds the ability to configure webhooks on a per-namespace basis. As of now the following events are emitted,

    • build_submitted
    • build_started
    • build_finished
    • build_tagged
    • collaborator_joined
    • cron
    • images
    • objects
    • variables
    • ssh_keys

    this includes changes to the necessary user and API documentation too.

    opened by andrewpillar 1
  • Feature request - Expand integration tests

    Feature request - Expand integration tests

    Feature overview

    The integration tests should be expanded to better capture some of the flows a user would experience whilst using the Djinn CI platform.

    Problem space

    The current integration tests are fine, but could be better expanded. We should try and construct flows of requests that better represent a user's journey through the platform.

    Additional context

    --

    feature 
    opened by andrewpillar 1
  • Feature request - build SVG badges

    Feature request - build SVG badges

    Feature overview Namespaces should provide a URL that would generate an SVG badge of the latest build submitted to that namespace. This would simply display the build's current status. The URL should look something like /n/<username>/<namespace>/-/latest.svg.

    Problem space This would allow for the badge to be embedded into the readmes of projects that use Djinn.

    Additional context Literally every other CI platform has this.

    feature 
    opened by andrewpillar 1
  • Provide clearer error messages when no driver is configured in the offline runner

    Provide clearer error messages when no driver is configured in the offline runner

    Version djinn devel +b2456a5 Sat Apr 24 16:50:25 2021 +0100 linux/amd64

    Environment --

    Describe the bug Segfault happens when you run the offline runner with an empty driver.conf file,

    [main][~/code/djinn-ci/djinn]$ cat ~/.config/djinn/driver.conf
    [main][~/code/djinn-ci/djinn]$ djinn
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xa0f214]
    
    goroutine 1 [running]:
    main.main()
        /home/andrew/code/djinn-ci/djinn/cmd/djinn/main.go:202 +0x11b4
    

    this should instead just throw up an error complaining about no drivers having been configured.

    Reproduction steps Create an empty driver.conf file and run djinn.

    bug 
    opened by andrewpillar 0
Owner
Djinn CI
A simple CI platform that stays out of your way
Djinn CI
Bubbly is an open-source platform that gives you confidence in your continuous release process.

Bubbly Bubbly - Release Readiness in a Bubble Bubbly emerged from a need that many lean software teams practicing Continuous Integration and Delivery

Valocode 34 Jul 27, 2022
KubeCube is an open source enterprise-level container platform

KubeCube English | 中文文档 KubeCube is an open source enterprise-level container platform that provides enterprises with visualized management of Kuberne

KubeCube IO 301 Sep 19, 2022
TriggerMesh open source event-driven integration platform powered by Kubernetes and Knative.

TriggerMesh open source event-driven integration platform powered by Kubernetes and Knative. TriggerMesh allows you to declaratively define event flows between sources and targets as well as add even filter, splitting and processing using functions.

TriggerMesh 327 Sep 15, 2022
The open source public cloud platform. An AWS alternative for the next generation of developers.

M3O M3O is an open source public cloud platform. We are building an AWS alternative for the next generation of developers. Overview AWS was a first ge

Micro Services 2.2k Sep 26, 2022
🔥 🔥 Open source cloud native security observability platform. Linux, K8s, AWS Fargate and more. 🔥 🔥

CVE-2021-44228 Log4J Vulnerability can be detected at runtime and attack paths can be visualized by ThreatMapper. Live demo of Log4J Vulnerability her

null 2k Sep 21, 2022
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers

Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? How to

null 0 Oct 19, 2021
Open Source runtime tool which help to detect malware code execution and run time mis-configuration change on a kubernetes cluster

Kube-Knark Project Trace your kubernetes runtime !! Kube-Knark is an open source tracer uses pcap & ebpf technology to perform runtime tracing on a de

Chen Keinan 32 Sep 19, 2022
Source code and slides for Kubernetes Community Days - Bangalore.

kcdctl This is the source code for the demo done as part of the talk "Imperative, Declarative and Kubernetes" at the Kubernetes Community Days, Bengal

Madhav Jivrajani 15 Sep 19, 2021
io-hat go board source code

io-hat go Kaip veikia Kaip paruošti Raspberry Pi Šaltiniai io-hat go Kaip veikia Įėjimų statusas publikuojamas periodiškai ir kai pasikeičia reikšmė s

Melitonas 0 Dec 24, 2021
BuildKit - A toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner

BuildKit BuildKit is a toolkit for converting source code to build artifacts in an efficient, expressive and repeatable manner. Key features: Automati

CrazyMax 5 Feb 19, 2022
K8s - A Collection of tools, hands-on walkthroughs with source code

The Ultimate Engineer Toolbox ?? ?? A Collection of tools, hands-on walkthroughs

null 0 Feb 14, 2022
Boxygen is a container as code framework that allows you to build container images from code

Boxygen is a container as code framework that allows you to build container images from code, allowing integration of container image builds into other tooling such as servers or CLI tooling.

nitric 5 Dec 13, 2021
Cross-platform beanstalkd queue server admin console.

Overview aurora is a web-based Beanstalkd queue server console written in Go and works on macOS, Linux, and Windows machines. The main idea behind usi

null 566 Sep 7, 2022
Fast cross-platform HTTP benchmarking tool written in Go

bombardier bombardier is a HTTP(S) benchmarking tool. It is written in Go programming language and uses excellent fasthttp instead of Go's default htt

Максим Федосеев 3.8k Sep 27, 2022
:rocket: Modern cross-platform HTTP load-testing tool written in Go

English | 中文 Cassowary is a modern HTTP/S, intuitive & cross-platform load testing tool built in Go for developers, testers and sysadmins. Cassowary d

Roger Welin 616 Sep 21, 2022
Enterprise-grade application development platform

Erda Overview Feature list Architecture Related repositories erda-proto erda-infra erda-ui Quick start To start using erda To start developing erda Do

Erda 2.4k Sep 24, 2022
The mec platform for service register/discovery/subscribe and other functions.roject main repo.

EdgeGallery MEP project Introduction Edgegallery MEP is an open source implementation of MEC platform according to ETSI MEC 003 [1] and 011 [2] docume

EdgeGallery 36 Jul 5, 2022
Enterprise-grade container platform tailored for multicloud and multi-cluster management

KubeSphere Container Platform What is KubeSphere English | 中文 KubeSphere is a distributed operating system providing cloud native stack with Kubernete

KubeSphere 11k Sep 19, 2022