Run Jobs on a schedule, supports fixed interval, timely, and cron-expression timers; Instrument your processes and expose metrics for each job.

Overview



🕰 Sched

Go In-Process Scheduler with Cron Expression Support

Run Jobs on a schedule, supports fixed interval, timely, and cron-expression timers; Instrument your processes and expose metrics for each job.

Go Doc Go Version Go Report GitHub license

Introduction

A simple process manager that allows you to specify a Schedule that execute a Job based on a Timer. Schedule manage the state of this job allowing you to start/stop/restart in concurrent safe way. Schedule also instrument this Job and gather metrics and optionally expose them via uber-go/tally scope.

Install

go get github.com/sherifabdlnaby/sched
import "github.com/sherifabdlnaby/sched"

Requirements

Go 1.13 >=


Concepts

Job

Simply a func(){} implementation that is the schedule goal to run, and instrument.

Timer

An Object that Implements the type Timer interface{}. A Timer is responsible for providing a schedule with the next time the job should run and if there will be subsequent runs.

Packaged Implementations:

  1. Fixed :- Infinitely Fires a job at a Fixed Interval (time.Duration)
  2. Cron :- Infinitely Fires a job based on a Cron Expression, all Expressions supported by gorhill/cronexpr are supported.
  3. Once :- A Timer that run ONCE after an optional specific delay or at a specified time, schedule will stop after it fires.

You can Implement your own Timer for your specific scheduling needs by implementing

type Timer interface {
	// done indicated that there will be no more runs.
    Next() (next time.Time, done bool)
}

Schedule

A Schedule wraps a Job and fires it according to Timer.

	fixedTimer30second, _ := sched.NewFixed(30 * time.Second)

	job := func() {
		log.Println("Doing some work...")
		time.Sleep(1 * time.Second)
		log.Println("Finished Work.")
	}

	// Create Schedule
	schedule := sched.NewSchedule("every30s", fixedTimer30second, job)

	// Start
	schedule.Start()

Options

Additional Options can be passed to Schedule to change its behavior.

// Create Schedule
schedule := sched.NewSchedule("every30s", fixedTimer30second, job,
	sched.WithLogger(sched.DefaultLogger()),
	opt2,
	opt3,
	....,
)

Logger Option

WithLogger( logger Logger) -> Supply the Schedule the Logger it is going to use for logging.

  1. func DefaultLogger() Logger : Provide a Default Logging Interface to be used Instead of Implementing your own.
  2. func NopLogger() Logger : A nop Logger that will not output anything to stdout.

Metrics Option

WithMetrics( scope tally.Scope) -> Supply the Schedule with a metrics scope it can use to export metrics.

  1. Use any of uber-go/tally implementations (Prometheus, statsd, etc)

Use func WithConsoleMetrics(printEvery time.Duration) Option Implementation to Output Metrics to stdout (good for debugging)

Expected Runtime

WithExpectedRunTime(d time.Duration) -> Supply the Schedule with the expected duration for the job to run, schedule will output corresponding logs and metrics if job run exceeded expected.

Schedule(r)

Scheduler manage one or more Schedule creating them using common options, enforcing unique IDs, and supply methods to Start / Stop all schedule(s).


Exported Metrics

Metric Type Desc
up Gauge If the schedule is Running / Stopped
runs Counter Number of Runs Since Starting
runs_overlapping Counter Number of times more than one job was running together. (Overlapped)
run_actual_elapsed_time Time Elapsed Time between Starting and Ending of Job Execution
run_total_elapsed_time Time Total Elapsed Time between Creating the Job and Ending of Job Execution, This differ from Actual Elapsed time when Overlapping blocking is Implemented
run_errors Counter Count Number of Times a Job error'd(Panicked) during execution.
run_exceed_expected_time Counter Count Number of Times a Job Execution Time exceeded the Expected Time

In Prometheus Format

# HELP sched_run_actual_elapsed_time sched_run_actual_elapsed_time summary
# TYPE sched_run_actual_elapsed_time summary
sched_run_actual_elapsed_time{id="every5s",quantile="0.5"} 0.203843151
sched_run_actual_elapsed_time{id="every5s",quantile="0.75"} 1.104031623
sched_run_actual_elapsed_time{id="every5s",quantile="0.95"} 1.104031623
sched_run_actual_elapsed_time{id="every5s",quantile="0.99"} 1.104031623
sched_run_actual_elapsed_time{id="every5s",quantile="0.999"} 1.104031623
sched_run_actual_elapsed_time_sum{id="every5s"} 1.307874774
sched_run_actual_elapsed_time_count{id="every5s"} 2
# HELP sched_run_errors sched_run_errors counter
# TYPE sched_run_errors counter
sched_run_errors{id="every5s"} 0
# HELP sched_run_exceed_expected_time sched_run_exceed_expected_time counter
# TYPE sched_run_exceed_expected_time counter
sched_run_exceed_expected_time{id="every5s"} 0
# HELP sched_run_total_elapsed_time sched_run_total_elapsed_time summary
# TYPE sched_run_total_elapsed_time summary
sched_run_total_elapsed_time{id="every5s",quantile="0.5"} 0.203880714
sched_run_total_elapsed_time{id="every5s",quantile="0.75"} 1.104065614
sched_run_total_elapsed_time{id="every5s",quantile="0.95"} 1.104065614
sched_run_total_elapsed_time{id="every5s",quantile="0.99"} 1.104065614
sched_run_total_elapsed_time{id="every5s",quantile="0.999"} 1.104065614
sched_run_total_elapsed_time_sum{id="every5s"} 1.307946328
sched_run_total_elapsed_time_count{id="every5s"} 2
# HELP sched_runs sched_runs counter
# TYPE sched_runs counter
sched_runs{id="every5s"} 2
# HELP sched_runs_overlapping sched_runs_overlapping counter
# TYPE sched_runs_overlapping counter
sched_runs_overlapping{id="every5s"} 0
# HELP sched_up sched_up gauge
# TYPE sched_up gauge
sched_up{id="every5s"} 1

Examples

  1. schedule-console-metrics
  2. schedule-cron
  3. schedule-fixed
  4. schedule-four-mixed-timers
  5. schedule-once
  6. schedule-overlapping
  7. schedule-panic
  8. schedule-prom-metrics
  9. schedule-warn-expected
  10. scheduler
  11. scheduler-extra-opts

Inline Example

package main

import (
    "fmt"
    "github.com/sherifabdlnaby/sched"
    "log"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {

    cronTimer, err := sched.NewCron("* * * * *")
    if err != nil {
        panic(fmt.Sprintf("invalid cron expression: %s", err.Error()))
    }

    job := func() {
        log.Println("Doing some work...")
        time.Sleep(1 * time.Second)
        log.Println("Finished Work.")
    }

    // Create Schedule
    schedule := sched.NewSchedule("cron", cronTimer, job, sched.WithLogger(sched.DefaultLogger()))

    // Start Schedule
    schedule.Start()

    // Stop schedule after 5 Minutes
    time.AfterFunc(5*time.Minute, func() {
        schedule.Stop()
    })

    // Listen to CTRL + C
    signalChan := make(chan os.Signal, 1)
    signal.Notify(signalChan, syscall.SIGTERM, syscall.SIGINT, syscall.SIGQUIT)
    _ = <-signalChan

    // Stop before shutting down.
    schedule.Stop()

    return
}

Output for 3 minutes

2021-04-10T12:30: 13.132+0200    INFO    sched   sched/schedule.go: 96    Job Schedule Started    {"id": "cron"}
2021-04-10T12:30: 13.132+0200    INFO    sched   sched/schedule.go: 168   Job Next Run Scheduled  {"id": "cron", "After": "47s", "At": "2021-04-10T12:31:00+02:00"}
2021-04-10T12: 31: 00.000+0200    INFO    sched   sched/schedule.go: 168   Job Next Run Scheduled  {"id": "cron", "After": "1m0s", "At": "2021-04-10T12:32:00+02:00"}
2021-04-10T12: 31:00.000+0200    INFO    sched   sched/schedule.go: 193   Job Run Starting        {"id": "cron", "Instance": "8e1044ab-20b6-4acf-8a15-e06c0418522c"}
2021/04/10 12: 31: 00 Doing some work...
2021/04/10 12: 31: 01 Finished Work.
2021-04-10T12: 31: 01.001+0200    INFO    sched   sched/schedule.go: 208   Job Finished    {"id": "cron", "Instance": "8e1044ab-20b6-4acf-8a15-e06c0418522c", "Duration": "1.001s", "State": "FINISHED"}
2021-04-10T12:32: 00.002+0200    INFO    sched   sched/schedule.go: 168   Job Next Run Scheduled  {"id": "cron", "After": "1m0s", "At": "2021-04-10T12:33:00+02:00"}
2021-04-10T12: 32: 00.002+0200    INFO    sched   sched/schedule.go: 193   Job Run Starting        {"id": "cron", "Instance": "baae94eb-f818-4b34-a1f4-45b521a360a1"}
2021/04/10 12: 32: 00 Doing some work...
2021/04/10 12: 32: 01 Finished Work.
2021-04-10T12:32: 01.005+0200    INFO    sched   sched/schedule.go: 208   Job Finished    {"id": "cron", "Instance": "baae94eb-f818-4b34-a1f4-45b521a360a1", "Duration": "1.003s", "State": "FINISHED"}
2021-04-10T12: 33: 00.001+0200    INFO    sched   sched/schedule.go:168   Job Next Run Scheduled  {"id": "cron", "After": "1m0s", "At": "2021-04-10T12:34:00+02:00"}
2021-04-10T12:33: 00.001+0200    INFO    sched   sched/schedule.go: 193   Job Run Starting        {"id": "cron", "Instance": "71c8f0bf-3624-4a92-909c-b4149f3c62a3"}
2021/04/10 12: 33: 00 Doing some work...
2021/04/10 12: 33: 01 Finished Work.
2021-04-10T12: 33: 01.004+0200    INFO    sched   sched/schedule.go:208   Job Finished    {"id": "cron", "Instance": "71c8f0bf-3624-4a92-909c-b4149f3c62a3", "Duration": "1.003s", "State": "FINISHED"}

Output With CTRL+C

2021-04-10T12:28: 45.591+0200    INFO    sched   sched/schedule.go: 96    Job Schedule Started    {"id": "cron"}
2021-04-10T12:28: 45.592+0200    INFO    sched   sched/schedule.go: 168   Job Next Run Scheduled  {"id": "cron", "After": "14s", "At": "2021-04-10T12:29:00+02:00"}
2021-04-10T12: 29: 00.000+0200    INFO    sched   sched/schedule.go: 168   Job Next Run Scheduled  {"id": "cron", "After": "1m0s", "At": "2021-04-10T12:30:00+02:00"}
2021-04-10T12: 29:00.000+0200    INFO    sched   sched/schedule.go: 193   Job Run Starting        {"id": "cron", "Instance": "786540f1-594b-44a0-9a66-7181619e38a6"}
2021/04/10 12: 29: 00 Doing some work...
CTRL+C
2021-04-10T12: 29: 00.567+0200    INFO    sched   sched/schedule.go: 125   Stopping Schedule...    {"id": "cron"}
2021-04-10T12: 29: 00.567+0200    INFO    sched   sched/schedule.go: 130   Waiting active jobs to finish...        {"id": "cron"}
2021-04-10T12: 29: 00.567+0200    INFO    sched   sched/schedule.go: 171   Job Next Run Canceled   {"id": "cron", "At": "2021-04-10T12:30:00+02:00"}
2021/04/10 12: 29: 01 Finished Work.
2021-04-10T12: 29:01.000+0200    INFO    sched   sched/schedule.go: 208   Job Finished    {"id": "cron", "Instance": "786540f1-594b-44a0-9a66-7181619e38a6", "Duration": "1s", "State": "FINISHED"}
2021-04-10T12: 29: 01.000+0200    INFO    sched   sched/schedule.go: 133   Job Schedule Stopped {"id": "cron" }

Todo(s) and Enhancements

  • Control Logging Verbosity
  • Make Panic Recovery Optional
  • Make Job a func() error and allow retry(s), backoff, and collect errors and their metrics
  • Make Jobs context aware and support canceling Jobs Context.
  • Make allow Overlapping Optional and Configure How Overlapping is handled/denied.
  • Global Package-Level Metrics

License

MIT License Copyright (c) 2021 Sherif Abdel-Naby

Contribution

PR(s) are Open and Welcomed. ❤️

You might also like...
Crane scheduler is a Kubernetes scheduler which can schedule pod based on actual node load.

Crane-scheduler Overview Crane-scheduler is a collection of scheduler plugins based on scheduler framework, including: Dynamic scheuler: a load-aware

A persistent and flexible background jobs library for go.

Jobs Development Status Jobs is no longer being actively developed. I will still try my best to respond to issues and pull requests, but in general yo

Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs
Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs

Jenkins Job DSL Plugin Introduction Jenkins is a wonderful system for managing builds, and people love using its UI to configure jobs. Unfortunately,

A simple Cron library for go that can execute closures or functions at varying intervals, from once a second to once a year on a specific date and time. Primarily for web applications and long running daemons.

Cron.go This is a simple library to handle scheduled tasks. Tasks can be run in a minimum delay of once a second--for which Cron isn't actually design

YTask is an asynchronous task queue for handling distributed jobs in golang
YTask is an asynchronous task queue for handling distributed jobs in golang

YTask is an asynchronous task queue for handling distributed jobs in golang

Tiny library to handle background jobs.

bgjob Tiny library to handle background jobs. Use PostgreSQL to organize job queues. Highly inspired by gue Features Durable job storage At-least-ones

A way of scheduling volcano jobs

JobFlow 背景 volcano Volcano是CNCF 下首个也是唯一的基于Kubernetes的容器批量计算平台,主要用于高性能计算场景。 它提供了Kubernetes目前缺 少的一套机制,这些机制通常是机器学习大数据应用、科学计算、 特效渲染等多种高性能工作负载所需的。 现状:当前vol

Scheduler: Go jobs execution system

Scheduler Go jobs execution system. Inspired by CI/CD and Unity task scheduler.

Easy and fluent Go cron scheduling

goCron: A Golang Job Scheduling Package. goCron is a Golang job scheduling package which lets you run Go functions periodically at pre-determined inte

Comments
  • Implement a logrus based logger

    Implement a logrus based logger

    (Thanks for sending a pull request! Please make sure you click the link above to view the contribution guidelines, then fill out the blanks below.)

    What does this implement/fix? Explain your changes.

    this is a logrus based logger instead of using zap

    Does this close any currently open issues?

    Any relevant logs, error output, etc?

    (If it’s long, please paste to https://ghostbin.com/ and insert the link here.)

    Any other comments?

    Where has this been tested?

    opened by Fishwaldo 1
  • Middleware Proof of Concept

    Middleware Proof of Concept

    Proof of Concept for Middleware

    Just a simple Proof of Concept for Middleware...

    A few things:

    1. State handling needs to be stored just in the Schedule. I see some states in Schedule, and some in jobs (and for example, FINISHED state in Jobs, but the doco says a Finished State can not be rescheduled...)
    2. the Schedule.transitionState is very simple now. It should handle:
    • [ ] Selectively Blocking on some State Transitions
    • [ ] Handle Panics (new State?)
    • [ ] Handle rescheduling Jobs (eg, so you could implement retry/backoff via middleware)
    • [ ] Handle Contexts when introduced?
    opened by Fishwaldo 1
  • NewJobWithID needed?

    NewJobWithID needed?

    Just seems redundant as the Only Caller is NewJob?

    https://github.com/sherifabdlnaby/sched/blob/6eeb2cc626e821489a11ee8219b239ced2f900a2/job/job.go#L32

    opened by Fishwaldo 1
  • Additions

    Additions

    Is your feature request related to a problem? Please describe. Wondering if you would be interested in the following PR's:

    1. Methods to return individual Schedule's from the Scheduler by ID?
    2. Return a slice of all Schedule ID's in the Scheduler
    3. ExpectedRunTime to be calculated from previous runs in the Scheduler (maybe controllable via a Option?)
    4. Callback function when we exceed the ExpectedRunTIme? (instead of just logging)
    opened by Fishwaldo 2
Releases(v0.0.1)
Owner
Sherif Abdel-Naby
Software Engineer; Loves anything that scales(And Go)!
Sherif Abdel-Naby
Executes jobs in separate GO routines. Provides Timeout, StartTime controls. Provides Cancel all running job before new job is run.

jobExecutor Library to execute jobs in GO routines. Provides for Job Timeout/Deadline (MaxDuration()) Job Start WallClock control (When()) Add a job b

Eswaran SK 0 Jan 10, 2022
Job worker service that provides an API to run arbitrary Linux processes.

Job Scheduler Summary Prototype job worker service that provides an API to run arbitrary Linux processes. Overview Library The library (Worker) is a r

Renato Guimarães 8 May 26, 2022
gron, Cron Jobs in Go.

gron Gron provides a clear syntax for writing and deploying cron jobs. Goals Minimalist APIs for scheduling jobs. Thread safety. Customizable Job Type

roylee0704 945 Dec 20, 2022
Lightweight, fast and dependency-free Cron expression parser (due checker) for Golang (tested on v1.13 and above)

adhocore/gronx gronx is Golang cron expression parser ported from adhocore/cron-expr. Zero dependency. Very fast because it bails early in case a segm

Jitendra Adhikari 239 Dec 30, 2022
This package provides the way to get the previous timestamp or the next timestamp that satisfies the cron expression.

Cron expression parser Given a cron expression, you can get the previous timestamp or the next timestamp that satisfies the cron expression. I have us

gitploy.io 1 May 3, 2022
Cloud-native, enterprise-level cron job platform for Kubernetes

Furiko Furiko is a cloud-native, enterprise-level cron and adhoc job platform for Kubernetes. The main website for documentation and updates is hosted

Furiko 268 Dec 30, 2022
Easily schedule commands to run multiple times at set intervals (like a cronjob, but with one command)

hakcron Easily schedule commands to run multiple times at set intervals (like a cronjob, but for a single command) Description hakcron allows you to r

Luke Stephens (hakluke) 78 Aug 17, 2022
A lightweight job scheduler based on priority queue with timeout, retry, replica, context cancellation and easy semantics for job chaining. Build for golang web apps.

Table of Contents Introduction What is RIO? Concern An asynchronous job processor Easy management of these goroutines and chaining them Introduction W

Supratim Samanta 52 Dec 9, 2022
go-sche is a golang library that lets you schedule your task to be executed later.

go-sche is a golang library that lets you schedule your task to be executed later.

cza 3 Dec 24, 2022
Reminder is a Golang package to allow users to schedule alerts.

Reminder is a Golang package to allow users to schedule alerts. It has 4 parts: Scheduler Repeater Notifier Reminder A scheduler takes in a t

null 17 Sep 28, 2022