Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

Description:

For batch workload, there're several "similar" tasks in one job; if one task failed to fit one node, the other tasks will have the same result (except pod affinity/anti-affinity). In kube-batch, we don-t know which tasks are similar, but we can support customized indexed, e.g. an annotation of task template uuid.

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug /kind feature /kind feature

What happened: For scenarios of BigData and ML, the batch jobs often summited in queues, we need to provide the capacity to control the number of running pod groups in queues with limited hardware resources. What you expected to happen: Add a parameter in Queue API:

type QueueSpec struct {
	Weight int32 `json:"weight,omitempty" protobuf:"bytes,1,opt,name=weight"`
         //  PodGroupNumber defines the max number of running podGroup in one queue, default to infinite.
        PodGroupNumber `json:"podgroupnumber,omitempty" protobuf:"bytes,1,opt,name=podgroupnumber"`
}

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

DESCRIPTION: -- symptom of the problem a customer would see

For now, kube-batch supports several features as follow; but we only have basic tutorial and design doc for gang-scheduling. It's better to have related docs for all features.

• [ ] Gang-scheduling/Coscheduling (https://github.com/kubernetes/community/pull/2337)
• [ ] Preemption/Reclaim
• [ ] Task Priority within Job
• [ ] Queue
• [ ] DRF for Job sharing within Queue

What this PR does / why we need it: Fix the separate build for the extended queuejob in contrib Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes # N/A Special notes for your reviewer:

Release note:

NONE

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

Description:

Test kube-batch against conformance test after https://github.com/kubernetes-sigs/kube-batch/issues/588 and https://github.com/kubernetes-sigs/kube-batch/issues/520

/kind feature

Since kube batch session open and seems that it will update its cache snapshot every 1 second (pls correct me if I didn't understand correctly), what if there are any other normal pod being scheduled by the k8s default scheduler during this time period of 1s and will the kube-batch cache still be consistent with the cluster status?

Or will kube-batch and kube-scheduler schedule pods at the same time?

Thanks in advance if there are some explanation or code reference could work as well.

Hi, ! I've been be interested in multi-tenant, so I was wondering if you can share how to implement multi-tenant an Resource sharing between multi-tenant .Thanks.

Added backfill action that allows lower priority jobs to run when higher priority jobs cannot run but still hold resources

Tested with added unit tests, e2e tests and local manual test

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

Description:

Currently, kube-batch only raised Unschedulable event for its status; it's better to include phase and conditions in PodGroup.Status to give more detail.

Tasks

• [x] Design Doc (https://github.com/kubernetes-sigs/kube-batch/pull/533)
• [x] Add phase/condition into PodGroup.Status CRD (https://github.com/kubernetes-sigs/kube-batch/pull/525)
• [x] Updated phase/condition by kube-batch (https://github.com/kubernetes-sigs/kube-batch/pull/560)

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

What happened:

I0516 04:37:06.887104       1 scheduler_helper.go:38] Considering Task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-7-61.ec2.internal>: <cpu 1500.00, memory 3670016000.00, GPU 0.00> vs. <cpu 1890.00, memory 3782279168.00, GPU 0.00>
I0516 04:37:06.887142       1 scheduler_helper.go:38] Considering Task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-19-145.ec2.internal>: <cpu 1500.00, memory 3670016000.00, GPU 0.00> vs. <cpu 1890.00, memory 3782279168.00, GPU 0.00>
E0516 04:37:06.887314       1 scheduler_helper.go:43] Predicates failed for task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-19-145.ec2.internal>: task <default/batch-1-p-w-2-cbtvn> does not tolerate node <ip-10-0-19-145.ec2.internal> taints
I0516 04:37:06.887162       1 scheduler_helper.go:38] Considering Task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-2-157.ec2.internal>: <cpu 1500.00, memory 3670016000.00, GPU 0.00> vs. <cpu 1900.00, memory 3782279168.00, GPU 0.00>
E0516 04:37:06.887334       1 scheduler_helper.go:43] Predicates failed for task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-2-157.ec2.internal>: task <default/batch-1-p-w-2-cbtvn> does not tolerate node <ip-10-0-2-157.ec2.internal> taints
I0516 04:37:06.887195       1 scheduler_helper.go:38] Considering Task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-21-200.ec2.internal>: <cpu 1500.00, memory 3670016000.00, GPU 0.00> vs. <cpu 1890.00, memory 3782279168.00, GPU 0.00>
E0516 04:37:06.887385       1 scheduler_helper.go:43] Predicates failed for task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-21-200.ec2.internal>: task <default/batch-1-p-w-2-cbtvn> does not tolerate node <ip-10-0-21-200.ec2.internal> taints
E0516 04:37:06.887553       1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/kubernetes/pkg/scheduler/cache/node_info.go:611
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder/nodeorder.go:66
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder/nodeorder.go:243
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/framework/session_plugins.go:364
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/actions/allocate/allocate.go:148
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/util/scheduler_helper.go:64
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:65
/usr/local/go/src/runtime/asm_amd64.s:1333
I0516 04:37:06.887857       1 asm_amd64.s:523] Leaving Allocate ...
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x168 pc=0x10ad54d]

goroutine 61624 [running]:
github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108
panic(0x126a4a0, 0x21a4e50)
        /usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/kubernetes-sigs/volcano/vendor/k8s.io/kubernetes/pkg/scheduler/cache.(*NodeInfo).SetNode(0xc000511790, 0x0, 0x0, 0xc000511790)
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/kubernetes/pkg/scheduler/cache/node_info.go:611 +0x3d
github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder.generateNodeMapAndSlice(0xc0005cc4b0, 0xc000486180, 0x1, 0x1, 0x1)
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder/nodeorder.go:66 +0x12b
github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder.(*nodeOrderPlugin).OnSessionOpen.func1(0xc000429700, 0xc0008f40a0, 0xc0006061c0, 0x9, 0xc0006f9338)
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder/nodeorder.go:243 +0x12e
github.com/kubernetes-sigs/volcano/pkg/scheduler/framework.(*Session).NodeOrderFn(0xc0003c60e0, 0xc000429700, 0xc0008f40a0, 0xc000826738, 0x4058f1, 0xc00090a058)
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/framework/session_plugins.go:364 +0x1bd
github.com/kubernetes-sigs/volcano/pkg/scheduler/framework.(*Session).NodeOrderFn-fm(0xc000429700, 0xc0008f40a0, 0x0, 0x0, 0xc000077b00)
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/actions/allocate/allocate.go:148 +0x3e
github.com/kubernetes-sigs/volcano/pkg/scheduler/util.PrioritizeNodes.func1(0x0)
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/util/scheduler_helper.go:64 +0x7f
github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue.ParallelizeUntil.func1(0xc0006c0720, 0xc00090a000, 0xc000486168, 0xc000780840)
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:65 +0x91
created by github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue.ParallelizeUntil
        /root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:57 +0x148

What you expected to happen: no err How to reproduce it (as minimally and precisely as possible):

job reach podGroup minMember

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: aws eks

• OS (e.g. from /etc/os-release): ubuntu

• Kernel (e.g. uname -a):

• Install tools: eks

• Others:

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: After the revert of batch job patches #806 , the gang scheduling testcase is falling, need investigate and fix.

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

What happened:

It's better to follow the practice for group name of k8s-sigs , e.g. x-k8s.io.

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

What happened:

Currently, we use cmd/deepcopy-gen to generate deepcopy codes; it's better to replace such kind of source code with release of code-generators.

Is this a BUG REPORT or FEATURE REQUEST?:

/kind feature

What happened:

1.12 is end of service, it's better to upgrade k8s version to the latest one :)