Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
What happened:
I0516 04:37:06.887104 1 scheduler_helper.go:38] Considering Task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-7-61.ec2.internal>: <cpu 1500.00, memory 3670016000.00, GPU 0.00> vs. <cpu 1890.00, memory 3782279168.00, GPU 0.00>
I0516 04:37:06.887142 1 scheduler_helper.go:38] Considering Task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-19-145.ec2.internal>: <cpu 1500.00, memory 3670016000.00, GPU 0.00> vs. <cpu 1890.00, memory 3782279168.00, GPU 0.00>
E0516 04:37:06.887314 1 scheduler_helper.go:43] Predicates failed for task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-19-145.ec2.internal>: task <default/batch-1-p-w-2-cbtvn> does not tolerate node <ip-10-0-19-145.ec2.internal> taints
I0516 04:37:06.887162 1 scheduler_helper.go:38] Considering Task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-2-157.ec2.internal>: <cpu 1500.00, memory 3670016000.00, GPU 0.00> vs. <cpu 1900.00, memory 3782279168.00, GPU 0.00>
E0516 04:37:06.887334 1 scheduler_helper.go:43] Predicates failed for task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-2-157.ec2.internal>: task <default/batch-1-p-w-2-cbtvn> does not tolerate node <ip-10-0-2-157.ec2.internal> taints
I0516 04:37:06.887195 1 scheduler_helper.go:38] Considering Task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-21-200.ec2.internal>: <cpu 1500.00, memory 3670016000.00, GPU 0.00> vs. <cpu 1890.00, memory 3782279168.00, GPU 0.00>
E0516 04:37:06.887385 1 scheduler_helper.go:43] Predicates failed for task <default/batch-1-p-w-2-cbtvn> on node <ip-10-0-21-200.ec2.internal>: task <default/batch-1-p-w-2-cbtvn> does not tolerate node <ip-10-0-21-200.ec2.internal> taints
E0516 04:37:06.887553 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/kubernetes/pkg/scheduler/cache/node_info.go:611
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder/nodeorder.go:66
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder/nodeorder.go:243
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/framework/session_plugins.go:364
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/actions/allocate/allocate.go:148
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/util/scheduler_helper.go:64
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:65
/usr/local/go/src/runtime/asm_amd64.s:1333
I0516 04:37:06.887857 1 asm_amd64.s:523] Leaving Allocate ...
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x168 pc=0x10ad54d]
goroutine 61624 [running]:
github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108
panic(0x126a4a0, 0x21a4e50)
/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/kubernetes-sigs/volcano/vendor/k8s.io/kubernetes/pkg/scheduler/cache.(*NodeInfo).SetNode(0xc000511790, 0x0, 0x0, 0xc000511790)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/kubernetes/pkg/scheduler/cache/node_info.go:611 +0x3d
github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder.generateNodeMapAndSlice(0xc0005cc4b0, 0xc000486180, 0x1, 0x1, 0x1)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder/nodeorder.go:66 +0x12b
github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder.(*nodeOrderPlugin).OnSessionOpen.func1(0xc000429700, 0xc0008f40a0, 0xc0006061c0, 0x9, 0xc0006f9338)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/plugins/nodeorder/nodeorder.go:243 +0x12e
github.com/kubernetes-sigs/volcano/pkg/scheduler/framework.(*Session).NodeOrderFn(0xc0003c60e0, 0xc000429700, 0xc0008f40a0, 0xc000826738, 0x4058f1, 0xc00090a058)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/framework/session_plugins.go:364 +0x1bd
github.com/kubernetes-sigs/volcano/pkg/scheduler/framework.(*Session).NodeOrderFn-fm(0xc000429700, 0xc0008f40a0, 0x0, 0x0, 0xc000077b00)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/actions/allocate/allocate.go:148 +0x3e
github.com/kubernetes-sigs/volcano/pkg/scheduler/util.PrioritizeNodes.func1(0x0)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/pkg/scheduler/util/scheduler_helper.go:64 +0x7f
github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue.ParallelizeUntil.func1(0xc0006c0720, 0xc00090a000, 0xc000486168, 0xc000780840)
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:65 +0x91
created by github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue.ParallelizeUntil
/root/asif/go/src/github.com/kubernetes-sigs/volcano/vendor/k8s.io/client-go/util/workqueue/parallelizer.go:57 +0x148
What you expected to happen:
no err
How to reproduce it (as minimally and precisely as possible):
job reach podGroup minMember
Anything else we need to know?:
Environment:
-
Kubernetes version (use kubectl version
):
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
-
Cloud provider or hardware configuration: aws eks
-
OS (e.g. from /etc/os-release): ubuntu
-
Kernel (e.g. uname -a
):
-
Install tools: eks
-
Others:
kind/bug priority/important-soon lifecycle/rotten