又一个help wanted： 我按照更新后的文档部署完之后，在部署cuda-gpu-test的时候，po的状态是CreateContainerConfigError。

describe： Normal Scheduled 6m37s default-scheduler Successfully assigned default/cuda-gpu-test-6754c65cb7-rmhxk to master-01 Warning FailedScheduling 6m37s default-scheduler pod cc3ec7a2-ad73-4c0f-a69e-474b8f149ae1 is in the cache, so can't be assumed Normal SandboxChanged 6m36s kubelet Pod sandbox changed, it will be killed and re-created. Warning Failed 4m47s (x12 over 6m37s) kubelet Error: device plugin PreStartContainer rpc failed with err: rpc error: code = Unknown desc = not such pod with the same devices list

schduler日志： I1129 02:04:20.628602 1 dealer.go:288] ------------ I1129 02:04:20.654523 1 controller.go:171] begin to sync controller for pod default/cuda-gpu-test-6754c65cb7-rmhxk I1129 02:04:20.656129 1 routes.go:70] start filter for pod default/cuda-gpu-test-6754c65cb7-rmhxk I1129 02:04:20.656164 1 predicate.go:48] Check if the pod default/cuda-gpu-test-6754c65cb7-rmhxk can be scheduled on nodes [master-01] I1129 02:04:20.656256 1 rater.go:100] d=[(20)],sortableDemand=[(20)], indexes=[0], resultIndexes=[0] I1129 02:04:20.656391 1 routes.go:83] NanoGPUFilter extenderFilterResult = {"Nodes":null,"NodeNames":["master-01"],"FailedNodes":{},"Error":""} I1129 02:04:20.657668 1 routes.go:142] start bind pod default/cuda-gpu-test-6754c65cb7-rmhxk to node master-01 I1129 02:04:20.657735 1 routes.go:143] GpuSharingBind ExtenderArgs ={cuda-gpu-test-6754c65cb7-rmhxk default cc3ec7a2-ad73-4c0f-a69e-474b8f149ae1 master-01} I1129 02:04:20.669627 1 routes.go:70] start filter for pod default/cuda-gpu-test-6754c65cb7-rmhxk I1129 02:04:20.669668 1 predicate.go:48] Check if the pod default/cuda-gpu-test-6754c65cb7-rmhxk can be scheduled on nodes [master-01] I1129 02:04:20.672587 1 dealer.go:284] ------resource status after bind for default/cuda-gpu-test-6754c65cb7-rmhxk------ I1129 02:04:20.672621 1 dealer.go:286] node master-01: (80) I1129 02:04:20.672650 1 dealer.go:288] ------------ I1129 02:04:20.672687 1 routes.go:159] extenderBindingResult = {"Error":""} I1129 02:04:20.672768 1 rater.go:100] d=[(20)],sortableDemand=[(20)], indexes=[0], resultIndexes=[0] I1129 02:04:20.672885 1 routes.go:83] NanoGPUFilter extenderFilterResult = {"Nodes":null,"NodeNames":["master-01"],"FailedNodes":{},"Error":""}

agent日志： E1129 02:12:39.642396 1 server.go:114] no pod with such device list: nano-0-93:nano-0-03:nano-0-76:nano-0-09:nano-0-73:nano-0-45:nano-0-79:nano-0-38:nano-0-16:nano-0-65:nano-0-80:nano-0-31:nano-0-70:nano-0-83:nano-0-99:nano-0-84:nano-0-56:nano-0-26:nano-0-55:nano-0-20 E1129 02:12:50.643334 1 server.go:114] no pod with such device list: nano-0-31:nano-0-45:nano-0-84:nano-0-26:nano-0-99:nano-0-09:nano-0-55:nano-0-20:nano-0-03:nano-0-70:nano-0-83:nano-0-76:nano-0-80:nano-0-93:nano-0-65:nano-0-73:nano-0-56:nano-0-79:nano-0-16:nano-0-38 E1129 02:13:04.643139 1 server.go:114] no pod with such device list: nano-0-26:nano-0-99:nano-0-73:nano-0-31:nano-0-93:nano-0-45:nano-0-65:nano-0-80:nano-0-55:nano-0-83:nano-0-20:nano-0-84:nano-0-56:nano-0-76:nano-0-79:nano-0-16:nano-0-38:nano-0-09:nano-0-03:nano-0-70 E1129 02:13:18.645539 1 server.go:114] no pod with such device list: nano-0-65:nano-0-80:nano-0-93:nano-0-70:nano-0-84:nano-0-76:nano-0-45:nano-0-83:nano-0-56:nano-0-26:nano-0-99:nano-0-09:nano-0-79:nano-0-38:nano-0-03:nano-0-20:nano-0-16:nano-0-73:nano-0-31:nano-0-55

感谢！

按照 Getting Started 中的运行步骤运行完前3步后在第4步出错

运行给定的 Create pod sharing one GPU 时报错

Events： Type Reason Age From Message

Normal Scheduled 30m default-scheduler Successfully assigned default/cuda-gpu-test-968d697b9-rnsrq to 10.10.0.15 Warning FailedScheduling 30m default-scheduler pod e4947b6d-c27d-424b-a7ac-0c04ed232ce0 is in the cache, so can't be assumed Warning Failed 28m (x12 over 30m) kubelet Error: device plugin PreStartContainer rpc failed with err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/lib/kubelet/pod-resources/kubelet.sock: connect: no such file or directory" Normal Pulled 41s (x141 over 30m) kubelet Container image "nvidia/cuda:10.0-base" already present on machine

K8s version: v1.21.3

elastic-gpu-scheduler logs: I0721 06:53:58.076290 1 main.go:44] priority algorithm: binpack I0721 06:53:58.096595 1 controller.go:57] Creating event broadcaster I0721 06:53:58.096769 1 controller.go:104] begin to wait for cache I0721 06:53:58.197332 1 controller.go:109] init the node cache successfully I0721 06:53:58.197374 1 controller.go:115] init the pod cache successfully I0721 06:53:58.197379 1 controller.go:118] end to wait for cache I0721 06:53:58.197407 1 main.go:97] server starting on the port: 39999 I0721 06:53:58.197440 1 controller.go:128] Starting GPU Sharing Controller. I0721 06:53:58.197451 1 controller.go:129] Waiting for informer caches to sync I0721 06:53:58.197454 1 controller.go:131] Starting 1 workers. I0721 06:53:58.197461 1 controller.go:136] Started workers I0721 07:10:32.095025 1 routes.go:66] start filter for pod default/cuda-gpu-test-968d697b9-2cc2z I0721 07:10:32.110343 1 gpu.go:66] Trade: (core: 0, memory: 256, gpu count: 0) I0721 07:10:32.110373 1 gpu.go:95] start to allocate request: scheduler.GPUUnit{Core:0, Memory:256, GPUCount:0}, gpus: scheduler.GPUs{(*scheduler.GPU)(0xc000116de0)} I0721 07:10:32.110412 1 gpu.go:116] allocate request: scheduler.GPUUnit{Core:0, Memory:256, GPUCount:0}, gpu: &scheduler.GPU{CoreAvailable:100, MemoryAvailable:15109, CoreTotal:100, MemoryTotal:15109} I0721 07:10:32.110422 1 scheduler.go:146] assume: 10.10.0.15 [[0]], err: I0721 07:10:32.110523 1 scheduler.go:170] node allcated: {"54ab24a2":{"Request":[{"Core":0,"Memory":256,"GPUCount":0}],"Allocated":[[0]],"Score":0}} I0721 07:10:32.110569 1 routes.go:79] ElasticGPUPredicate extenderFilterResult = {"Nodes":null,"NodeNames":["10.10.0.15"],"FailedNodes":{},"Error":""} I0721 07:10:32.111581 1 routes.go:138] start bind pod default/cuda-gpu-test-968d697b9-2cc2z to node 10.10.0.15 I0721 07:10:32.114008 1 node.go:95] allocated option: &{Request:(core: 0, memory: 256, gpu count: 0) Allocated:[[0]] Score:0} I0721 07:10:32.123467 1 routes.go:66] start filter for pod default/cuda-gpu-test-968d697b9-2cc2z I0721 07:10:32.128699 1 routes.go:155] extenderBindingResult = {"Error":""} I0721 07:10:32.128888 1 gpu.go:66] Trade: (core: 0, memory: 256, gpu count: 0) I0721 07:10:32.128926 1 gpu.go:95] start to allocate request: scheduler.GPUUnit{Core:0, Memory:256, GPUCount:0}, gpus: scheduler.GPUs{(*scheduler.GPU)(0xc000116de0)} I0721 07:10:32.128966 1 gpu.go:116] allocate request: scheduler.GPUUnit{Core:0, Memory:256, GPUCount:0}, gpu: &scheduler.GPU{CoreAvailable:100, MemoryAvailable:14853, CoreTotal:100, MemoryTotal:15109} I0721 07:10:32.128975 1 scheduler.go:146] assume: 10.10.0.15 [[0]], err: I0721 07:10:32.129012 1 scheduler.go:170] node allcated: {"54ab24a2":{"Request":[{"Core":0,"Memory":256,"GPUCount":0}],"Allocated":[[0]],"Score":0}} I0721 07:10:32.129035 1 routes.go:79] ElasticGPUPredicate extenderFilterResult = {"Nodes":null,"NodeNames":["10.10.0.15"],"FailedNodes":{},"Error":""} I0721 07:16:32.058377 1 routes.go:66] start filter for pod default/cuda-gpu-test-968d697b9-5h7k4 I0721 07:16:32.058425 1 scheduler.go:146] assume: 10.10.0.15 [[0]], err: I0721 07:16:32.058461 1 scheduler.go:170] node allcated: {"54ab24a2":{"Request":[{"Core":0,"Memory":256,"GPUCount":0}],"Allocated":[[0]],"Score":0}} I0721 07:16:32.058497 1 routes.go:79] ElasticGPUPredicate extenderFilterResult = {"Nodes":null,"NodeNames":["10.10.0.15"],"FailedNodes":{},"Error":""} I0721 07:16:32.059431 1 routes.go:138] start bind pod default/cuda-gpu-test-968d697b9-5h7k4 to node 10.10.0.15 I0721 07:16:32.067092 1 node.go:95] allocated option: &{Request:(core: 0, memory: 256, gpu count: 0) Allocated:[[0]] Score:0} I0721 07:16:32.075019 1 routes.go:66] start filter for pod default/cuda-gpu-test-968d697b9-5h7k4 I0721 07:16:32.078795 1 routes.go:155] extenderBindingResult = {"Error":""} I0721 07:16:32.078864 1 gpu.go:66] Trade: (core: 0, memory: 256, gpu count: 0) I0721 07:16:32.078932 1 gpu.go:95] start to allocate request: scheduler.GPUUnit{Core:0, Memory:256, GPUCount:0}, gpus: scheduler.GPUs{(*scheduler.GPU)(0xc000116de0)} I0721 07:16:32.078950 1 gpu.go:116] allocate request: scheduler.GPUUnit{Core:0, Memory:256, GPUCount:0}, gpu: &scheduler.GPU{CoreAvailable:100, MemoryAvailable:14853, CoreTotal:100, MemoryTotal:15109} I0721 07:16:32.078957 1 scheduler.go:146] assume: 10.10.0.15 [[0]], err: I0721 07:16:32.078970 1 scheduler.go:170] node allcated: {"54ab24a2":{"Request":[{"Core":0,"Memory":256,"GPUCount":0}],"Allocated":[[0]],"Score":0}} I0721 07:16:32.078976 1 routes.go:79] ElasticGPUPredicate extenderFilterResult = {"Nodes":null,"NodeNames":["10.10.0.15"],"FailedNodes":{},"Error":""} I0721 07:16:39.620552 1 controller.go:296] delete pod default/cuda-gpu-test-968d697b9-2cc2z I0721 07:17:04.615647 1 controller.go:296] delete pod default/cuda-gpu-test-968d697b9-5h7k4 I0721 07:17:27.528301 1 routes.go:66] start filter for pod default/cuda-gpu-test-968d697b9-rnsrq I0721 07:17:27.528349 1 scheduler.go:146] assume: 10.10.0.15 [[0]], err: I0721 07:17:27.528393 1 scheduler.go:170] node allcated: {"54ab24a2":{"Request":[{"Core":0,"Memory":256,"GPUCount":0}],"Allocated":[[0]],"Score":0}} I0721 07:17:27.528407 1 routes.go:79] ElasticGPUPredicate extenderFilterResult = {"Nodes":null,"NodeNames":["10.10.0.15"],"FailedNodes":{},"Error":""} I0721 07:17:27.530168 1 routes.go:138] start bind pod default/cuda-gpu-test-968d697b9-rnsrq to node 10.10.0.15 I0721 07:17:27.548958 1 node.go:95] allocated option: &{Request:(core: 0, memory: 256, gpu count: 0) Allocated:[[0]] Score:0} I0721 07:17:27.559443 1 routes.go:66] start filter for pod default/cuda-gpu-test-968d697b9-rnsrq I0721 07:17:27.563396 1 routes.go:155] extenderBindingResult = {"Error":""} I0721 07:17:27.563491 1 gpu.go:66] Trade: (core: 0, memory: 256, gpu count: 0) I0721 07:17:27.563507 1 gpu.go:95] start to allocate request: scheduler.GPUUnit{Core:0, Memory:256, GPUCount:0}, gpus: scheduler.GPUs{(*scheduler.GPU)(0xc000116de0)} I0721 07:17:27.563533 1 gpu.go:116] allocate request: scheduler.GPUUnit{Core:0, Memory:256, GPUCount:0}, gpu: &scheduler.GPU{CoreAvailable:100, MemoryAvailable:14853, CoreTotal:100, MemoryTotal:15109} I0721 07:17:27.563543 1 scheduler.go:146] assume: 10.10.0.15 [[0]], err: I0721 07:17:27.563561 1 scheduler.go:170] node allcated: {"54ab24a2":{"Request":[{"Core":0,"Memory":256,"GPUCount":0}],"Allocated":[[0]],"Score":0}} I0721 07:17:27.563574 1 routes.go:79] ElasticGPUPredicate extenderFilterResult = {"Nodes":null,"NodeNames":["10.10.0.15"],"FailedNodes":{},"Error":""}

elastic-gpu-agent logs: I0721 06:53:53.073287 1 main.go:31] start to run elastic gpu agent I0721 06:53:53.073314 1 manager.go:146] start to run gpu manager I0721 06:53:53.073345 1 manager.go:150] polling if the sitter has done listing pods:false I0721 06:53:53.174001 1 manager.go:150] polling if the sitter has done listing pods:true I0721 06:53:53.174045 1 base.go:237] start plugin elasticgpu.io/gpu-memory I0721 06:53:53.174053 1 base.go:237] start plugin elasticgpu.io/gpu-core I0721 06:54:53.174171 1 base.go:250] gpushare plugin starts to GC I0721 06:55:53.174616 1 base.go:250] gpushare plugin starts to GC I0721 06:56:53.175347 1 base.go:250] gpushare plugin starts to GC I0721 06:57:53.176272 1 base.go:250] gpushare plugin starts to GC I0721 06:58:53.177349 1 base.go:250] gpushare plugin starts to GC I0721 06:59:53.178092 1 base.go:250] gpushare plugin starts to GC I0721 07:00:53.178252 1 base.go:250] gpushare plugin starts to GC I0721 07:01:53.179227 1 base.go:250] gpushare plugin starts to GC I0721 07:02:53.180218 1 base.go:250] gpushare plugin starts to GC I0721 07:03:53.180939 1 base.go:250] gpushare plugin starts to GC I0721 07:04:53.181332 1 base.go:250] gpushare plugin starts to GC I0721 07:05:53.181719 1 base.go:250] gpushare plugin starts to GC I0721 07:06:53.182780 1 base.go:250] gpushare plugin starts to GC I0721 07:07:53.183888 1 base.go:250] gpushare plugin starts to GC I0721 07:08:53.184118 1 base.go:250] gpushare plugin starts to GC I0721 07:09:53.185022 1 base.go:250] gpushare plugin starts to GC I0721 07:10:53.185990 1 base.go:250] gpushare plugin starts to GC E0721 07:10:55.889130 1 gpushare.go:220] no pod with such device list: 0-3343:0-6581:0-5234:0-12894:0-7636:0-6888:0-4835:0-9011:0-8006:0-6861:0-1042:0-6959:0-14053:0-509:0-10899:0-7266:0-14452:0-1227:0-4281:0-13003:0-13958:0-1698:0-2474:0-4914:0-1056:0-4461:0-5743:0-8145:0-10397:0-4574:0-4712:0-9063:0-14279:0-7690:0-2665:0-3927:0-2140:0-8928:0-13664:0-4660:0-12006:0-349:0-3692:0-13498:0-6416:0-2820:0-6215:0-4206:0-12306:0-13773:0-8747:0-1643:0-12232:0-11414:0-8718:0-9506:0-10449:0-1593:0-5686:0-14147:0-12616:0-12921:0-3725:0-10957:0-9107:0-6061:0-1675:0-10198:0-1969:0-2793:0-6456:0-11085:0-10066:0-11882:0-13693:0-10912:0-13679:0-1604:0-13171:0-10028:0-10307:0-2865:0-7240:0-6591:0-7183:0-8774:0-1445:0-6064:0-13261:0-700:0-8486:0-7533:0-3108:0-6295:0-1228:0-14363:0-14691:0-2755:0-9188:0-11138:0-13751:0-4396:0-4673:0-7527:0-8713:0-12582:0-9839:0-9814:0-2558:0-9850:0-14212:0-7968:0-7261:0-7206:0-9499:0-6289:0-11172:0-14683:0-8812:0-1316:0-15076:0-8205:0-1083:0-3454:0-11144:0-3161:0-3389:0-12705:0-7300:0-11754:0-2775:0-12189:0-11084:0-2654:0-1246:0-13387:0-9103:0-8207:0-3903:0-11021:0-548:0-1886:0-10441:0-1312:0-11527:0-4247:0-12398:0-7862:0-4874:0-8739:0-9199:0-11251:0-6430:0-10498:0-12020:0-7648:0-2390:0-10994:0-7306:0-5316:0-9179:0-7841:0-9633:0-103:0-12639:0-1546:0-12134:0-11651:0-5150:0-6772:0-1729:0-9793:0-11632:0-8060:0-767:0-5867:0-14830:0-12005:0-2264:0-1181:0-8836:0-4173:0-5627:0-2954:0-9415:0-13840:0-14879:0-9787:0-5261:0-7048:0-11162:0-7094:0-7665:0-11361:0-11129:0-904:0-4225:0-6400:0-12790:0-12920:0-12244:0-13686:0-8915:0-5882:0-4138:0-3202:0-566:0-10562:0-9305:0-2550:0-14799:0-10298:0-13103:0-4904:0-115:0-13839:0-1210:0-13409:0-10036:0-9355:0-9686:0-13114:0-14686:0-12524:0-14628:0-14752:0-13436:0-3273:0-6540:0-11628:0-8079:0-6230:0-4218:0-8831:0-593:0-15067:0-13974:0-6399:0-10514:0-4308:0-11764:0-11163:0-1793:0-7385:0-8848:0-10001:0-9543:0-3005:0-11847:0-14245:0-3496:0-13000:0-2435:0-3297:0-5697:0-5872 E0721 07:10:56.382019 1 gpushare.go:220] no pod with such device list: 0-3297:0-8915:0-8836:0-6289:0-4173:0-4461:0-6581:0-14879:0-11414:0-8774:0-8486:0-11163:0-2558:0-4225:0-6591:0-10397:0-6959:0-4308:0-3692:0-12189:0-12398:0-14799:0-5697:0-9793:0-8713:0-9199:0-1246:0-3725:0-7183:0-14628:0-4904:0-13409:0-5882:0-3161:0-12006:0-13839:0-8831:0-548:0-12921:0-3927:0-3005:0-8739:0-1604:0-11361:0-11754:0-1793:0-7690:0-8205:0-3343:0-1969:0-8718:0-1643:0-9506:0-13387:0-8006:0-9787:0-2550:0-8928:0-14686:0-1083:0-9011:0-767:0-12134:0-8747:0-11847:0-11632:0-12524:0-5150:0-1546:0-8207:0-1228:0-6064:0-11129:0-12894:0-13686:0-7266:0-6230:0-4712:0-9543:0-12639:0-1675:0-10066:0-14212:0-4218:0-14053:0-5234:0-1210:0-6861:0-14147:0-8060:0-12790:0-4396:0-5261:0-2390:0-7841:0-13693:0-4574:0-566:0-10562:0-9188:0-6295:0-13840:0-7648:0-2793:0-11085:0-13773:0-5867:0-9103:0-1886:0-509:0-4281:0-14691:0-7206:0-6540:0-9107:0-6888:0-6400:0-10994:0-13664:0-10198:0-10498:0-1312:0-115:0-5627:0-10298:0-13103:0-2435:0-13498:0-4874:0-1181:0-7968:0-4914:0-4247:0-6430:0-11084:0-5743:0-349:0-13000:0-7261:0-8848:0-12306:0-8145:0-6061:0-13261:0-2665:0-8812:0-9499:0-5316:0-6215:0-593:0-1593:0-10001:0-3454:0-9415:0-2474:0-5686:0-6456:0-7527:0-13958:0-9850:0-1042:0-2865:0-10912:0-12616:0-7240:0-4673:0-12705:0-11138:0-10441:0-14279:0-14363:0-1227:0-2654:0-5872:0-11651:0-7665:0-11021:0-8079:0-3108:0-12020:0-1729:0-9839:0-2264:0-10449:0-7636:0-12244:0-13436:0-4660:0-9179:0-7048:0-2140:0-11162:0-700:0-9814:0-15076:0-7385:0-9633:0-3903:0-15067:0-10899:0-2954:0-103:0-6772:0-11764:0-10514:0-7862:0-9063:0-1698:0-14752:0-13003:0-3273:0-14452:0-1445:0-13751:0-2775:0-7533:0-7306:0-4138:0-10028:0-14683:0-13114:0-10957:0-13171:0-12232:0-11144:0-10307:0-904:0-9686:0-11882:0-3496:0-9305:0-7300:0-11172:0-11251:0-11628:0-1056:0-11527:0-6399:0-9355:0-7094:0-13679:0-14245:0-1316:0-13974:0-4206:0-2820:0-10036:0-12005:0-14830:0-2755:0-4835:0-12920:0-12582:0-3202:0-3389:0-6416

后面全是 no pod with such device list: 的重复

As you mentioned in your README.md

Elastic GPU scheduler is based on Kubernetes extended scheduler, which can schedule gpu cores, memories, percents, share gpu with multiple containers and even spread containers of pod to different GPUs.

but in the example, I am only can see the percent elastic-gpu/gpu-percent: "20"

so do you still support GPU-cores and GPU-memory

              nano-gpu.io/gpu-core: "20"
              nano-gpu.io/gpu-memory: "1"

that was in V1.0 If not, can you share the reason behind this feature removal? I am open to contributing to support it. Thanks for sharing an awesome project.

操作第4步创建gpu测试deployment时报错，不知道是不是我前面标签没有生效的原因，下面是我的scheduler-policy-config.yaml `--- apiVersion: kubescheduler.config.k8s.io/v1beta2 kind: KubeSchedulerConfiguration clientConnection: kubeconfig: /etc/kubernetes/scheduler.conf extenders:

• urlPrefix: "http://:39999/scheduler" filterVerb: filter prioritizeVerb: priorities bindVerb: bind weight: 1 enableHTTPS: false nodeCacheCapable: true managedResources:
  • name: elasticgpu.io/gpu-core
  • name: elasticgpu.io/gpu-memory`

1. Where is /etc/kubernetes located -> is that on the k8s node?

Enable Kubernetes scheduler extender Add the following configuration to extender in /etc/kubernetes/scheduler-policy-config.json: { "urlPrefix": "http:///api/v1/namespaces/kube-system/services//proxy/scheduler", "filterVerb": "filter", "prioritizeVerb": "priorities", "bindVerb": "bind", "weight": 1, "enableHttps": false, "nodeCacheCapable": true, "managedResources": [ { "name": "nano-gpu/gpu-percent" } ] }

2. Is kube-scheduler a kubectl api command? I cannot recognize it or do we need to create a k8s Policy resource?

You can set a scheduling policy by running kube-scheduler --policy-config-file or kube-scheduler --policy-configmap . Here is a scheduler policy config sample.

When I tried to debug the elastic-gpu-scheduler, the controller always missed to grab any elastic pod. So I found that there was a mismatching const from elastic-gpu to elastic-gpu-scheduler:

In elastic-gpu/apis/elasticgpu/v1alpha1/types.go:

const (
	ResourceGPUCore         v1.ResourceName = "elasticgpu.com/gpu-core"
	ResourceGPUMemory       v1.ResourceName = "elasticgpu.com/gpu-memory"
)

In elastic-gpu-scheduler/vendor/elasticgpu.io/elastic-gpu/api/elasticgpu.io/v1alpha1/types.go:

const (
	ResourceGPUCore         v1.ResourceName = "elasticgpu.io/gpu-core"
	ResourceGPUMemory       v1.ResourceName = "elasticgpu.io/gpu-memory"
)

Therefore, if I run go mod vendor again, there is a mismatching happened.

Hello, I have some questions about the logic of selecting GPU cards:

• In method Rate, the score is defined as below: https://github.com/elastic-ai/elastic-gpu-scheduler/blob/87e3fa4e1d1ac9a47007cd727a0e6e55ae378816/pkg/scheduler/rater.go#L48-L49

  • In this formula, the score is defined as range / number of allocated GPUs.
  • Is this the best objective function that can exactly reflect the Binpack policy, or is it only an approximation?
  • Will it be better to use variance as the score?

• In the stages of Assume/Score/Allocate, a map is used to cache the scheduling result, which is defined as allocated map[string]*GPUOption. And the key of this map is defined as the hash of GPURequest: https://github.com/elastic-ai/elastic-gpu-scheduler/blob/87e3fa4e1d1ac9a47007cd727a0e6e55ae378816/pkg/scheduler/node.go#L62-L64

  • If two pods' GPU requests are same, will it cause a confilct?
  • Otherwise, can we suppose that they can share the same scheduling result?
  • Will it be better to use pod's UID as the key of this map?

Thanks for any reply!

我按照Get started部署了agent和scheduler，并且把scheduler policy配置到了kube-scheduler里面。以上的步骤显示执行成功了，但当我部署文中的cuda-gpu-test的时候，它卡在了pending阶段。通过describe我看到以下事件：

Events: Type Reason Age From Message

Warning FailedScheduling 12s default-scheduler failed filter with extender at URL http://192.168.3.76:6443/api/v1/namespaces/kube-system/services/nano-gpu-scheduler/proxy/scheduler/filter, code 400 Warning FailedScheduling 11s default-scheduler failed filter with extender at URL http://192.168.3.76:6443/api/v1/namespaces/kube-system/services/nano-gpu-scheduler/proxy/scheduler/filter, code 400

然后查看了scheduler的日志：

I1125 03:26:06.565888 1 main.go:65] Priority algorithm is binpack W1125 03:26:06.566008 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I1125 03:26:06.567723 1 controller.go:62] Creating event broadcaster I1125 03:26:06.588004 1 controller.go:116] begin to wait for cache I1125 03:26:06.588025 1 controller.go:121] init the node cache successfully I1125 03:26:06.688480 1 controller.go:127] init the pod cache successfully I1125 03:26:06.688521 1 controller.go:130] end to wait for cache I1125 03:26:06.688652 1 main.go:112] server starting on the port :39999 I1125 03:26:06.688684 1 controller.go:144] Starting GPU Sharing Controller. I1125 03:26:06.688744 1 controller.go:145] Waiting for informer caches to sync I1125 03:26:06.688763 1 controller.go:147] Starting 1 workers. I1125 03:26:06.688798 1 controller.go:152] Started workers I1125 05:02:16.010601 1 controller.go:171] begin to sync controller for pod default/cuda-gpu-test-6754c65cb7-cd84p

k8s version: 1.21.6 nvidia driver：407.82

我看了下代码，它在PredicateRoute这里检测了请求的body为空返回400。是什么地方配置错了吗？

As referred in https://kubernetes.io/docs/reference/scheduling/policies/ from v1.23 of Kubernetes the --policy-config-fileflag for the kube scheduler is not supported anymore. By following the instructions in readme.md you will get an error in kube-scheduler logs that tells you that. The solution to this is to use the flag --config=/etc/kubernetes/scheduler-policy-config.yaml and to create a file scheduler-policy-config.yaml compliant to KubeSchedulerConfiguration requirements. I tried converting the file scheduler-policy-config.json that was suggested to use in the readme.md and i got this scheduler-policy-config.yaml :

apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
extenders:
- urlPrefix: "http://< ip >:39999/scheduler"
  filterVerb: filter
  prioritizeVerb: priorities
  bindVerb: bind
  weight: 1
  enableHTTPS: false
  nodeCacheCapable: true
  managedResources:
  - name: nano-gpu/gpu-percent

And now everything works fine. I think it would be nice to add this in the readme.md for latest versions of K8s.

Add /metrics for profiling scheduling performance. The metrics are as follows:

• nano_gpu_scheduler_assuming_duration_seconds: record during Predicate
• nano_gpu_scheduler_sorting_duration_seconds: record during Priority
• nano_gpu_scheduler_binding_duration_seconds: record during Bind
• nano_gpu_scheduler_scheduling_duration_seconds: record during scheduling of nodes and gpus

1. spread: choose the node with more free gpu cards, more total available gpu and less gpus
2. binpack: rate higher score to nodes with more usage and less gpus

Question

if workload use Deployment or StatefulSet, annotation elasticgpu.io/assumed: "true" and elasticgpu.io/container-xxx: "0" can add into pod yaml. But , use Pod, annotation can not add.

对于gpu-core分配策略的设计有不解之处

阅读scheduler和agent的代码后发现在agent中通过这段代码把pod获得的设备传入：

	if len(devicesIDs) > 100 {
		for i := 0; i < len(devicesIDs)/100; i++ {
			devices = append(devices, &pluginapi.DeviceSpec{
				ContainerPath: fmt.Sprintf("/host/dev/elastic-gpu-%s-%d", faker, i),
				HostPath:      fmt.Sprintf("/dev/elastic-gpu-%s-%d", faker, i),
				Permissions:   "rwm",
			})
		}
	}

在当前设计下，如果某节点有3张GPU，有2个elasticgpu.io/gpu-core为150的pod都被调度到该节点，那么调度器会给这两个pod都只挂载一张显卡。此时该节点的gpu-core资源已经是300/300，不会再有其他需要gpu-core资源的pod会被调度到该节点，但该节点上只有两张显卡被挂载利用起来，还有一张显卡处于空置状态。

我想了解一下这是设计的Bug还是特意设计成这样的呢？