Simplify setup with using dynamic scheduler extenders

[liyimeng]

Why you need it?

Right now open-local use an init-job to statically configure scheduler extends, this is not convenient for some k8s distribution like k3s. I would like to propose doing this at controller startup instead of using external json/yaml file.

How it could be?

when open-local controller startup, it uses client-go to create the scheduler config (https://kubernetes.io/docs/reference/config-api/kube-scheduler-config.v1beta3/), when it quick delete the config.

This should make open-local work much smart and stable. :)

Other related information

https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#extended-resources

[liyimeng]

I.e. use client-go in open-local controller populate this info into etcd database.

{
            "kind" : "Policy",
            "apiVersion" : "v1",
            "extenders" : [{
                "urlPrefix": "http://{{ .Values.extender.name }}.{{.Values.namespace}}:23000/scheduler",
                "filterVerb": "predicates",
                "prioritizeVerb": "priorities",
                "preemptVerb": "",
                "bindVerb": "",
                "weight": 10,
                "enableHttps": false,
                "nodeCacheCapable": true,
                "ignorable": true
            }],
            "hardPodAffinitySymmetricWeight" : 10
            }

[TheBeatles1994]

Thx for your proposal, that's what we planned to do, and it would be great if you could do it😊.

[liyimeng]

Thx for your proposal, that's what we planned to do, and it would be great if you could do it😊.

I can do it, but I prefer doing it on top of your refact :)

[songshusen1978]

1. 停止K3S, 手动编写/var/lib/rancher/k3s/server/kube-scheduler-configuration.yaml配置文件

    systemctl stop k3s

apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig
extenders:
- urlPrefix: http://open-local-scheduler-extender.open-local:23000/scheduler
  filterVerb: predicates
  prioritizeVerb: priorities
  weight: 10
  ignorable: true
  nodeCacheCapable: true

2. 修改K3S的启动参数

在/etc/systemd/system/k3s.service文件的k3s启动参数增加如下内容

'--kube-scheduler-arg'
'config=/var/lib/rancher/k3s/server/kube-scheduler-configuration.yaml' \

3. 重新启动K3S

systemctl daemon-reload
systemctl start k3s