custom-kube-scheduler

Custom Kubernetes Scheduler

1. Create your custom plugin

There is a list of stages during the scheduling procedure you can interve as it also stated in scheduling framework- v1.17:

Queue sort

These plugins are used to sort Pods in the scheduling queue. A queue sort plugin essentially will provide a “less(Pod1, Pod2)” function. Only one queue sort plugin may be enabled at a time.

Pre-filter

These plugins are used to pre-process info about the Pod, or to check certain conditions that the cluster or the Pod must meet. If a pre-filter plugin returns an error, the scheduling cycle is aborted.

Filter

These plugins are used to filter out nodes that cannot run the Pod. For each node, the scheduler will call filter plugins in their configured order. If any filter plugin marks the node as infeasible, the remaining plugins will not be called for that node. Nodes may be evaluated concurrently.

Post-filter

This is an informational extension point. Plugins will be called with a list of nodes that passed the filtering phase. A plugin may use this data to update internal state or to generate logs/metrics.

Note: Plugins wishing to perform “pre-scoring” work should use the post-filter extension point.

Scoring

These plugins are used to rank nodes that have passed the filtering phase. The scheduler will call each scoring plugin for each node. There will be a well defined range of integers representing the minimum and maximum scores. After the normalize scoring phase, the scheduler will combine node scores from all plugins according to the configured plugin weights.

Normalize scoring

These plugins are used to modify scores before the scheduler computes a final ranking of Nodes. A plugin that registers for this extension point will be called with the scoring results from the same plugin. This is called once per plugin per scheduling cycle.

For example, suppose a plugin BlinkingLightScorer ranks Nodes based on how many blinking lights they have.

In the path kubernetes/pkg/scheduler/framework/plugins you can find all the existing plugins and also add your own.

In this repository we will add a new Score plugin. According to the kubernetes/pkg/scheduler/framework/v1alpha/interface.go , the interface for a Score plugin should be the following:

 type  ScorePlugin  interface {

Plugin

// Score is called on each filtered node. It must return success and an integer

// indicating the rank of the node. All scoring plugins must return success or

// the pod will be rejected.

Score(ctx context.Context, state *CycleState, p *v1.Pod, nodeName string) (int64, *Status)

  

// ScoreExtensions returns a ScoreExtensions interface if it implements one, or nil if does not.

ScoreExtensions() ScoreExtensions

}

Our custom Score plugin can be found in pkg/scheduler/framework/plugins/noderesources/custom_resource_alocation.go

2. Register the plugin

Afterwards, in order to use the plugin with an in-tree implementation, you need to register it along with the other plugins in the pkg/scheduler/framework/plugins/registry.go

func  NewInTreeRegistry() framework.Registry {

	return framework.Registry{
		... , 
		noderesources.CustomAllocatedName: noderesources.NewCustomAllocated,
		...,
	}
}

3. Enable the plugin

You can enable or disable plugins using the configuration yaml file on runtime. However, here we will show the hard-coded way.

By editing the pkg/scheduler/algorithmprovider/registry.go file.

func  getDefaultConfig() *schedulerapi.Plugins {

	return &schedulerapi.Plugins{

...

		Score: &schedulerapi.PluginSet{

			Enabled: []schedulerapi.Plugin{

				{Name: noderesources.CustomAllocatedName, Weight: 100000},

				{Name: imagelocality.Name, Weight: 1},

				...
			},

		},
...
	}

}