Tasks

Step-by-step instructions for performing operations with Kubernetes.

Edit This Page

在联邦中设置放置策略

本文展示了如何使用外部策略引擎来对联邦资源执行基于策略的放置决策。

Before you begin

您需要拥有一个运行的 Kubernetes 集群 (它被引用为 host 集群)。 请查看 入门 指导,根据您的平台选择相应的安装说明。

部署联邦并配置外部策略引擎

可以使用 kubefed init 来部署联邦控制平面。

在部署联邦控制平面后,您必须在联邦 API server 中配置一个准入控制器, 该控制器执行从外部策略引擎接收到的放置决策。

kubectl create -f scheduling-policy-admission.yaml

下面展示的是准入控制器的一个 ConfigMap 示例:

scheduling-policy-admission.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: admission
  namespace: federation-system
data:
  config.yml: |
    apiVersion: apiserver.k8s.io/v1alpha1
    kind: AdmissionConfiguration
    plugins:
    - name: SchedulingPolicy
      path: /etc/kubernetes/admission/scheduling-policy-config.yml
  scheduling-policy-config.yml: |
    kubeconfig: /etc/kubernetes/admission/opa-kubeconfig
  opa-kubeconfig: |
    clusters:
      - name: opa-api
        cluster:
          server: http://opa.federation-system.svc.cluster.local:8181/v0/data/kubernetes/placement
    users:
      - name: scheduling-policy
        user:
          token: deadbeefsecret
    contexts:
      - name: default
        context:
          cluster: opa-api
          user: scheduling-policy
    current-context: default

该 ConfigMap 包含三个文件:

编辑联邦 API server 的 deployment 来启用 SchedulingPolicy 准入控制器。

kubectl -n federation-system edit deployment federation-apiserver

更新联邦 API server 命令行参数来启用准入控制器,并将 ConfigMap 挂载到容器中。 如果已存在 --admission-control 参数,请追加 ,SchedulingPolicy 而不是添加另一行。

--admission-control=SchedulingPolicy
--admission-control-config-file=/etc/kubernetes/admission/config.yml

添加以下卷到联邦 API server pod 中:

- name: admission-config
  configMap:
    name: admission

添加以下卷挂载到联邦 API server 容器中:

volumeMounts:
- name: admission-config
  mountPath: /etc/kubernetes/admission

部署一个外部策略引擎

Open Policy Agent (OPA) 是一种开源的、 通用的策略引擎。 您可以使用它在联邦控制平面执行基于策略的放置决策。

在 host 集群中创建一个 Service 来联系外部策略引擎:

kubectl create -f policy-engine-service.yaml

下面展示了一个 OPA 的 Service 示例。

policy-engine-service.yaml
kind: Service
apiVersion: v1
metadata:
  name: opa
  namespace: federation-system
spec:
  selector:
    app: opa
  ports:
  - name: http
    protocol: TCP
    port: 8181
    targetPort: 8181

使用联邦控制平面在 host 集群中创建一个 Deployment:

kubectl create -f policy-engine-deployment.yaml

下面展示了一个 OPA 的 Deployment 示例。

policy-engine-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  name: opa
  namespace: federation-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        - name: opa
          image: openpolicyagent/opa:0.4.10
          args:
          - "run"
          - "--server"
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:0.2
          args:
          - "-kubeconfig=/srv/kubernetes/kubeconfig"
          - "-cluster=federation/v1beta1/clusters"
          volumeMounts:
           - name: federation-kubeconfig
             mountPath: /srv/kubernetes
             readOnly: true
      volumes:
      - name: federation-kubeconfig
        secret:
          secretName: federation-controller-manager-kubeconfig

通过 ConfigMap 配置放置策略

外部策略引擎会发现联邦 API server 的 kube-federation-scheduling-policy namespace 下创建的放置策略。

如果该 namespace 不存在,请创建:

kubectl --context=federation create namespace kube-federation-scheduling-policy

配置一个示例策略来测试外部策略引擎:

policy.rego
# OPA supports a high-level declarative language named Rego for authoring and
# enforcing policies. For more infomration on Rego, visit
# http://openpolicyagent.org.

# Rego policies are namespaced by the "package" directive.
package kubernetes.placement

# Imports provide aliases for data inside the policy engine. In this case, the
# policy simply refers to "clusters" below.
import data.kubernetes.clusters

# The "annotations" rule generates a JSON object containing the key
# "federation.kubernetes.io/replica-set-preferences" mapped to <preferences>.
# The preferences values is generated dynamically by OPA when it evaluates the
# rule.
#
# The SchedulingPolicy Admission Controller running inside the Federation API
# server will merge these annotatiosn into incoming Federated resources. By
# setting replica-set-preferences, we can control the placement of Federated
# ReplicaSets.
#
# Rules are defined to generate JSON values (booleans, strings, objects, etc.)
# When OPA evaluates a rule, it generates a value IF all of the expressions in
# the body evaluate successfully. All rules can be understood intuitively as
# <head> if <body> where <body> is true if <expr-1> AND <expr-2> AND ...
# <expr-N> is true (for some set of data.)
annotations["federation.kubernetes.io/replica-set-preferences"] = preferences {
    input.kind = "ReplicaSet"
    value = {"clusters": cluster_map, "rebalance": true}
    json.marshal(value, preferences)
}

# This "annotations" rule generates a value for the "federation.alpha.kubernetes.io/cluster-selector"
# annotation.
#
# In English, the policy asserts that resources in the "production" namespace
# that are not annotated with "criticality=low" MUST be placed on clusters
# labelled with "on-premises=true".
annotations["federation.alpha.kubernetes.io/cluster-selector"] = selector {
    input.metadata.namespace = "production"
    not input.metadata.annotations.criticality = "low"
    json.marshal([{
        "operator": "=",
        "key": "on-premises",
        "values": "[true]",
    }], selector)
}

# Generates a set of cluster names that satisfy the incoming Federated
# ReplicaSet's requirements. In this case, just PCI compliance.
replica_set_clusters[cluster_name] {
    clusters[cluster_name]
    not insufficient_pci[cluster_name]
}

# Generates a set of clusters that must not be used for Federated ReplicaSets
# that request PCI compliance.
insufficient_pci[cluster_name] {
    clusters[cluster_name]
    input.metadata.annotations["requires-pci"] = "true"
    not pci_clusters[cluster_name]
}

# Generates a set of clusters that are PCI certified. In this case, we assume
# clusters are annotated to indicate if they have passed PCI compliance audits.
pci_clusters[cluster_name] {
    clusters[cluster_name].metadata.annotations["pci-certified"] = "true"
}

# Helper rule to generate a mapping of desired clusters to weights. In this
# case, weights are static.
cluster_map[cluster_name] = {"weight": 1} {
    replica_set_clusters[cluster_name]
}

下面展示的是创建示例策略的命令:

kubectl --context=federation -n kube-federation-scheduling-policy create configmap scheduling-policy --from-file=policy.rego

这个示例策略说明了几个关键概念:

测试放置策略

为其中一个集群添加注解,以表明它是经过 PCI 认证的。

kubectl --context=federation annotate clusters cluster-name-1 pci-certified=true

部署一个联邦 ReplicaSet 来测试放置策略。

replicaset-example-policy.yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  labels:
    app: nginx-pci
  name: nginx-pci
  annotations:
    requires-pci: "true"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-pci
  template:
    metadata:
      labels:
        app: nginx-pci
    spec:
      containers:
      - image: nginx
        name: nginx-pci

下面展示的是部署一个 能够 匹配策略的 ReplicaSet 的命令。

kubectl --context=federation create -f replicaset-example-policy.yaml

检查 ReplicaSet,确认适当的注解已经被应用。

kubectl --context=federation get rs nginx-pci -o jsonpath='{.metadata.annotations}'

Analytics

Create an Issue Edit this Page