【kubernetes】Minikube-Kubernetes本地环境进行开发

Minikube-Kubernetes本地环境进行开发

使用Minikube

启动Minikube

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 启动 指定国内仓库
# minikube start --image-mirror-country='cn' --registry-mirror https://dockerhub.azk8s.cn

# 指定内存大小以及启动驱动和K8S版本， 自动选择 docker 驱动。其他选项：podman, virtualbox, none, ssh
# minikube start --memory=8192MB --cpus=4 --image-mirror-country='cn' --vm-driver=virtualbox --kubernetes-version v1.23.1 
minkube start

# 检查状态
minikube status

host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100


# 访问面板
$ minikube dashboard

# 访问web前端
$ kubectl proxy

开启Addons监控等

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# 获取插件列表
$ minikube addons list

|-----------------------------|----------|--------------|                                                                                                                                                                                                                                                                                                                  
|         ADDON NAME          | PROFILE  |    STATUS    |                                                                                                                                                                                                                                                                                                                  
|-----------------------------|----------|--------------|                                                                                                                                                                                                                                                                                                                  
| ambassador                  | minikube | disabled     |                                                                                                                                                                                                                                                                                                                  
| csi-hostpath-driver         | minikube | disabled     |                                                                                                                                                                                                                                                                                                                  
| dashboard                   | minikube | enabled ✅   |                                                                                                                                                                                                                                                                                                                  
| default-storageclass        | minikube | enabled ✅   |                                                                                                                                                                                                                                                                                                                  
| efk                         | minikube | disabled     |                                                                                                                                                                                                                                                                                                                  
| freshpod                    | minikube | disabled     |                                                                                                                                                                                                                                                                                                                  
| gcp-auth                    | minikube | disabled     |                                                                                                                                                                                                                                                                                                                  
| gvisor                      | minikube | disabled     |                                                                                                                                                                                                                                                                                                                  
| helm-tiller                 | minikube | disabled     |                                                                                                                                                                                                                                                                                                                  
| ingress                     | minikube | disabled     |                                                                                                                                                                                                                                                                                                                  
| ingress-dns                 | minikube | disabled     |                                 
| istio                       | minikube | disabled     |                                 
| istio-provisioner           | minikube | disabled     |                                 
| kubevirt                    | minikube | disabled     |                                 
| logviewer                   | minikube | disabled     |                                 
| metallb                     | minikube | disabled     |                                 
| metrics-server              | minikube | disabled     |                                 
| nvidia-driver-installer     | minikube | disabled     |                                 
| nvidia-gpu-device-plugin    | minikube | disabled     |                                 
| olm                         | minikube | disabled     |                                 
| pod-security-policy         | minikube | disabled     |                                 
| registry                    | minikube | disabled     |                                 
| registry-aliases            | minikube | disabled     |                                 
| registry-creds              | minikube | disabled     |                                 
| storage-provisioner         | minikube | enabled ✅   |                                 
| storage-provisioner-gluster | minikube | disabled     |                                 
| volumesnapshots             | minikube | disabled     |                                 
|-----------------------------|----------|--------------| 


# 开启监控，可以使用 kubectl top pod,node 命令
$ minikube addons enable heapster

获取命名空间

1
2
3
4
5
6
7
8
# kubectl get namespaces
# 切换命名空间可以使用： kubens
$ kubectl get ns
NAME              STATUS   AGE
default           Active   156m
kube-node-lease   Active   156m
kube-public       Active   156m
kube-system       Active   156m

获取集群信息

1
2
3
4
5
6
$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
KubeDNS is running at https://192.168.99.100:8443/api/v1/namespaces/kube-system/services/kube-
dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

获取节点信息

1
2
3
4
$ kubectl get node

NAME       STATUS   ROLES    AGE    VERSION
minikube   Ready    master   162m   v1.14.1

运行nginx

直接生成Pod，暴露Service

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 运行nginx 这里是直接运行pod
$ kubectl run nginx --image=nginx:1.21 --port=80 --labels="app=nginx,env=dev"

# 增加新标签
# kubectl label pod $POD_NAME app2=test

# 发布服务采用对外暴露节点
# NodePort 需要被外部访问
# TargetPort
# Port
# HostPort
$ kubectl expose pod nginx --type=NodePort

# 访问
$ minikube service nginx --url

# 删除
kubectl delete -n default service,pod nginx
kubectl delete -n default service,pod -l app=nginx
kubectl delete -n default all -l app=nginx

生成Deployment

1
2
3
4
kubectl create deployment nginx --image=nginx:1.21 --port=80 --replicas=1

# 这里会生成Service，通过--type=NodePort可被外部访问
kubectl expose deployment nginx --type=NodePort --port=80 --labels="app=nginx,env=dev"

获取pods

1
2
3
4
5
6
7
8
9
10

$ kubectl get pods
# 根据标签查询
# kubectl get pods -l app=nginx

NAME                    READY   STATUS    RESTARTS   AGE
nginx-fcb945956-t8bqq   1/1     Running   0          67s

# 获取pod详细信息
# kubectl describe pods -l app=nginx

查看Pod日志

1
kubectl logs $POD_NAME

进入Pod中

1
2
3
4
5
# 查看环境
# kubectl exec $POD_NAME env

$ kubectl exec -ti nginx-fcb945956-t8bqq /bin/bash

获取deployment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
$ kubectl get deployment

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           18m


# 获取描述
$ kubectl describe deployment

Name:                   nginx
Namespace:              default
CreationTimestamp:      Wed, 15 May 2019 23:29:23 +0800
Labels:                 app=nginx
                        env=dev
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=nginx,env=dev
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx
           env=dev
  Containers:
   nginx:
    Image:        nginx:1.16
    Port:         8081/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-fcb945956 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  18m   deployment-controller  Scaled up replica set nginx-fcb945956 to 1

获取svc(services)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# kubectl get svc
$ kubectl get services

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        3h8m
nginx        NodePort    10.105.192.17   <none>        80:31199/TCP   5m9s

# 获取详情
$ kubectl describe services

Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                10.96.0.1
Port:              https  443/TCP
TargetPort:        8443/TCP
Endpoints:         192.168.99.100:8443
Session Affinity:  None
Events:            <none>


Name:                     nginx
Namespace:                default
Labels:                   app=nginx
                          env=dev
Annotations:              <none>
Selector:                 app=nginx,env=dev
Type:                     NodePort
IP:                       10.105.192.17
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31199/TCP
Endpoints:                172.17.0.5:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

使用Minikube获取服务访问地址

1
2
3
4
5
# 获取服务url
$ minikube service nginx --url

# 访问测试
$ curl $(minikube service nginx --url)

删除服务

1
2
3
4
5
6
7
8
# kubectl delete deployments --all
$ kubectl delete deployments -l app=nginx

# kubectl delete pods --all
$ kubectl delete pods -l app=nginx

# 删除service
kubectl delete service -l app=nginx

停止Minikube

1
$ minikube stop

卸载Minikube

1
2
$ minikube delete 
$ rm -rf ~/.minikube # 删除配置

可能遇到的问题

kube-proxy无法正常启动
可能造成的影响有minikube dashboard --alsologtostderr -v=1 启动dashboard发现报错503

通过以下命令查看日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 获取所有运行的pod
$ kubectl get pods --all-namespaces

NAMESPACE              NAME                                        READY   STATUS             RESTARTS   AGE
default                nginx                                       1/1     Running            10         8d
kube-system            coredns-54d67798b7-q7css                    0/1     Running            10         8d
kube-system            etcd-minikube                               1/1     Running            11         8d
kube-system            kube-apiserver-minikube                     1/1     Running            11         8d
kube-system            kube-controller-manager-minikube            1/1     Running            11         8d
kube-system            kube-proxy-97lq4                            0/1     CrashLoopBackOff   93         8d
kube-system            kube-scheduler-minikube                     1/1     Running            11         8d
kube-system            storage-provisioner                         0/1     CrashLoopBackOff   81         8d
kubernetes-dashboard   dashboard-metrics-scraper-c95fcf479-65spj   1/1     Running            10         8d
kubernetes-dashboard   kubernetes-dashboard-6cff4c7c4f-46x4k       0/1     CrashLoopBackOff   79         8d


# 查看proxy日志
$ kubectl logs -f kubernetes-dashboard-6cff4c7c4f-46x4k -n kube-system

W0207 06:50:25.361857       1 proxier.go:661] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0207 06:50:25.363017       1 proxier.go:661] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0207 06:50:25.364149       1 proxier.go:661] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0207 06:50:25.365165       1 proxier.go:661] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
I0207 06:50:25.383182       1 node.go:172] Successfully retrieved node IP: 192.168.49.2
I0207 06:50:25.383211       1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.49.2), assume IPv4 operation
W0207 06:50:25.401386       1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0207 06:50:25.401514       1 server_others.go:185] Using iptables Proxier.
I0207 06:50:25.401820       1 server.go:650] Version: v1.20.0
I0207 06:50:25.402289       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 524288
F0207 06:50:25.402321       1 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied

修复：

1
2
3
4
5
6
7
sysctl net.netfilter.nf_conntrack_max  # 查看限制
sysctl net.netfilter.nf_conntrack_count # 查看当前是否超限

# 修改限制
sudo echo net.ipv4.netfilter.ip_conntrack_max=524288 >> /etc/sysctl.conf
# 刷新
sudo sysctl -p

参考

Minikube Document

K8S官网文档

Minikube - Kubernetes本地实验环境

Minikube：使用 Kubernetes 进行本地开发

Kubernetes基础：查看状态、管理服务