# Create OSEv3 group that contains the masters and nodes groups [OSEv3:children] masters nodes etcd nfs
# Set variables common for all OSEv3 hosts [OSEv3:vars] # SSH user, this user should allow ssh based auth without requiring a password ansible_ssh_user=root openshift_deployment_type=origin openshift_release=3.6.0 openshift_disable_check=disk_availability,docker_storage,memory_availability,docker_image_availability,package_availability,package_version openshift_metrics_install_metrics=true openshift_master_default_subdomain=apps.oc.tw openshift_logging_install_logging=true openshift_master_cluster_public_hostname=master.oc.tw openshift_hosted_registry_storage_kind=nfs openshift_hosted_registry_storage_access_modes=['ReadWriteMany'] openshift_hosted_registry_storage_nfs_directory=/exports openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)' openshift_hosted_registry_storage_volume_name=registry openshift_hosted_registry_storage_volume_size=10Gi openshift_logging_storage_kind=nfs openshift_logging_storage_access_modes=['ReadWriteOnce'] openshift_logging_storage_nfs_directory=/exports openshift_logging_storage_nfs_options='*(rw,root_squash)' openshift_logging_storage_volume_name=logging openshift_logging_storage_volume_size=10Gi openshift_metrics_storage_kind=nfs openshift_metrics_storage_access_modes=['ReadWriteOnce'] openshift_metrics_storage_nfs_directory=/exports openshift_metrics_storage_nfs_options='*(rw,root_squash)' openshift_metrics_storage_volume_name=metrics openshift_metrics_storage_volume_size=10Gi
# host group for masters [masters] master.oc.tw openshift_public_hostname=master.oc.tw
# host group for nodes, includes region info [nodes] master.oc.tw openshift_public_hostname=master.oc.tw node1.oc.tw openshift_node_labels="{'region':'infra'}" openshift_public_hostname=node1.oc.tw node2.oc.tw openshift_node_labels="{'region':'primary'}" openshift_public_hostname=node2.oc.tw node3.oc.tw openshift_node_labels="{'region':'primary'}" openshift_public_hostname=node3.oc.tw
run ansible-playbook -i ./hosts ~/openshift-ansible/playbooks/byo/config.yml to install
run ansible-playbook -i ./hosts ~/openshift-ansible/playbooks/adhoc/uninstall.yml to uninstall. (ensure configurations are all removed, by checking /etc/origin folder on every node)
NetworkManager must be installed
install NetworkManager on each node: run yum install -y NetworkManager && systemctl enable NetworkManager && systemctl start NetworkManager and hostname master.oc.tw to correct the hostname
pay attention to master/node restarting, since it will reconfigure the hostname, quickly run hostname master.oc.tw when you found out that (maybe configure NetworkManager to assign a static hostname will work as well)
Cannot find file /etc/origin/node/resolv.conf
Create resolv.conf on each node: cp /etc/resolv.conf /etc/origin/node/resolv.conf
Cannot resolve external dns (eg. github.com) from container
edit file /etc/dnsmasq.d/origin-dns.conf to add server=10.202.129.100
run cp -v /etc/origin/master/admin.kubeconfig .kube/config to copy the config to your local
Common ways to debug
systemctl status *.service
journalctl -xe
journalctl --unit dnsmasq
nfs pv configuration
Errors from oc describe pv pv0001
Recycle failed: unexpected error creating recycler pod: pods “recycler-for-pv0001” is forbidden: service account openshift-infra/pv-recycler-controller was not found, retry after the service account is created
Create sa: oc create sa pv-recycler-controller -n openshift-infra
EFK stack use kubernetes DaemonSet to schecule logging-fluentd-xx pod on every node to collect container logs.
List DaemonSet oc get ds and make sure the pods count is the same as node count
Verify the DaemonSet settings oc edit ds/logging-fluentd. Check nodeSelector property.
Rsh into one of the fluentd pods, and check run.sh to find what is been done.
Add environment variables to help debug, edit the DaemonSet oc edit ds/logging-fluentd and add environment variables ENABLE_MONITOR_AGENT=true;ENABLE_MONITOR_AGENT=true (Refer here)
Common management tasks
Login as admin: ssh into master and you’ll be admin directly when you run oc commands
Add user through htpasswd: run htpasswd -b /etc/origin/master/htpasswd admin admin on master