cheatsheet-pks-A4
cheatsheet-pks-A4 copied to clipboard
Published
20 hours ago •
dennyzhang
dennyzhang
CheatSheet For PKS: Pivotal Kubernetes Service
- PKS CheatSheet :Cloud:
:PROPERTIES:
:type: kubernetes, pks, vmware
:export_file_name: cheatsheet-pks-A4.pdf
:END:
#+BEGIN_HTML
#+END_HTML
- PDF Link: [[https://github.com/dennyzhang/cheatsheet-pks-A4/blob/master/cheatsheet-pks-A4.pdf][cheatsheet-pks-A4.pdf]], Category: [[https://cheatsheet.dennyzhang.com/category/cloud/][Cloud]]
- Blog URL: https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4
- Related posts: [[https://cheatsheet.dennyzhang.com/cheatsheet-vmware-A4][CheatSheet: VMware Products]], [[https://cheatsheet.dennyzhang.com/cheatsheet-openshift-A4][OpenShift CheatSheet]], [[https://github.com/topics/denny-cheatsheets][#denny-cheatsheets]]
File me [[https://github.com/dennyzhang/cheatsheet.dennyzhang.com/issues][Issues]] or star [[https://github.com/dennyzhang/cheatsheet.dennyzhang.com][this repo]]. ** PKS Reference | Name | Summary | |-----------+---------------------------------------------------------------------------------------------| | YouTube | [[https://www.youtube.com/playlist?list=PL7bmigfV0EqQzsvOcT8KYfulg-lpNsooC][YouTube: PKS Demos and Webcasts]], [[https://www.youtube.com/watch?v=d_NuJ7skpuY&index=19&list=PL7bmigfV0EqQzsvOcT8KYfulg-lpNsooC][YouTube: PKS overview]] | | Reference | [[https://network.pivotal.io/][pivnet download]], [[https://bosh.io/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent][Download stemcells]] | | Reference | [[https://docs.pivotal.io/runtimes/pks/1-2/index.html][PKS Documentation]], [[https://docs.pivotal.io/runtimes/pks/1-2/concepts.html][PKS Concepts]], [[https://docs.pivotal.io/runtimes/pks/1-2/release-notes.html][PKS 1.2 Release Notes]], [[https://docs.pivotal.io/runtimes/pks/1-2/managing.html][Managing PKS 1.2]], [[https://www.virtuallyghetto.com/category/cloud-native][PKS personal blog]] | | Reference | [[https://cheatsheet.dennyzhang.com/cheatsheet-infra-A4][CheatSheet: IT Infrastructure Fundamentals]], [[https://cheatsheet.dennyzhang.com/category/vmware][VMware CheatSheet]] | | Reference | [[https://cheatsheet.dennyzhang.com/cheatsheet-openshift-A4][OpenShift CheatSheet]], [[https://cheatsheet.dennyzhang.com/cheatsheet-rancher-A4][Rancher CheatSheet]] | | Reference | [[https://cheatsheet.dennyzhang.com/cheatsheet-bosh-A4][Bosh CheatSheet]], [[https://cheatsheet.dennyzhang.com/cheatsheet-tile-A4][Tile CheatSheet]], [[https://cheatsheet.dennyzhang.com/cheatsheet-uaa-A4][UAA CheatSheet]], [[https://cheatsheet.dennyzhang.com/cheatsheet-vmware-A4][CheatSheet: VMware Products]] |
[[https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4][https://raw.githubusercontent.com/dennyzhang/cheatsheet-pks-A4/master/pks-highlevel.png]]
** PKS Key Components
| Name | Summary |
|-----------------------------------+---------------------------------------------------------------------|
| Computing virtualization | [[https://docs.vmware.com/en/VMware-vSphere/index.html][Link: VMware vSphere Documentation]] |
| SDN networking | [[https://docs.vmware.com/en/VMware-NSX-T/2.2/com.vmware.nsxt.ncp_kubernetes.doc/GUID-52A92986-0FDF-43A5-A7BB-C037889F7559.html][NCP: CNI for Vmware NSX-T]] |
| VM/Cluster lifecycle management | [[https://cheatsheet.dennyzhang.com/cheatsheet-bosh-A4][Link: BOSH CHEATSHEET]], [[https://github.com/cloudfoundry/bosh][Github: Bosh]] |
| Node healing | [[https://cheatsheet.dennyzhang.com/cheatsheet-bosh-A4][Link: BOSH CHEATSHEET]] |
| Container optimized OS | [[https://bosh.cloudfoundry.org/stemcells/][CloudFoundry Stemcell]], [[https://bosh.cloudfoundry.org/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent][bosh vsphere ubuntu stemcell]] |
| Container runtime | [[https://docs.docker.com/engine/reference/commandline/dockerd/][dockerd]] |
| Container Image Compliance Scan | Harbor Clair |
| Docker image registry | [[https://goharbor.io/][VMware Harbor]] |
| Packages k8s cluster orchestrator | [[https://docs-cfcr.cfapps.io/][CFCR/Kubo]], [[https://github.com/cloudfoundry-incubator/cfcr-etcd-release][cfcr-etcd-release]], [[https://github.com/cloudfoundry-incubator/kubo-release#option-2-three-masters][GitHub: kubo-release]], [[https://github.com/cloudfoundry-incubator/kubo-deployment][kubo-deployment]] |
| Component Packaging | [[https://cheatsheet.dennyzhang.com/cheatsheet-tile-A4][CheatSheet: Cloudfoundry Tile & OpsManager]] |
| CSI for persistent volume | [[https://vmware.github.io/hatchway/#overview][GitHub: hatchway]] |
| Reference | [[https://cheatsheet.dennyzhang.com/cheatsheet-openshift-A4][Link: OpenShift Key Components]], [[https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4][Link: PKS Key Components]] |
| Reference | [[https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4][PKS CheatSheet]], [[https://cheatsheet.dennyzhang.com/cheatsheet-bosh-A4][Bosh CheatSheet]], [[https://cheatsheet.dennyzhang.com/cheatsheet-tile-A4][Tile CheatSheet]], [[https://cheatsheet.dennyzhang.com/cheatsheet-uaa-A4][UAA CheatSheet]] |
** VMware Product Integrations
| Name | Summary |
|------------------------------+----------------------------------------------|
| wavefront | [[https://www.youtube.com/watch?v=NAOUUSr9HDU&t=9s][YouTube: PKS and VMware Wavefront]] |
| log insight | [[https://www.youtube.com/watch?v=h_99uAgopAQ&t=2s][YouTube: PKS and VMware vRealize Log Insight]] |
| vrops | [[https://www.youtube.com/watch?v=YHpI_DcBlDM&list=PLrFo2o1FG9n4q6F9hjK1-OxI-3s7luhmJ][YouTube: VMware vRealize Operations]] |
| vcd (VMware vCLoud Director) | [[https://www.youtube.com/watch?v=95Pgh0QTQnE][YouTube: VMware vCloud Director Overview]] |
#+BEGIN_HTML
#+END_HTML
** PKS cli
| Name | Command |
|--------------------------------+------------------------------------------------------------------------------------|
| Check cli version | =pks --version= |
| List all pks clusters | =pks clusters= |
| Create cluster | =pks create-cluster
#+END_HTML
** PKS Troubleshooting
| Name | Summary |
|---------------------------------------------+------------------------------------------------------------------|
| Log files in pks vms | =/var/vcap/sys/log= |
| How to run pks cli commands | [[https://github.com/dennyzhang/cheatsheet-pks-A4/blob/master/run-pks-cli.md][run-pks-cli.md]] |
| How to run bosh cli commands, like bosh ssh | [[https://github.com/dennyzhang/cheatsheet-pks-A4/blob/master/run-bosh-cli.md][run-bosh-cli.md]] |
| How to run kubectl command | [[https://github.com/dennyzhang/cheatsheet-pks-A4/blob/master/run-kubectl-in-pks.md][run-kubectl-in-pks.md]] |
| How PKS supports k8s master HA | [[https://github.com/cloudfoundry-incubator/kubo-release#option-2-three-masters][GitHub: kubo-release]], [[https://github.com/cloudfoundry-incubator/cfcr-etcd-release/blob/60ce4836faa06a0de6781ec2ce1a0c34a4ea35f2/jobs/etcd/templates/bin/post-start.erb#L13-L19][GitHub: cfcr-etcd-release]] |
| Workflow of how PKS creates a k8s cluster | [[https://docs.pivotal.io/runtimes/pks/1-2/create-cluster.html][Link: Create a Kubernetes Cluster]] |
| How airgap integration tests are enforced | For each node, load specific iptable rules. [[https://github.com/dennyzhang/cheatsheet-pks-A4/blob/master/airgap-iptable.rules][airgap-iptable.rules]] |
| Reference | [[https://docs.pivotal.io/pivotalcf/2-3/customizing/troubleshooting-diagnostics.html][Link: PKS Troubleshoot]] |
** Deployment with NSX-T + NAT [[https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4][https://raw.githubusercontent.com/dennyzhang/cheatsheet-pks-A4/master/pks-nsxt-nat.png]] ** Deployment with NSX-T + No-NAT + vswitch [[https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4][https://raw.githubusercontent.com/dennyzhang/cheatsheet-pks-A4/master/pks-nsxt-no-nat-virtual-switch.png]] ** Deployment with NSX-T + No-NAT + logical switch [[https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4][https://raw.githubusercontent.com/dennyzhang/cheatsheet-pks-A4/master/pks-nsxt-no-nat-logical-switch.png]] ** PKS Assumptions | Name | Summary | |-------------------------------------------+------------------------------------------------------------------------------------| | Multiple instances of K8S | One PKS Multiple K8S clusters. Use k8s clusters or k8s namespace for multi-tenancy | | No mixed versions | Versions must match between master and node hosts, excluding upgrade | | Support customization mainly at PKS level | Not at k8s clusters or k8s namespace level | | No hybrid cloud providers | Support multi-clouds, but only can't mix | | Don't run user workload in k8s master VMs | Avoid messing up k8s control plane | | Node roles | Master nodes, worker nodes, and etcd nodes. | ** What PKS Adds to Kubernetes | Name | Summary | |------------------------------------------------------+--------------------------------| | Secure multi-tenant ingress | NSX-T | | Secure container registry | VMware Harbor | | Rolling upgrades to cluster infrastructure | IaaS: bosh VM upgrade | | Cluster provisioning and scaling | IaaS: VM lifecycle management | | Monitoring and recovery of cluster VMs and processes | IaaS: VM lifecycle management | | Embedded, hardened operating system | Linux release for OS hardening | | Log sink | K8S Namespace multi-tenancy | ** PKS Challenges & Future Opportunities | Name | Summary | |---------------------------------+------------------------------------------------------------------------------| | Faster for typical use cases | Create k8s clusters, resize k8s cluster, create pods, etc | | Tile & OpsManager is not agile | It slows down everything. The development, testing and deployment. | | Extend PKS API layer | Easy to add more functionalities for PKS admins | | UX of PKS CLI | The usage of pks cli could be more intuitive | | Improve PKS control panel HA | Online rolling upgrade for opsmanager, uaa, pks api, etc | | Better storage support of PV | HA for PV, and support more CSI providers | | Cleanup for stale resources | When operations have failed, need to do the cleanup in a safe way | | More built-in security supports | PKS supports most common security enhancements, but it doesn't provides them | ** PKS Strengths | Name | Summary | |---------------------------+-----------------------------------------------------------------------------| | Kubernetes Federation | Multiple clusters on-demand. Not only one kubernetes cluster for your infra | | End-to-end integration | Monitoring and logging works out of box | | VM LCM: auto healing | VM health check and auto-replacement | | Less vendor lock-in | Vanilla Kubernetes; Any infra; Any OS | | Networking with NSX-T | Advanced CNI | | Image registry & security | Image sign, audit, replication; vulnerabilities scan | ** Deployment Diagram | Name | Summary | |-----------------------------------+-----------------| | Bosh director vm | VM manager | | Ops manager vm | Package manager | | PKS API server vm | See below | | Build-in process in k8s master vm | See below | | Build-in process in k8s worker vm | See below | ** PKS footprint: in control panel
- Get process list in pks 1.2.0: ssh to the pks api vm, then =sudo monit summary= | Name | Memory (RES) | |----------------------------------+--------------| | =pks-api= | 1 GB | | =uaa= | 500 MB | | =mysqld= | 500 MB | | =pks-nsx-t-osb-proxy= | 25 MB | | =telemetry= | 25 MB | | =bosh-agent= | 17 MB | | =bosh-dns= | 16 MB | | =on-demand-service-broker= | 16 MB | | =event-emitter= | 10 MB | | =galera-healthcheck= | 7 MB | | =bosh-dns-healthcheck= | 6 MB | | =cf-mysql-cluster-health-logger= | 6 MB | | =gra-log-purger-executable= | 2 MB | ** PKS footprint: in k8s master vms
- Get process list in pks 1.2.0: ssh to k8s master vm, then =sudo monit summary= | Name | Summary | |-----------------------------+---------| | =etcd= | 120 MB | | =kube-apiserver= | 520 MB | | =kube-controller-manager= | 100 MB | | =kube-scheduler= | 35 MB | | =blackbox= syslog | 530 MB | | =fluentd= | 100 MB | | =ncp= | 70 MB | | =bosh-dns= | 19 MB | | =bosh-agent= | 15 MB | | =bosh-dns-nameserverconfig= | 5 MB | | =bosh-dns-health= | 10 MB | ** PKS footprint: in k8s worker vms
- Get process list in pks 1.2.0: ssh to the k8s worker vm, then =sudo monit summary= | Name | Summary | |---------------------------------+---------| | =kube-proxy= | 30 MB | | =kubelet= | 100 MB | | =docker= | 70 MB | | =fluentd= | 180 MB | | =cadvisor= | 85 MB | | =blackbox= syslog | 60 MB | | =metrics-server= | 36 MB | | =ovs-vswitchd= open vSwitch | 35 MB | | =bosh-dns= | 20 MB | | =bosh-agent= | 18 MB | | =bosh-dns-health= | 7 MB | | =bosh-dns-namesever= | 5 MB | | =ovsdb-server= vSwitch database | 5 MB | | =nsx-node-agent= | 3 MB | | =nsx_kube_proxy= | 3 MB | ** PKS CLI Online Help #+BEGIN_EXAMPLE [ec2-user@ip-172-31-33-176 ~]$ pks --help
The Pivotal Container Service (PKS) CLI is used to create, manage, and delete Kubernetes clusters. To deploy workloads to a Kubernetes cluster created using the PKS CLI, use the Kubernetes CLI, kubectl.
Version: 1.1.1-build.8
Usage: pks [command]
Available Commands: cluster View the details of the cluster clusters Show all clusters created with PKS create-cluster Creates a kubernetes cluster, requires cluster name, an external host name, and plan delete-cluster Deletes a kubernetes cluster, requires cluster name get-credentials Allows you to connect to a cluster and use kubectl help Help about any command login Log in to PKS logout Log out of PKS plans View the preconfigured plans available resize Increases the number of worker nodes for a cluster
Flags: -h, --help help for pks --version version for pks
Use "pks [command] --help" for more information about a command. #+END_EXAMPLE ** More Resources https://docs.pivotal.io/runtimes/pks/1-2/index.html
License: Code is licensed under [[https://www.dennyzhang.com/wp-content/mit_license.txt][MIT License]].
#+BEGIN_HTML
- org-mode configuration :noexport: #+STARTUP: overview customtime noalign logdone showall #+DESCRIPTION: #+KEYWORDS: #+LATEX_HEADER: \usepackage[margin=0.6in]{geometry} #+LaTeX_CLASS_OPTIONS: [8pt] #+LATEX_HEADER: \usepackage[english]{babel} #+LATEX_HEADER: \usepackage{lastpage} #+LATEX_HEADER: \usepackage{fancyhdr} #+LATEX_HEADER: \pagestyle{fancy} #+LATEX_HEADER: \fancyhf{} #+LATEX_HEADER: \rhead{Updated: \today} #+LATEX_HEADER: \rfoot{\thepage\ of \pageref{LastPage}} #+LATEX_HEADER: \lfoot{\href{https://github.com/dennyzhang/cheatsheet-pks-A4}{GitHub: https://github.com/dennyzhang/cheatsheet-pks-A4}} #+LATEX_HEADER: \lhead{\href{https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4}{Blog URL: https://cheatsheet.dennyzhang.com/cheatsheet-pks-A4}} #+AUTHOR: Denny Zhang #+EMAIL: [email protected] #+TAGS: noexport(n) #+PRIORITIES: A D C #+OPTIONS: H:3 num:t toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t #+OPTIONS: TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc #+EXPORT_EXCLUDE_TAGS: exclude noexport #+SEQ_TODO: TODO HALF ASSIGN | DONE BYPASS DELEGATE CANCELED DEFERRED #+LINK_UP: #+LINK_HOME:
-
--8<-------------------------- separator ------------------------>8-- :noexport:
- DONE pks errands :noexport: CLOSED: [2019-02-03 Sun 22:32] ** pks errand #+BEGIN_EXAMPLE kubo@jumper:~$ bosh -d pivotal-container-service-69c3e199ae8c7bf63cd0 errands Using environment '30.0.0.11' as client 'ops_manager'
Using deployment 'pivotal-container-service-69c3e199ae8c7bf63cd0'
Name delete-all-clusters pks-nsx-t-precheck smoke-tests upgrade-all-service-instances wavefront-alert-creation wavefront-alert-deletion
6 errands
Succeeded #+END_EXAMPLE ** k8s errand #+BEGIN_EXAMPLE kubo@jumper:~$ bosh -d service-instance_5f2a64e6-ad62-4147-a85e-13213e922876 errands Using environment '30.0.0.11' as client 'ops_manager'
Using deployment 'service-instance_5f2a64e6-ad62-4147-a85e-13213e922876'
Name apply-addons apply-specs drain-cluster smoke-tests telemetry-agent wavefront-proxy-errand
6 errands
Succeeded #+END_EXAMPLE
- DONE pks tile example :noexport: CLOSED: [2018-10-19 Fri 16:46] ** pks tile: with syslog forwarder enabled :noexport: #+BEGIN_EXAMPLE instance_groups:
- azs:
- az-1 env: bosh: password: $6$7314c7e2e5596617$x9X8OmjaW2/gylHt5L1YympigKkbGf6shDTmwudeqj6kYXUe8elQAJah5fIfUL6eVyIOXAUJI/fBemCcXALkE. instances: 1 jobs:
- consumes: {} name: service-adapter properties: deployment: broker_deployment_name: pivotal-container-service-dda71dbb88455ace2ade director_url: https://30.0.0.11:25555 kubo_odb_ca: ((kubo_odb_ca.certificate)) kubo_odb_ca_2018: ((kubo_odb_ca_2018.certificate)) nsxt: upgrade_defaults: nsxt_fip_pool_ids: [] nsxt_lb_service_id: "" nsxt_lb_service_size: "" nsxt_pod_enable_snat: true nsxt_pod_ip_block_ids: [] nsxt_pod_subnet_prefix: 24 nsxt_t0_router_id: "" syslog: address: 127.0.0.1 ca_cert: null forward_files: true migration: disabled: false permitted_peer: null port: 53 tls_enabled: true transport: tcp provides: {} release: kubo-service-adapter
- consumes: broker: from: proxy-broker name: pks-api properties: pks: db_password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_pks_db_password.value)) fqdn: api.pks.local internal_tls: certificate: ((pks_api_internal.certificate)) private_key: ((pks_api_internal.private_key)) password: ((pks_api_basicauth.password)) pks_client_secret: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pks_api_uaa_client.value)) telemetry: authenticationMode: service_account enabled: true eventEmitterBaseUrl: http://localhost:8888 tls: certificate: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pivotal-container-service/pks_tls.cert_pem)) private_key: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pivotal-container-service/pks_tls.private_key_pem)) uaa_service_admin_client_id: service_admin_client uaa_service_admin_client_secret: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pks_services_admin_uaa_client.value)) username: ((pks_api_basicauth.username)) provides: pks_api: as: pks_api_http pks_api_shared: as: pks_api_shared_http shared: true pks_uaa_service_admin_client: as: pks_uaa_service_admin_client shared: true release: pks-api
- consumes: {} name: bosh-update-config properties: bosh: authentication: uaa: client_id: pivotal-container-service-dda71dbb88455ace2ade client_secret: ((/opsmgr/director/pivotal-container-service-dda71dbb88455ace2ade/uaa_client_secret.value)) url: https://30.0.0.11:8443 root_ca_cert: | -----BEGIN CERTIFICATE----- MIIDUDCCAjigAwIBAgIUb/i3hdeWUxDK+sxBzJaFy7jr11swDQYJKoZIhvcNAQEL BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTkwMzEzMTgw NTAzWhcNMjMwMzE0MTgwNTAzWjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2 b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN0unMJA288egzX1 Gbq2MpdQlQ7Wghj+BMn4SELBqJVK+gLhn50dbRy75VNyyS4P65qCXyObMVSa0Ytn gSTLQuUBxbtf7mYeZbeaI0h6VGSrSI98xI9YF20FV/sl3UYuZ/GCFZplEuenYFME Acr/r1vA79UOtJqkEvGeQCXASC0U5kyCXtiIlzXlVws1nksDnZkZVAW1tQTqGE5P /QNtjR94es39LW2VAlEwTE0ESzt4BbQtk24z8W6sLUMvraEBt35rEdCHOcfe4dRK EWXd+MVYPMF3NqM98EEoHFNsR0lRM2UNPZfROceJDmGK3ik0M2C4Gk8Ztd5pFLWD yRHxq9UCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUIs+qfP0G6qAiA2b6ps6JrqTtlt8w HwYDVR0jBBgwFoAUIs+qfP0G6qAiA2b6ps6JrqTtlt8wHQYDVR0lBBYwFAYIKwYB BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG MA0GCSqGSIb3DQEBCwUAA4IBAQBWxExob/oBCKuGH/ZhJvfkOYVy9tLeNlbZP847 24hlUu4P11e26QMlYaM09B7Hq5rkqTzlcEa2vOOD33ZNPipCX14ILmSQncAQeUor b9vphE+Q8hjHcXGrqeNE8K2y7L7/gSmrLSXQIOkARkCVGG5QsHxpFNJRZxNc3+HH MBZcdkjKnmh5USBlP8qoJkjrbBZuV2j7GQYhs5mILfNvxAwf1EnzaNd7UYYiqTCK l2SjfBJKK4n5qz5DrHMfKy9cDduVUlPDIo/WlMvbjOoZ2mi/Cyk2JaNnUZ58Mso8 rTDLGr6pxhWwXGmj/2J0zDIx4CZ4U+yrbCBwcLnb6CyzYrD8 -----END CERTIFICATE----- url: https://30.0.0.11:25555 cloud_config: vm_extensions: - cloud_properties: vmx_options: disk.enableUUID: "1" name: disk_enable_uuid cloud_config_name: pivotal-container-service-dda71dbb88455ace2ade provides: {} release: pks-api
- consumes: {} name: broker provides: broker: as: odb-broker release: on-demand-service-broker
- consumes:
broker:
from: odb-broker
name: pks-nsx-t-osb-proxy
properties:
bosh:
authentication:
uaa:
client_id: pivotal-container-service-dda71dbb88455ace2ade
client_secret: ((/opsmgr/director/pivotal-container-service-dda71dbb88455ace2ade/uaa_client_secret.value))
cloud_config_dns: 192.168.115.1
cloud_config_prefix: pks
log_level: INFO
root_ca_cert: |
-----BEGIN CERTIFICATE-----
MIIDUDCCAjigAwIBAgIUb/i3hdeWUxDK+sxBzJaFy7jr11swDQYJKoZIhvcNAQEL
BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTkwMzEzMTgw
NTAzWhcNMjMwMzE0MTgwNTAzWjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2
b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN0unMJA288egzX1
Gbq2MpdQlQ7Wghj+BMn4SELBqJVK+gLhn50dbRy75VNyyS4P65qCXyObMVSa0Ytn
gSTLQuUBxbtf7mYeZbeaI0h6VGSrSI98xI9YF20FV/sl3UYuZ/GCFZplEuenYFME
Acr/r1vA79UOtJqkEvGeQCXASC0U5kyCXtiIlzXlVws1nksDnZkZVAW1tQTqGE5P
/QNtjR94es39LW2VAlEwTE0ESzt4BbQtk24z8W6sLUMvraEBt35rEdCHOcfe4dRK
EWXd+MVYPMF3NqM98EEoHFNsR0lRM2UNPZfROceJDmGK3ik0M2C4Gk8Ztd5pFLWD
yRHxq9UCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUIs+qfP0G6qAiA2b6ps6JrqTtlt8w
HwYDVR0jBBgwFoAUIs+qfP0G6qAiA2b6ps6JrqTtlt8wHQYDVR0lBBYwFAYIKwYB
BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG
MA0GCSqGSIb3DQEBCwUAA4IBAQBWxExob/oBCKuGH/ZhJvfkOYVy9tLeNlbZP847
24hlUu4P11e26QMlYaM09B7Hq5rkqTzlcEa2vOOD33ZNPipCX14ILmSQncAQeUor
b9vphE+Q8hjHcXGrqeNE8K2y7L7/gSmrLSXQIOkARkCVGG5QsHxpFNJRZxNc3+HH
MBZcdkjKnmh5USBlP8qoJkjrbBZuV2j7GQYhs5mILfNvxAwf1EnzaNd7UYYiqTCK
l2SjfBJKK4n5qz5DrHMfKy9cDduVUlPDIo/WlMvbjOoZ2mi/Cyk2JaNnUZ58Mso8
rTDLGr6pxhWwXGmj/2J0zDIx4CZ4U+yrbCBwcLnb6CyzYrD8
-----END CERTIFICATE-----
url: https://30.0.0.11:25555
create_network_with_lb: false
enabled: true
fip_address_parameter: nsxt_fip_address
generate_lb_name: true
kubernetes_master_host_parameter: kubernetes_master_host
lb_service_id_parameter: nsxt_lb_service_id
log_level: INFO
network_name_parameter: nsxt_network_name
nsxt:
ca_cert: |
-----BEGIN CERTIFICATE-----
MIIDZDCCAkygAwIBAgIGAWl9WmgwMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM
G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww
CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ
UGFsbyBBbHRvMB4XDTE5MDMxNDE3NTc1MFoXDTI0MDMxMjE3NTc1MFowczEkMCIG
A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh
cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD
VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCK
OhRAYMDiXgobQOEW6uT4/vFWjWYErkkgykUDm8Tihy7AQ4HEq882AQV3Fski9D9O
fxb8HQZU54IltofbCjsmpPJ3cCCmm8FB3sOYCS0/ss/JCpzEdZXC9gbGAv6j81gq
80Xcw77OMG7ZK2v8c9mTMDbPy0c7n1/ebPY1ZCmD9mRqJ+SeDfKmvP/cecnhgbYD
G7rcqbFAEGY+UaIhsOujZs1oge3SetkI1+ql6ODGRnZW94o7gp5NHqOtFjwMKFCH
Nj0oTBQwbS39x+Z2BUH5LYQSNtEQy8hg5W+BgP/iKG3I342c+yEk6MtKsS7mBM3B
xhckOtZzRWhM43onQMfNAgMBAAEwDQYJKoZIhvcNAQELBQADggEBADfjaoHg+JB0
DMTmW2fQwR3ArEkOh8tHvU5Hjip6uYHi16kL0ZrX1rCn6dbpiuT3GsDX+in/+FWC
t3N02lzBRlvxITjL7TgPZ4C0T9FSUlqq8w1wJjPpLfwpY4pQTrZkV2i7MNczhZvT
TkQQ4H6p+iV+H9KSbgx1yGG9nGj19iq9C2oWMiQ7s6NigtFN0zn1/QqT8ZVcxhQu
LMnsjCH0yWQjSc4bz+10qJDZposkFwzzUOz/y/Mmg7qOLumIl6fZ61h8zUcJ89vF
/pyPQgNGZzB8f9rlMlqro7uDCS4mmq7qJZp4A8RH92p29IKIuJBvttV9da/9n1l6
yHsxTdMUzrM=
-----END CERTIFICATE-----
floating_ip_pool_ids:
- 9a6a336c-7fdb-4361-904f-123ff7c595a9
host: nsxmanager.pks.vmware.local
insecure: true
ip_block_id: eb65b80e-3f1f-4abc-bf4f-34666c553b81
lb_size_large_supported: true
lb_size_medium_supported: true
log_level: INFO
nat_mode: true
network_prefix: pks
pod_ip_block_id: cf16a74c-1383-4505-abe8-e48516a2fe35
superuser_cert: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/network_selector/nsx/nsx-t-superuser-certificate.cert_pem))
superuser_key: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem))
t0_router_id: 24bcaa9a-ddc0-44f3-a783-23c91d20d6e2
password: ((odb_broker_basicauth.password))
plans:
- description: 'Example: This plan will configure a lightweight kubernetes cluster.
Not recommended for production workloads.'
instance_groups:
- azs:
- az-1
- az-2
- az-3 instances: 1 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
- azs:
- az-1
- az-2
- az-3 instances: 3 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
- azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
disabled: true name: wavefront-proxy-errand
-
name: vrops-errand
-
name: vrops-registration
-
name: telemetry-agent pre_delete:
-
name: drain-cluster metadata: allow-privileged-containers: true master_instances: 1 max_worker_instances: 50 worker_instances: 3 name: Plan 1 plan_id: 8A0E21A8-8072-4D80-B365-D1F502085560 properties: addons-spec: |+
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Sinks [{"addr":"example.com:12345"}]
parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log
apiVersion: v1 kind: Service metadata: name: fluent-bit namespace: pks-system spec: selector: k8s-app: logging-agent ports: - protocol: TCP port: 24224 targetPort: forward-plugin type: ClusterIP
apiVersion: batch/v1 kind: Job metadata: labels: job: cert-generator name: cert-generator-v0.11 namespace: pks-system spec: backoffLimit: 0 template: metadata: labels: job: cert-generator spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/cert-generator:v0.12 imagePullPolicy: IfNotPresent name: cert-generator restartPolicy: Never serviceAccountName: cert-generator
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: fluent-bit resources: limits: memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true - command: - ghostunnel - server - --listen - :24224 - --target - localhost:24225 - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --allow-dns-san - event-controller - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel ports: - containerPort: 24224 name: forward-plugin volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /fluent-bit-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /fluent-bit-certs name: fluent-bit-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config - emptyDir: {} name: keystore - name: fluent-bit-certs secret: secretName: fluent-bit - name: pks-ca secret: secretName: pks-ca updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: event-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: event-controller spec: containers: - env: - name: FORWARDER_HOST value: localhost image: oratos/event-controller:v0.12 imagePullPolicy: IfNotPresent name: event-controller - command: - ghostunnel - client - --listen - localhost:24224 - --target - fluent-bit.pks-system.svc.cluster.local:24224 - --override-server-name - fluent-bit - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /event-controller-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /event-controller-certs name: event-controller-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: event-controller volumes: - emptyDir: {} name: keystore - name: event-controller-certs secret: secretName: event-controller - name: pks-ca secret: secretName: pks-ca
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.12 imagePullPolicy: IfNotPresent name: sink-controller serviceAccountName: sink-controller
-
- azs:
- description: 'Example: This plan will configure a medium sized kubernetes
cluster, suitable for more pods.'
instance_groups:
-
azs:
- az-1
- az-2
- az-3 instances: 3 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
-
azs:
- az-1
- az-2
- az-3 instances: 5 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
-
azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
disabled: true name: wavefront-proxy-errand
-
name: vrops-errand
-
name: vrops-registration
-
name: telemetry-agent pre_delete:
-
name: drain-cluster metadata: allow-privileged-containers: true master_instances: 3 max_worker_instances: 50 worker_instances: 5 name: multi-master plan_id: 58375a45-17f7-4291-acf1-455bfdc8e371 properties: addons-spec: |
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Sinks [{"addr":"example.com:12345"}]
parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log
apiVersion: v1 kind: Service metadata: name: fluent-bit namespace: pks-system spec: selector: k8s-app: logging-agent ports: - protocol: TCP port: 24224 targetPort: forward-plugin type: ClusterIP
apiVersion: batch/v1 kind: Job metadata: labels: job: cert-generator name: cert-generator-v0.11 namespace: pks-system spec: backoffLimit: 0 template: metadata: labels: job: cert-generator spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/cert-generator:v0.12 imagePullPolicy: IfNotPresent name: cert-generator restartPolicy: Never serviceAccountName: cert-generator
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: fluent-bit resources: limits: memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true - command: - ghostunnel - server - --listen - :24224 - --target - localhost:24225 - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --allow-dns-san - event-controller - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel ports: - containerPort: 24224 name: forward-plugin volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /fluent-bit-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /fluent-bit-certs name: fluent-bit-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config - emptyDir: {} name: keystore - name: fluent-bit-certs secret: secretName: fluent-bit - name: pks-ca secret: secretName: pks-ca updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: event-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: event-controller spec: containers: - env: - name: FORWARDER_HOST value: localhost image: oratos/event-controller:v0.12 imagePullPolicy: IfNotPresent name: event-controller - command: - ghostunnel - client - --listen - localhost:24224 - --target - fluent-bit.pks-system.svc.cluster.local:24224 - --override-server-name - fluent-bit - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /event-controller-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /event-controller-certs name: event-controller-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: event-controller volumes: - emptyDir: {} name: keystore - name: event-controller-certs secret: secretName: event-controller - name: pks-ca secret: secretName: pks-ca
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.12 imagePullPolicy: IfNotPresent name: sink-controller serviceAccountName: sink-controller
allow-privileged-containers: true disable_deny_escalating_exec: true max_worker_instances: 50
-
-
- null port: 3000 proxy: null username: ((odb_broker_basicauth.username)) provides: {} release: pks-nsx-t
- description: 'Example: This plan will configure a lightweight kubernetes cluster.
Not recommended for production workloads.'
instance_groups:
- consumes: {} name: mysql properties: cf_mysql: mysql: admin_password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_mysql_admin_password.value)) cluster_health: password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_mysql_cluster_health_password.value)) galera_healthcheck: db_password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_mysql_galera_healthcheck_db_password.value)) endpoint_password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_mysql_galera_healthcheck_endpoint_password.value)) seeded_databases: - name: pks password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_pks_db_password.value)) username: pks - name: uaa password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_uaa_db_password.value)) username: uaa - name: telemetry password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_telemetry_db_password.value)) username: telemetry - name: billing password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_billing_db_password.value)) username: billing provides: mysql: as: mysql release: cf-mysql
- consumes: {} name: uaa properties: encryption: active_key_label: key-1 encryption_keys: - label: key-1 passphrase: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/uaa_encryption_passphrase.value)) login: saml: activeKeyId: active-pks-saml-key keys: active-pks-saml-key: certificate: ((uaa_active_pks_saml_key_2018.certificate)) key: ((uaa_active_pks_saml_key_2018.private_key)) passphrase: "" signatureAlgorithm: SHA256 release_level_backup: true uaa: clients: admin: authorities: uaa.admin,clients.read,clients.write,clients.secret,scim.read,scim.write,clients.admin,pks.clusters.admin,pks.clusters.manage authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pks_uaa_management_admin_client.value)) pks_cli: access-token-validity: 7200 authorities: uaa.resource authorized-grant-types: password,refresh_token refresh-token-validity: 21600 scope: pks.clusters.admin,pks.clusters.manage secret: "" pks_client: access-token-validity: 86400 authorities: pks.clusters.admin,pks.clusters.manage,uaa.resource authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pks_api_uaa_client.value)) pks_cluster_client: authorities: uaa.resource authorized-grant-types: password,refresh_token scope: openid,roles secret: "" service_admin_client: authorities: clients.admin authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pks_services_admin_uaa_client.value)) jwt: policy: active_key_id: key-1 keys: key-1: signingKey: ((uaa_jwt_signing_key_1.private_key)) ldap: emailDomain: [] enabled: false externalGroupsWhitelist: - '*' groups: groupSearchFilter: member={0} profile_type: no-groups searchBase: null mailAttributeName: mail referral: follow searchBase: null searchFilter: cn={0} sslCertificate: null sslCertificateAlias: null url: null userDN: null userPassword: null port: 35684 scim: groups: pks.clusters.admin: Allows a user to admin PKS pks.clusters.manage: Allows a user to manage PKS clusters user: override: true users: - groups: - uaa.admin - pks.clusters.admin name: admin password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/uaa_admin_password.value)) sslCertificate: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pivotal-container-service/pks_tls.cert_pem)) sslPrivateKey: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pivotal-container-service/pks_tls.private_key_pem)) url: https://api.pks.local:8443 uaadb: address: 127.0.0.1 databases: - name: uaa tag: uaa db_scheme: mysql port: 3306 roles: - name: uaa password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_uaa_db_password.value)) tag: admin provides: {} release: uaa
- consumes: {} name: bbr-uaadb provides: {} release: uaa
- consumes: {} name: database-backup-restorer provides: {} release: backup-and-restore-sdk
- consumes: broker: from: proxy-broker name: upgrade-all-service-instances provides: {} release: on-demand-service-broker
- consumes: {} name: syslog_forwarder properties: syslog: address: 127.0.0.1 ca_cert: null forward_files: true migration: disabled: false permitted_peer: null port: 53 tls_enabled: true transport: tcp provides: {} release: syslog
- consumes: {} name: sink-resources-images provides: {} release: sink-resources-release
- consumes: {} name: sink-resources-images-ops-files provides: {} release: sink-resources-release
- consumes: pks_api: from: pks_api_http name: delete-all-clusters provides: {} release: pks-api
- consumes: {}
name: pks-nsx-t-precheck
properties:
floating-ip-pool-ids:
- 9a6a336c-7fdb-4361-904f-123ff7c595a9 ip-block-id: cf16a74c-1383-4505-abe8-e48516a2fe35 network-automation: true nodes-ip-block-id: eb65b80e-3f1f-4abc-bf4f-34666c553b81 nsx-t-ca-cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWl9WmgwMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE5MDMxNDE3NTc1MFoXDTI0MDMxMjE3NTc1MFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCK OhRAYMDiXgobQOEW6uT4/vFWjWYErkkgykUDm8Tihy7AQ4HEq882AQV3Fski9D9O fxb8HQZU54IltofbCjsmpPJ3cCCmm8FB3sOYCS0/ss/JCpzEdZXC9gbGAv6j81gq 80Xcw77OMG7ZK2v8c9mTMDbPy0c7n1/ebPY1ZCmD9mRqJ+SeDfKmvP/cecnhgbYD G7rcqbFAEGY+UaIhsOujZs1oge3SetkI1+ql6ODGRnZW94o7gp5NHqOtFjwMKFCH Nj0oTBQwbS39x+Z2BUH5LYQSNtEQy8hg5W+BgP/iKG3I342c+yEk6MtKsS7mBM3B xhckOtZzRWhM43onQMfNAgMBAAEwDQYJKoZIhvcNAQELBQADggEBADfjaoHg+JB0 DMTmW2fQwR3ArEkOh8tHvU5Hjip6uYHi16kL0ZrX1rCn6dbpiuT3GsDX+in/+FWC t3N02lzBRlvxITjL7TgPZ4C0T9FSUlqq8w1wJjPpLfwpY4pQTrZkV2i7MNczhZvT TkQQ4H6p+iV+H9KSbgx1yGG9nGj19iq9C2oWMiQ7s6NigtFN0zn1/QqT8ZVcxhQu LMnsjCH0yWQjSc4bz+10qJDZposkFwzzUOz/y/Mmg7qOLumIl6fZ61h8zUcJ89vF /pyPQgNGZzB8f9rlMlqro7uDCS4mmq7qJZp4A8RH92p29IKIuJBvttV9da/9n1l6 yHsxTdMUzrM= -----END CERTIFICATE----- nsx-t-host: nsxmanager.pks.vmware.local nsx-t-insecure: true nsx-t-superuser-certificate: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) nsx-t-superuser-key: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) proxy: null t0-router-id: 24bcaa9a-ddc0-44f3-a783-23c91d20d6e2 vcenter-cluster: kubo-az-1 vcenter-datacenter: kubo-dc vcenter-host: 192.168.111.79 vcenter-insecure: true vcenter-password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cloud_provider/vsphere/vcenter_master_creds.password)) vcenter-user: [email protected] provides: {} release: pks-nsx-t
- consumes: {}
name: pks-nsx-t-ops-files
properties:
fip_address_parameter: nsxt_fip_address
floating-ip-pool-ids:
- 9a6a336c-7fdb-4361-904f-123ff7c595a9 kubo-odb-ca-2018-added: true kubo-odb-ca-2018-used: true lb-created-by-proxy: false nsx-t-ca-cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWl9WmgwMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE5MDMxNDE3NTc1MFoXDTI0MDMxMjE3NTc1MFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCK OhRAYMDiXgobQOEW6uT4/vFWjWYErkkgykUDm8Tihy7AQ4HEq882AQV3Fski9D9O fxb8HQZU54IltofbCjsmpPJ3cCCmm8FB3sOYCS0/ss/JCpzEdZXC9gbGAv6j81gq 80Xcw77OMG7ZK2v8c9mTMDbPy0c7n1/ebPY1ZCmD9mRqJ+SeDfKmvP/cecnhgbYD G7rcqbFAEGY+UaIhsOujZs1oge3SetkI1+ql6ODGRnZW94o7gp5NHqOtFjwMKFCH Nj0oTBQwbS39x+Z2BUH5LYQSNtEQy8hg5W+BgP/iKG3I342c+yEk6MtKsS7mBM3B xhckOtZzRWhM43onQMfNAgMBAAEwDQYJKoZIhvcNAQELBQADggEBADfjaoHg+JB0 DMTmW2fQwR3ArEkOh8tHvU5Hjip6uYHi16kL0ZrX1rCn6dbpiuT3GsDX+in/+FWC t3N02lzBRlvxITjL7TgPZ4C0T9FSUlqq8w1wJjPpLfwpY4pQTrZkV2i7MNczhZvT TkQQ4H6p+iV+H9KSbgx1yGG9nGj19iq9C2oWMiQ7s6NigtFN0zn1/QqT8ZVcxhQu LMnsjCH0yWQjSc4bz+10qJDZposkFwzzUOz/y/Mmg7qOLumIl6fZ61h8zUcJ89vF /pyPQgNGZzB8f9rlMlqro7uDCS4mmq7qJZp4A8RH92p29IKIuJBvttV9da/9n1l6 yHsxTdMUzrM= -----END CERTIFICATE----- nsx-t-host: nsxmanager.pks.vmware.local nsx-t-insecure: true nsx-t-superuser-certificate: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) nsx-t-superuser-key: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) pod-ip-block-id: cf16a74c-1383-4505-abe8-e48516a2fe35 t0-router-id: 24bcaa9a-ddc0-44f3-a783-23c91d20d6e2 provides: {} release: pks-nsx-t
- consumes: {} name: pks-vrops-ops-files properties: vc_url: null vrops_admin: null vrops_admin_pass: null vrops_ca: null vrops_enabled: "false" vrops_ignore_hostname_verification: true vrops_insecure: true vrops_url: null provides: {} release: pks-vrops
- consumes: {} name: pks-wavefront-ops-files properties: wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: wavefront-alert-creation properties: wavefront-alert-targets: ignored wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: wavefront-alert-deletion properties: wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: pks-vrli-ops-files properties: fluentd_vrli_ca_cert: ignored fluentd_vrli_host: ignored fluentd_vrli_rate_limit_msec: ignored fluentd_vrli_skip_cert_verify: ignored fluentd_vrli_use_ssl: ignored provides: {} release: pks-vrli
- consumes: mysql: from: mysql name: telemetry-server properties: billing: db-name: billing db-password: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/cf_mysql_billing_db_password.value)) db-username: billing cloud-provider: vSphere forward: tls: certificate: ((telemetry_server_tls_2018.certificate)) private_key: ((telemetry_server_tls_2018.private_key)) hostname: telemetry.pks.internal networking: nsx pks-instance-id: pivotal-container-service-dda71dbb88455ace2ade product-version: 1.3.0-build.43 telemetry-enabled: true vac-server-url: http://35.188.203.192 vcenter-server-url: https://192.168.111.79 provides: telemetry-server: as: telemetry-server shared: true release: pks-telemetry
- consumes: {} name: telemetry-ops-files properties: telemetry-agent: billing: polling-interval-seconds: 60 telemetry: polling-interval-seconds: 600 telemetry-server: ca: certificate: | ((telemetry_ca_2018.certificate)) provides: {} release: pks-telemetry
- consumes: {} name: bpm provides: {} release: bpm
- consumes: pks_api: from: pks_api_http name: smoke-tests properties: smoke_tests: client: admin secret: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pks_uaa_management_admin_client.value)) provides: {} release: pks-api lifecycle: service name: pivotal-container-service networks:
- default:
- dns
- gateway
name: deployment-network
persistent_disk_type: "10240"
properties:
bosh:
authentication:
uaa:
client_id: pivotal-container-service-dda71dbb88455ace2ade
client_secret: ((/opsmgr/director/pivotal-container-service-dda71dbb88455ace2ade/uaa_client_secret.value))
url: https://30.0.0.11:8443
root_ca_cert: |
-----BEGIN CERTIFICATE-----
MIIDUDCCAjigAwIBAgIUb/i3hdeWUxDK+sxBzJaFy7jr11swDQYJKoZIhvcNAQEL
BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTkwMzEzMTgw
NTAzWhcNMjMwMzE0MTgwNTAzWjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2
b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN0unMJA288egzX1
Gbq2MpdQlQ7Wghj+BMn4SELBqJVK+gLhn50dbRy75VNyyS4P65qCXyObMVSa0Ytn
gSTLQuUBxbtf7mYeZbeaI0h6VGSrSI98xI9YF20FV/sl3UYuZ/GCFZplEuenYFME
Acr/r1vA79UOtJqkEvGeQCXASC0U5kyCXtiIlzXlVws1nksDnZkZVAW1tQTqGE5P
/QNtjR94es39LW2VAlEwTE0ESzt4BbQtk24z8W6sLUMvraEBt35rEdCHOcfe4dRK
EWXd+MVYPMF3NqM98EEoHFNsR0lRM2UNPZfROceJDmGK3ik0M2C4Gk8Ztd5pFLWD
yRHxq9UCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUIs+qfP0G6qAiA2b6ps6JrqTtlt8w
HwYDVR0jBBgwFoAUIs+qfP0G6qAiA2b6ps6JrqTtlt8wHQYDVR0lBBYwFAYIKwYB
BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG
MA0GCSqGSIb3DQEBCwUAA4IBAQBWxExob/oBCKuGH/ZhJvfkOYVy9tLeNlbZP847
24hlUu4P11e26QMlYaM09B7Hq5rkqTzlcEa2vOOD33ZNPipCX14ILmSQncAQeUor
b9vphE+Q8hjHcXGrqeNE8K2y7L7/gSmrLSXQIOkARkCVGG5QsHxpFNJRZxNc3+HH
MBZcdkjKnmh5USBlP8qoJkjrbBZuV2j7GQYhs5mILfNvxAwf1EnzaNd7UYYiqTCK
l2SjfBJKK4n5qz5DrHMfKy9cDduVUlPDIo/WlMvbjOoZ2mi/Cyk2JaNnUZ58Mso8
rTDLGr6pxhWwXGmj/2J0zDIx4CZ4U+yrbCBwcLnb6CyzYrD8
-----END CERTIFICATE-----
url: https://30.0.0.11:25555
disable_cf_startup_checks: true
expose_operational_errors: true
password: ((odb_broker_basicauth.password))
port: 8080
service_adapter:
path: /var/vcap/jobs/service-adapter/bin/service-adapter
service_catalog:
bindable: true
global_properties:
authorization_mode: rbac
deployments_network: service-network
iaas: vsphere
oidc: false
oidc_ca: ((/opsmgr/pivotal-container-service-dda71dbb88455ace2ade/pivotal-container-service/pks_tls.cert_pem))
oidc_client_id: pks_cluster_client
oidc_groups_claim: roles
oidc_groups_prefix: ""
oidc_issuer_url: https://api.pks.local:8443/oauth/token
oidc_username_claim: user_name
oidc_username_prefix: '-'
ops_files_paths:
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_pks_nsx_t.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/prepare_master_vm.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_fip_to_tls_certs.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/remove_flannel.yml
- ""
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_master_vms_to_nsgroup.yml
- /var/vcap/jobs/pks-wavefront-ops-files/manifests/add-wavefront-job.yml
- null
- /var/vcap/jobs/telemetry-ops-files/manifests/add-telemetry-dns.yml
- /var/vcap/jobs/telemetry-ops-files/manifests/add-telemetry-agent-image.yml
- /var/vcap/jobs/telemetry-ops-files/manifests/add-telemetry-agent-deploy-errand.yml
- /var/vcap/jobs/pks-vrops-ops-files/manifests/remove-vrops.yml
- /var/vcap/jobs/sink-resources-images-ops-files/manifests/add-sink-resources-images.yml pod_network_cidr: null proxy: null routing_mode: external service_cluster_cidr: null vcenter_dc: kubo-dc vcenter_ds: iscsi-ds-0 vcenter_ip: 192.168.111.79 vcenter_vms: pcf_vms worker_max_in_flight: 1 id: DF8EECC4-7225-42D0-8459-4A6C584314CA metadata: display_name: Kubernetes documentation_url: TBA image_url: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAAKp2lDQ1BJQ0MgUHJvZmlsZQAASImVlwdUFEkax6t7ciINDEHCkJPkDJLjkCWDqAwzhCGMw8CQzMqigmtARQQURVeSAosKiAkQBZVFMIB5QRYV5VwMiIo618Ax3N69u3v39atXv/d19b++rq56798AkHuYPF4KLAFAKjeDH+TpQo+IjKLjRgAauQhAAlCYrHSec2CgL0Biof9rfBoE0Gx/12BW69/v/9eQZMelswCAAhGOZaezUhE+i7RGFo+fAQCKjeTVszJ4s7wdYWk+UiDC5bOcMM+Nsxw7z11zY0KCXBF+AACezGTyEwAg/YHk6ZmsBESHjEbYmMvmcBE2R9iBlchE5iEj98DS1NQ1s3wUYZ3Yf9JJ+ItmrEiTyUwQ8fy7zAXejZPOS2Hm/J/L8b8jNUWwMIca0siJfK+g2fmQNatOXuMjYm6sf8ACc9jzNc1yosArdIFZ6a5RC8xmuvkssCA51HmBmfzFZzkZjJAF5q8JEulzU/x9RfpxDBHHpbsHL3A8x4OxwLmJIeELnMkJ81/g9ORgn8UxrqI8XxAkqjme7yF6x9T0xdpYzMW5MhJDvBZriBDVw45zcxfluaGi8bwMF5EmLyVwsf4UT1E+PTNY9GwGssEWOInpHbioEyhaH8ABfoAJWBlx2bP7Criu4eXwOQmJGXRn5JTE0RlcluFSuqmxiRUAs2du/pN+oM2dJYh2czG39QgA9meFQuGFxZxPGwBnCgEgDi3mtNcDINYBQE8ZS8DPnM/NbnWAAUQgDqSBPFAG6kAHGABTYAnsgBNwB94gAISASLAKsEAiSAV8kAXWgc0gHxSCPeAAKAUV4DioBqdBE2gBF0EHuA5ugX5wHzwGw2AMvAGT4BOYgSAIB1EgKiQPqUCakD5kCllDDpA75AsFQZFQDJQAcSEBtA7aChVCRVApdAyqgX6FzkMd0A1oAHoIjUDj0HvoK4yCybA0rARrwUawNewM+8Ah8Eo4AU6Dc+E8eBdcAlfCp+BmuAO+Bd+Hh+E38BQKoEgoGkoVZYCyRrmiAlBRqHgUH7UBVYAqRlWi6lFtqG7UXdQwagL1BY1FU9F0tAHaDu2FDkWz0GnoDeid6FJ0NboZ3YW+ix5BT6J/YCgYRYw+xhbDwERgEjBZmHxMMeYk5hzmGuY+ZgzzCYvF0rDaWCusFzYSm4Rdi92JPYxtwLZjB7Cj2CkcDieP08fZ4wJwTFwGLh93CHcKdwV3BzeG+4wn4VXwpngPfBSei9+CL8bX4i/j7+Bf4mcIEgRNgi0hgMAm5BB2E04Q2gi3CWOEGaIkUZtoTwwhJhE3E0uI9cRrxCfEDyQSSY1kQ1pO4pA2kUpIjaQe0gjpC1mKrEd2JUeTBeRd5CpyO/kh+QOFQtGiOFGiKBmUXZQaylXKM8pnMaqYoRhDjC22UaxMrFnsjthbcYK4priz+CrxXPFi8TPit8UnJAgSWhKuEkyJDRJlEuclhiSmJKmSJpIBkqmSOyVrJW9IvpLCSWlJuUuxpfKkjktdlRqloqjqVFcqi7qVeoJ6jTomjZXWlmZIJ0kXSp+W7pOelJGSMZcJk8mWKZO5JDNMQ9G0aAxaCm03rYk2SPsqqyTrLBsnu0O2XvaO7LTcEjknuTi5ArkGuftyX+Xp8u7yyfJ75VvknyqgFfQUlitkKRxRuKYwsUR6id0S1pKCJU1LHinCinqKQYprFY8r9ipOKSkreSrxlA4pXVWaUKYpOyknKe9Xvqw8rkJVcVDhqOxXuaLymi5Dd6an0EvoXfRJVUVVL1WB6jHVPtUZNW21ULUtag1qT9WJ6tbq8er71TvVJzVUNPw01mnUaTzSJGhaayZqHtTs1pzW0tYK19qm1aL1SltOm6Gdq12n/USHouOok6ZTqXNPF6trrZuse1i3Xw/Ws9BL1CvTu60P61vqc/QP6w8sxSy1WcpdWrl0yIBs4GyQaVBnMGJIM/Q13GLYYvjWSMMoymivUbfRD2ML4xTjE8aPTaRMvE22mLSZvDfVM2WZlpneM6OYeZhtNGs1e2eubx5nfsT8gQXVws9im0WnxXdLK0u+Zb3luJWGVYxVudWQtbR1oPVO6x4bjI2LzUabizZfbC1tM2ybbP+0M7BLtqu1e7VMe1ncshPLRu3V7Jn2x+yHHegOMQ5HHYYdVR2ZjpWOz53UndhOJ51eOus6Jzmfcn7rYuzCdznnMu1q67retd0N5ebpVuDW5y7lHupe6v7MQ80jwaPOY9LTwnOtZ7sXxsvHa6/XEEOJwWLUMCa9rbzXe3f5kH2CfUp9nvvq+fJ92/xgP2+/fX5P/DX9uf4tASCAEbAv4GmgdmBa4IXl2OWBy8uWvwgyCVoX1B1MDV4dXBv8KcQlZHfI41CdUEFoZ5h4WHRYTdh0uFt4UfhwhFHE+ohbkQqRnMjWKFxUWNTJqKkV7isOrBiLtojOjx5cqb0ye+WNVQqrUlZdWi2+mrn6TAwmJjymNuYbM4BZyZyKZcSWx06yXFkHWW/YTuz97PE4+7iiuJfx9vFF8a8S7BP2JYwnOiYWJ05wXDmlnHdJXkkVSdPJAclVycKU8JSGVHxqTOp5rhQ3mdu1RnlN9poBnj4vnzecZpt2IG2S78M/mQ6lr0xvzZBGzE2vQEfwk2Ak0yGzLPNzVljWmWzJbG52b45ezo6cl7keub+sRa9lre1cp7pu87qR9c7rj22ANsRu6NyovjFv49gmz03Vm4mbkzf/tsV4S9GWj1vDt7blKeVtyhv9yfOnunyxfH7+0Da7bRXb0ds52/t2mO04tONHAbvgZqFxYXHht52snTd/Nvm55Gfhrvhdfbstdx/Zg93D3TO413FvdZFkUW7R6D6/fc376fsL9n88sPrAjWLz4oqDxIOCg8MlviWthzQO7Tn0rTSx9H6ZS1lDuWL5jvLpw+zDd444HamvUKoorPh6lHP0wTHPY82VWpXFx7HHM4+/OBF2ovsX619qTiqcLDz5vYpbNVwdVN1VY1VTU6tYu7sOrhPUjZ+KPtV/2u10a71B/bEGWkNhI2gUNL7+NebXwSafps4z1mfqz2qeLT9HPVfQDDXnNE+2JLYMt0a2Dpz3Pt/ZZtd27oLhhaqLqhfLLslc2n2ZeDnvsvBK7pWpdl77REdCx2jn6s7HVyOu3uta3tV3zedaz3WP61e7nbuv9Nj3XLxhe+P8TeubLbcsbzX3WvSe+83it3N9ln3Nt61ut/bb9LcNLBu4fMfxTsddt7vX7zHu3brvf39gMHTwwVD00PAD9oNXD1MevnuU+Wjm8aYnmCcFTyWeFj9TfFb5u+7vDcOWw5dG3EZ6nwc/fzzKGn3zR/of38byXlBeFL9UeVnzyvTVxXGP8f7XK16PveG9mZnI/5vk38rf6rw9+6fTn72TEZNj7/jvhO93fpD/UPXR/GPnVODUs0+pn2amCz7Lf67+Yv2l+2v415czWd9w30q+635v++Hz44kwVSjkMfnMOSuAQhocHw/A+yoAKJEAUPsR/yA274nnApr38XME/hPP++a5sASgHulmrZBrOwCNSNNyQrSRftYShTgB2MxM1P4R6fFmpvNaZMRZYj4LhR+UAMAhfuY7XyicOSwUfj+BFPsQgPa0eS8+G1jkD6Ue1zS6T3ygzWgT+Jf4O0btBkBVU4mLAAAB1WlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyI+CiAgICAgICAgIDx0aWZmOkNvbXByZXNzaW9uPjE8L3RpZmY6Q29tcHJlc3Npb24+CiAgICAgICAgIDx0aWZmOk9yaWVudGF0aW9uPjE8L3RpZmY6T3JpZW50YXRpb24+CiAgICAgICAgIDx0aWZmOlBob3RvbWV0cmljSW50ZXJwcmV0YXRpb24+MjwvdGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KAtiABQAAO3BJREFUeAHtfQmcXUWZb93e0kuW7iSQEBLosAYQCFtocICOOqLyU4KOT984jonPeU9nVBpmngOIksjuqHQct9E3Ehxn0fccgqKyDJAQCE3Ymn0JkA4JCQlJujtJL+n1ff+699w+9/apc+qcU3WWe6vy69x7z6nlq6/q++r7vvrqqwwzqaQw0HJVWzOrYs3o1Ngoa2QZtjhUB8dZZ0Ul6+F1jLCujlvau0LVZwonCgOZREFjgPHEwDnXtC2urGCNY2OsdXyczRhnWQLPZFirZ2GFGajtdaiOJlAntd1bUcHWjY6xniduau9U2IypSjMGDAPQjOCg1bfednPz4N5dzSD0McZOp4FqprrCreZBgfFfrpMYU1cFY8+CMTAjOfjHYEQlDAOICNFezbRc24YVffHoOLuIBgWE3uxVJmXvu4gpdFZm2HqSGDo7bmhflzL4SxJcwwBiGlaI8oT8ZWMg+IjF95i6PKlZqBEVxBCIMaw1qsMk9ETywDCASNDMWOvKtsa+IbaMmruIFOdlbJwMdCZNYCBDhsZxtpYerG+oYWvXrWzvmXhpvunCgGEAujBL9cIiP17JlpFx7NJyXeWDohfSARk778qMsrVm5yEoFr3LGQbgjSNfOWwr/eVUcLGvwiazCAOd9GK1kQxE6An+3DCA4LgrKLnk6rZlYxn2OULosoIX5odSDMBeUDHO7th0c/tapRWXaWWGAYQYeIj4oxl2Oatgy41OHwKRQYrCZjDG1lSOs9VGRQiCwGwZwwAC4O68669eNjwwYFb7ALjTUcRIBcGxahiAD9zR1t1yyn4d/TXTn0nJw0AXgbSKthTXJA+0ZEJkGIDHuMCo1z/E2sYh6putOw9sJeQ1qQcZUg3qa1i72U50HxPDAAT4MYQvQEyaHhtG4DlahgEUocgQfhFCSuGnYQTCUTQMwIaaJde0rTSivg0hpfY1xwg23dS+stS6FrQ/hgEQ5oxxL+j0SW25LoLcGAsJCWXNAHACb2SUXWfcdFNLyKEAh7txVSVbVc4nE8uSAeTcdW+j2bM81AwyhUsFA2vIzfiKctwxKDsGwMX9DLvNbOmVCu0q6gfZB2hOXFFuPgRlwwDgtjuSYbeXk7hfUVnJqmprCyikZurUgt/FP4YOHix4NDI4SLEFRwuelfIPrhaMsxXl4l5cFgygVK37IO7KmhpWXVfHGMXeqq6nT0rVDQ1aaHS4r4/XO9w/QBFHxxi5Q7PRoSEGJlFSqYx2C0qaAZTKqm+t5Fi9K+uyRF+8ssdNgGACYAajA4MMUkQpSA7lIA2ULAM49+ttbRRM87o06vpY1WtoFcdKXkWretKIXZbZgAmMkLQAyWGI/sAgUpdIGqDgpqsev7G9PXWwSwBccgwAFv6DQ6Trp+xc/pTp0znBT5kxnYv1EmOXuixgAId693OGcGj//lTBjxOHU2vYilLbKSgpBoB9fYqqe2caVn2I9SD6GiJ4fJZjAhMYIoaAz1QYGkkaoKjGl5WS30DJMABu6Mse1U0sLRmiFw9NmpgBEc2qUnEnTj0DSIPIjxUef7Uzm8QUYN7kMTC4r5tLBUlWE0pFJUg1A0BsfZo1t9MfPhOVYMira2piU5oaS1an141wbjPo7mED3d1JNSB2Eg5WpPlOg9QyAAThHK8g4k9YkA5Y7OtnzzarvWLuAKmgf8+e5PkcwGdgjK1Ia5DSVDIAvsU3Tu68CUrYtmuYO0ebE06CuhorKNhS7HtnF99WjBWQosbphqMr0rhVmDoGQGI/RP7lRfiP7SfE/Po5hxsxP+IRgHrQv2s3Vw8ibtqtuTWkDqxwy5C0d6lhAEkz9mHFn7ZgviH8mGc0GMGBbdsTIxGkzTiYCgaQO777EM212I19RtSPmeIFzSdMNeik48VL0+A0lHgGkBTih1V/6rwjytZpR0B3iXuMrcODO3YmYdcgFUwg0Qwgt813J82y5rhmGpx3YNWHnm9SejAA+wB2DWL2MEz8NmFiGQAn/gx7KM5tPjjvYNXH6m9S+jAA+wCkgVgdimibkObw0qT6CiSSAcRN/Fj1YeBLm49+hkI8khGKp+KBtZ7zl9YPZKLvFCSloBwdg81Gi7Tn4wUnB5G0ZxnP15LLnJAPMAAYCmOTBhLMBIrnSexDFjfxQ9xvIHE/Q0wgmQlkniPYHKHy3xYlxgx0npnYYeMwxQvgOEU16supBbGgKKFMIFEMIE7iT+Kqz1f0HN0UEFYsMzhkozTT7Iwq25/omUKs0kACmUBiGEDO2v8MTbPmkFPNd3GI+tNJ5I9z1c+L70QTnDiipw3feFNRIN9Xi0FEoEZAGthPKkFMtoFE7Q4kggHEudU39YgjWP1hs1XM5QB1EPpB6NYolAnRCxFlxwP/rhch/e/uYQd37hSCo/FFYpiAhXKNfXWvOi7ih2V/RvPRkYbbymrvWXxwQ5s7asxbwgAkBCvpMDIibFlv19Y4/AYSwQQo3Fm8CeG7CIJIPfwg8s88/rhIiB9Eb/0DpkH4hvjl55wdXxYe8akq4fQm5kIMOz6Lc3NfVVcC1ROrqZuMfojd9+lAkAcsNHXOHDZt/pEsQ2G09SZM0uxUNQSvFtNZqSCLXxU1Yy7UNjby0ULw0qgS9WDRkRe0NO/Y0HFXVG0WtxMbA8CRXloMryoGSNdvbuU/8khWp1Hft1YortTrVV91oSl99RI3sP6FBb56agOrrK7hQUvHo+Paixdc2NL79oaOjrDwBykPNhp54sE8MhS8M6IE4m889hhtIj8mIJ8vwKYh/IhGtaiZHO75rkLIQYBdoOeNNyN1HMqMs8viCCoSOQOIeq8fOh6MfardeS2it4xU0S0YRRPf/CzAgH08wjADuBHDOAhmEEmKyUdAtyJcgDtY/OlBZGG8QPxNtPJrIf5cz0D4hvgLhjnWH/bxgDAGRh0kYc5g7mAORZKyoe1uz9FIJE2ikUgZQJQWf5zbxwCqcu7hE2mcJhP9cYI3on5kkzRwQxZzzo2bX2aAuYM5hLkUUYp8ZyAyIyDi9hMSvxgFIhGmC2K/Ckt/nvCjANy0EQEGaEQtPUGiNb5DQOHcx4aGI1EHaIlZNP+ClgwZBddJgBc6SzD5yGez/MaeMTraG0EC8eMkn5qUXTOMiK8Gm4mphTMAiHD+xDicKESI8ihSZQVbGsUNRNoZAPf0G2ZbCNfQ/7UmdcQPUV8rqKbypGCAU4D8YEfGBMgo2FDNFuoOK6bdBsD1/pQQf17cl58PSZnGBo6gGMBYk40gK+t5VwLpEguN9kQ0E4WnoFYbAJx9CFH405pgpIHOHyphEmiXh0JBaAprxEBWK6AJIDEHcIPz8EG67nx4WCNEHJRFup2EJLobrI8tV7U1j1ayZ3SL/tZWX1BrPzi/0fGDjXGplgIz8Dp4hCPF3eQspN1PgFSBylF2Rsct7V068K1NBRjJ6N/vN8SvY0qYOrEgeKkE1hahdj8BUgU4LWkaFi0qAN/yy7DlmmDm1VruvRXV1b6b4YOLvWGTDAZcMQC1UDxPsEVYM20aO0QXmOo8O0AgNOvaGhT3zhUx4pdRiP4W8QfjviB/iP3G0iceRfPGwgAYQFYdEM+XSM4OaFIFlKsAUYj+iOITiPi5Rxjpd4b4rfltPj0wwOcKaN9FYsRcxJzUmjSpAkpVADros5wYplarP87z+z3SmxVzlAs7WsfbVJ5ADNDkFs2iqro6LlnqjCdAzTdT/ICtFD+gUxV2RP3xXX8UDj+I2uJ3u88ifrPo+x5SU8AJA5wJjAn9xHCCUGuwUcUOQspUgL4hdpvOLT+czkLkXn8J5G+2+fzhzOR2xQBfSbLzyikf5qjq06cF7ZAqwGmt4GHwH0oYAHz9CYTlwcHwLskP9/i5rCOns5mV3xu3Joc/DOTnlINdANuDfqVUf63z3MtzNBegaGERJQxgZJRdV1it2l/+jX45zUZsuFULoKmt/DBgzS0HJhCFUVAVzYU2Auo2/EHvn3bkPPkJRgNCappJBgORYIDPNTCBojlX3VDPRgYG2eihQ1rgoHaVGARVSADaVn/s9/vR+y0bbV5E04J6U6nBwAQGrLlmzb2JN4zPXcxhjSk07YWCLhfkY5muDk4/+iiG7RW5RCzYEsvkCphcBgNqMVAkesJTsLJ2CjvU06u2nYnaGsN6CBYJLhM1e33Tve2HW3qnzpN0rnDQw7zgN+8NBrRhgEL82tPBHTtZ/5499kfqvofcFgysAvQPkcMPbUmo68lETRCbcEW3VDI6vxSaTKZoMJC3Cdiaw1zWpgoQDXJatLXn52sgCUD36o9tFJmrmqB3cV5byHD99N/kNRhQjwGiKhCW/UgxnIPgJKQlhZACAkkAOld/EL4M8XMHH2DTEL+WOWUqDYEBmpPZaTmxvsrP6wDthpACJiCUbFfn6g8xqYkuavT2pCKwDeFLjpjJFisGOIVlJysuG+ne/LqeG4cCSgG+JQCdqz8Mf17ED7HfN9eKdQaYxssZA5ir1ozF3MYc15ICSgG+GQDZ3C7X0QGOHAnDH/Zdrb1XHXCYOg0GVGKgeL7W0xz3WuSCth+ENn0xAHj9keitxfIvt+Vn1v6gk8OUixkDtq1qubkeAF6iTU6jPor6YgBU73U+6pbOiqi+3oY/Q/zSCDUZk4cBPn2zcxhzXeN1Y75oVJoB4EpvwmqzDsw2zJ3jWm1W7yfkZW0prnnNS4OBRGKA5m5+HhOAXnM+RB+az7v+amnvXGkGMJZhnwsBlLAoOGG1xOWLJoyXEIXmRUowYJ/DmPO6pIDhgQFpWs3KJB4I5IE+K+h6Lw1p1qITPYwiZuXXgHZTZZwY4FQ3zrAtuPeVV7VAUjnGFsrcJSAlAYxqsvzjiiUvi6gUh9KCQlOpwYAeDFhzGnNf1zVjsjQrxQBYhZ5oP9gScU1kOTVbfq4YMi9TiAE+p3O7Ap40ELR/kjTryQC48U/D1h/0H7fVnxtMLFYZFAmmnMFAQjGAQ0OY46ABLbYAotmc4d4VA54MQJfxz8sKWuxA4doL89JgIGUYsM9vL1oI2jUZ2nVlAPD7J0YlvaUgCyhiprlb/s3SL4tLky/tGMhwWgh00Y1H10G7oGG3bK4MgMIPKyd+AKPNH9qtp+adwUCCMaCLJrxo2JUBEL6U+/1D56md2SQciqyzhPC1eWEwUGIYyM540ISbTSxEp11pWMgAsPdPjS4O0bBjUZltD2P5d0SdeViKGLBNdhnaCICCxTladiwqZADjlXrE/ylNYpUEvNAkg4FyxADmvhtthMGJGy0LGcDoGLs0TKNOZXEIwk3MsVtGncqbZwYDpYgBa96DNrwPxfnHgBstOzIAbv3PsFb/TbmX0NE59xbNW4OBdGFAB42Qz0GraDfAkQF4WQ6DoBThvryMf0HqNWUMBkoFA1ADQCM6IgiLaNqRARBCL1KNVC/OZrOFqG7a1GcwkAoMWDTgRSsBO+NI084MIKPeAFgzY7oQbmP8E6LGvCgjDFjuwW60EhgdApqexAAopNBi1WG/INK4cbXxwL0yBQ0GSgcD3BhI3QGtKFcDsuHCJm3rT2IAOlx/3YifD5/hAKUzi01PwmEgRwueNBOgFSfansQAxsbV6/+eIo3Z/g8wnKZISWIgRwueNBOg8060PYkBYMsgQN2uRdy4Ge+vkQBc8WdelhEGiBZAE240ExQbTrRdwABarm1rDVq5qJxXR8Zt4ZJFdZjnBgPlhAGLJrxoJwhOimm8gAGQEWKSkSBII/Yy7sd+KacR/+3oMt8NBvI04Uk7AXBVTOMFDGBUg/4/xWX7j8NvxP8Aw2iKlDQGLEOgF+0EQEIxjVfZ66DFWKkEAN9mN99/vv9PjVoOEHZYZL6fdtRCdtGRRzlm/dWrL7Dd+/Y6vlPxMEMK1Z+ddhabW9/gWt3bBw+wO194OnAfXSuP4OUHFr2HndQ0S3lLozTo+4cOsZ5Dg/R3iHUPDrDNe3axocFB5W2lrUL4A0AMsOgH0YNVpWIazzMAHvqbsWZVDaEemVhnQYkf9R83o4ldfPQx+Dop3f/WFrab6WEAIP6vnHcR+yAxILf01oH97J+ffTK1xI++nXnYHHahgMm69T3IOyx8L+/bwzbufJs9tPUN1ru/N0g1qS8DmsgygSwNDShkAIScZpwLWLeyvQeImlABqtQSPyr30mHCED/qjyPJEv+W/T3sygf/wPr7++IAM5Vt0urETp45m33hlNPZLz58GfuLM89l1SRFlmOyaMOLhoLgZnBkQtLPM4CxMdYapDK3MlX1dW6v88YO90zJeStL/Jt7utnfPvhHNjgwkBzgUwZJJS2Bnz7hZHb7JZ9gx8ydlzLoFYALbkjJk4ay2Xz9b6f1CQbA2Om+avHIDFdGHYEOPZrV9lqW+F8mu8PXHvqj0WUVjUTjlFr27Qvez+aTKlKOCTSk2i14zEbreQZADKdZJYKliD8lOwCyxP/83nfZ1Q/dw4bJqGWSOgzUVlax71z4ATa7aaa6SpNek402pGjJR3/stJ43AlJ5pTsANVOneoIEQ4el63hmjimDLPE/vXsXW/XwfWx0ZCQmSKNv9qpHH2K7Du730XCG1VVXs4bqGv43Y8oUdsqsw9jZh89lM2vd1cWpVOarZ7Swb5JdpRwSp41cR0FLQ31KbUl5WucMgJ8AVIzVyrpazxpLhfg3vbOD3fjIA2VF/Bjct3t7WDcZO8OkB3KF55Dx78tnLGFnuIj6Zx4+hx1FzOKt3e+EaTIVZTlt0AKJJENL2Zzy/4Pmn7ipvZOrAJUVrFG+qFxOt/1/uRrizSW78j+6Yzu7YcN/lR3xqx6dXbT9940H/sB+8vwzrlUvPyW/eLnmK6WXOmjJonnOAOxWQVWIU623qIJLph5Z4l+3/S12C638Y6OjMtWaPBIYuPvFTvbbNzcLcy6ZcwSr93C+EhZO6QsdtGTRPGcAJG7MUIkbWYAtZweVbYetS5b47yNHo+9ufJCNEyZNUouBXzz/FIOnoCjNmaZ0uoqaifV5MW3I0pQs0BbNZxmAYgOgDpFFtmNh8skS/x+63mD/+Nh6In7xJA0DR7mXhf8EVCtRmj9NHF5OVCbtz1XTFM1crktxBqAaOdV17hZdqz0XJm9liexTlvjXvvEa+/HjD9PuhSF+nYPzRm+3sPoFZcAAiqeXLE0JkSZ4wXcBSNxoFbwP9rhCC18JBotEKVni//Xml9kvnu6gvUuJSk2WUBjY7eJCPYX8AlQmjP8C2n04ng49YTsSf7PIAWeY1Lt9dEhpHx1Q2tF3gD33zttsoL9fZdPydSmmKYvm1WIy151qLxdg+W5rzylL/L985QX2H51PKIPn1AXNbCn9FacRmnQ/6ni4+LHv34uPOoZdOH/ySckdfQfZ/6MDSklPbjx26wE1h4ROOGI++wAdJnvvvPlsRs0UKZRAMnn47W3sd6+9GKm3py6aqsqdApTqvOpMxIVidQSSJf6fv/gs+8/nn1ba/WPoJKPTacIh2lFQwQCObXSu/9XuvcQAlHZFS2WH1dUL630rpO/BTML9V+ig0Tm0o+A3HUtl8YdzCv/x2kvst688z4bVntbLggTi0CxqgvarGE4BKjZky55gKtZz/A5GmPyyxP8TIvy7iQGYFC0GFs4Qu6bsDHpMmGjqk6edzT5LMQ4qOIEF71NdVRVbcfJp7KMLj2PXkEfkjj27g1fmVBLEAR6QS7I0ZeWX+iTaj1dZt3VQCmBFmWSJfzWJ/Ib4FSHdRzU4AnwBieVOqZeCiPT1H3R65foMY/6lJRewz510amjitzc0mySVH77vw+z8Y0+0Pw7/PSLaqBgbVesFqPrkUnhMFtYgQ/zQP7/z9OPsftL7TYoeA5eedBqrrqh0bPhnL3T6lowzZEC78r1L2SW0WnulMVp59wz0s1e795Hh7yAZAr2dvKqp/mvOOZ8hepLOpJq2QPtVJGYo9a305bDgZunRgEkZ4scEuOXJx9jGN17VAIGp0gsDp8w/mq/STvneoLMH62knxm/6bxS6bSnVK0oITXbv1jfZfeTf8Q65JBds8dJKXEu7AkvmN7OPHXM8W+QSHu2rp5/NtlAsiDdotyB0Am0USQGgLaWHgoj2tewCyHY+SiOgLPFfv+kR9sSW12W7YPIpwkAFOad/8tSz2GdOPKV43udbaKct2ALizL8Rf0EcAdQpSht3bmffeXyD2KJPhAjHpIeJ8eDvJAqPdl3LBQynE4sT7Ao3nN/K/ud9v2UHKBZkmMRpI0wFkmVjZQCAMQomIEP8gGU3iX5Pb+vCV5M0YwBi+RQ6DnzCbIo5SKsztuKmORCVBcZPX3iGbdm10/op9VlJhrpvnHuBUOf/1Wsvs395xp9fx8tvv0UE/jt264V/ypwckqaR/eLzJHGs3rhOCkanTKCJqFKsDGAc2xyaLwaRJX4gHBF+v3DWeeyfaEUwyRsDN/zJ+9iQz7MQ9USUiAPgtII6tQgd/B+eejyQSvb+405iR06d5lQtQ/yGX3Y+7tuegMr2U7DXb2x8iP3sTz9KtorJdvT3k3/H7eStiHxBEt8dy0AH0J+UMwCZQCAF3QK309RXP8RvwYRtncdILHyODvuY5I6Bo6frPZTTQbr095/qCERIWEU/efwixw7AznPr4+HOcuwhIyGkkr+h1b44QRX4NBkyf0rqZKAkkAA0BAaxRQUOBKmCQpqIH0ElZUJ3O/XgarLoltuRUyc8xPlscHSEdVCglT5Sy4KkRfOOYkc0OEeleoQOGvUpiLBzD+0SHRx2jtn/IfIwhF0jUNJEE06wBITQqapgzzKaRJ2vLj7H0dNOBkroolefd1E+NrtMGZNHLQYQB7CNxvCXH/sU+wStstWkNvhJy1z25de+/oqfqoR5cRQcbsFOqYaC4s6fdbjTK89numjCqeEKYjZHO72I6hlndgKRJwwMIt0PdWLb5/rHH3Hd40VoqktOPj0MCKasAgzAqAaPu599aBlrlnTdhfi/WBBabCft7W9+Z7sCyLJVPLhti7Cu0w+bK3wnfEGwc5oQZlD7oqq6gU499cV8HVOEPe6mk11t6+5le3v2sR/SqoJVRpT+6j1nsKdIDN1J0X5NmowBEJNfIyBqmUKGs+mE+/qq6smVCp7A4+77rReT3t1J3pnkDOSS4OvfQMFHnRKOc3Mjm9PLAM9epVuM+oaHHds7g+IX/u5Fn5WCFjQsiCIoqkb6BreKXkb2HB2OgAm8S/pkG4Xttq6c+i+6P/CCeQvYWTRQTgl2hJWkCvz1vXexURpkkwox8LX194cKCpqpyNDNP1PYEdMb2Zlz5rEldAHIKTNnC7ftYFz74qln8HsEH3Vx1FrkInpvIgOvygQ1YDvt+Z/oELL86CBxCyIkfuAhdhsAgECfIbbpTFitvkpBJy3i520R07mVjt6KDDnIA1Xif9HWoEnqMYCISrgMdCtF+b2TDl1dff/dbPk9d5Hxz92T7n+f1cIOnym+sPRkCjUuSr0BzhGI6rKedx9yvgFquuQRY6se0IBmMrCayn8mggHAH0ClWJbvXe4LLum8nIjfyTsLd/f9A201uaWPNB/LziCrrkn6MbCPztvfsO4+7o4taq2KVIgvnHqm6DVrotuEnNIhOmo9pOHSFgQMcUo4MQiHJ9kEGuC+MbIFFOSTh06ysaGD/k9qoWpdEgACOFxBxO92SedT5AOO24Td0t+ffR5rEGwruZUz74Jh4BGy1P/AJXDJ+RTMQyQFTBdcKLqLmL2OtJeiBokSvB1lkxcNBKUtt/aVMwC3xtzegfN5IcCtvNO7Vyj4xd/RJZ2HXAbIKvejJzcy2AhECZ5r17RgazBqIU0EkfvztMDp1ot7Xn6e4ZZlUTqP9vqd0jSB6K2LARxwCQhSI2noxLSKevUH7hLDAACMSjXgOQrQcBURv+w9fYjqcgNtDbrZIk8/7HD2Ubq6Og2pThg3Lx0MzMLxv7/6kvV10ucSMhw6paglALg2i9IhgaNQcX6Vc7+4brffiWIA4IKqFthf0Oox4tNyj2Ocv6YDIm7pC3QzzbzZwRw83OpV/Q7756WQOl2s9ic4WN7d+gznHB3JLXzZkIt0YMGict5bdcp+VtCS576pKltTLt+IwCAiU00cIlAxXP/67BOsyyXkFLaisDVYJdhnLq4vrt+lwgAGBvoYAqU6JRjZnIJkwNHLKeGwl450uCB+IYyOshfHyMz9MLTl2G+i/QoKvCJWshxLuT9Uck2WKjHAHVTHt4D/Wx3rhZMOheaRMfCLZ5/vWD4pD6dXi8XSpMAoAwdEY5GVHeWdjGz7Bavu3HrnswEycLjlEUkAIkZUUJePua6EtmyNg/YTpQIAtiwndNPEbT3Q9HX3vr3c48ytehz2OIu2B5Oa3PTSpMLsBBfoo6nWmZlhljiJ2AcEEsBsurDGz7acEzxOz0QM4E2KYOSdMOPjm+8VbIR1eQPpL8ewgpNW/lpUn/uPLz/HYEh0S18jB6GpgvPmbuV0v4N64hSsQne7OurHqUxRfEBY9UdHRiY1u1Mw/6C+NQjE9UmVSD5ooujFItvCiwpdyLXQFNF+Rcct7V2SfY0wW3wc0eokQk/dTF6C/SNiF2D4m3+d7AFwaVWVKuE4ErK648i1GW7MpZDmTmsUdmOLYIV9Ya+YcS9onCmsL8iLpXQBiyi9RPEFvVN8cx20r0UFGO4XO0Z4IySbA/M37jkMz8HvPb3JFeRTye300lPOcM3j9LJPwFhAuNUuobGc6ip+dtbhzttjxfnS8PvjgqAegF3kI/Dqu7uEXfv4cc5BQoQFPF5cLPAQBVlvcZEA/M5vFTTl1BXOAGixW+f0MvAzgdXWT31cM4qPOeZB7XjzNbb+7bfyv52+fJ6OqyL4pJ/U7eKc1EBRaAMnYpznC2LqB64zpoKzaZvvQgrCKUqvUVQepwSvz92CO/xajjiSNdHhIxVp7qzZwpBjT1H8QjcHNBg3fen+CmjK3meL5rVIAMpcFilYSNxSAJC2+olH+SWRdgTav/OtwZYLaWtQfu/djQGc4HKazd6u0/dzm49ngU6hOVUW4zNs711x5nlCbegdOtz19LY3hRDeLzinD8Xo4yeeLCzn58X7jxIbgX/75mZhVXxO+wyEo4ymiqDiDICQ0ln0PNRPldsVCRAC+Im1Gzc96oqTubQ1+NcUSkw29VCoaVF6HwWVDJIQBfdLp4kPyQSpM44yIP5rL/gAg+elKP2YTg+OjTr7B6DM7+jePsT+c0ofpt2bGhfvPacyxc9qaUfhkoXODACuwZ3bu4qL5H87Q5V/7fhFJU2hAYvmswwgw9Rct5oDXa3DQjKkgFd3bGN3upxBR9dx2eeShcfnsOD+0UvXTeMiUKd0HompEC/9JMSfu4LOKiBwRpoTbk3+8Ycv47EBRP3ABadPbX1D9Jo/P0gSwgYaM6eEcGN/e+6Foc51XH7Oe5nouO//pfsDRASblWj9swC1NMXta5zmOQMgw/M6J0SFeaYeYAhv8aY1z2xi2zxCPf/dWeeyaRJbg9i+ekTg5gpD4DeJmOvq5YiZx7+/8IOs1eE68HgxJm4dhIA4fzOmzWC4pvszdFvvDz/ycXYzXeHlFs4NNf4A17RL0NAdFDloVCAF4B6Cz5Mzl98dHByy+tTic3ggGafeIe7E2pecL5MNekBLNS0BbovmeVhwkqRkPBac+it8NkpikK9rwoQ12Ywl4AESA+9SVahXINpv0dbgP33gEmHUGoS5uva8VnbVA3eTG6g7sPfSMeT3Ca6sOoqiyfzs4ktZOzEdXFbitKJMIWPhh48/if3ZcSeyxqIz8JiIoqi4oZBgK3z7hy61/fL3FXYT/PlN/0hHhGUvCIFDF+4SRBQhp3TZsSewkymwyC0UvvtdgUHRXg6M/Upa+d2uFf8u3V7kNFaQubNGP/c5YW/P+g5aUp0smucM4Imb2jvPuaZNaRujAxQkYfp0hXUCcf4njEIAeFWID/jPdF34X71nsbDqU0h8//h7zmS/ee4pYR68eHnnNgpvNUiebs4BLEDUK8m42EcOR9hT3kM7B8OkNsDecAQ5yIDAERyjOOFyy+seW89+SoxKZ3JqW2d77c88wRDGzU/6Pa3GWO2xXeuUTqS7/m4nRgv8PkiMdjMxgp7BftZPgUPq6FhxE6lU86ZNI/XuGB5o1G0G/oEY+iuCKMHZtv0TP8pxWnICPsQz0DyKcwaQqwcPxLM6l0n2A1bL+jliI45sPYX5ksEEfvtSJ99qQ/w6UcI11E/S6UKEuxIlGLFWkkTR3vpBV9YGhyO3VcdeP+Lp//2GB1jfIecoNfa8afmOPq0m4t8QIJw3HLq+9ciD7NtLL2YLXbb/TqaxxF/Q9ND2rezHmza4FA9G/KhQww4AJ37UnV8+CLwuPFCVdOgtgA1SYwDJUVW3eD0Q7W967GGGiSlKEG+/Sas37rp3SziCfDtJFCoSptiqjg1sF61mQfVNFXCoqgNW/LvpotbP3v2bQMRvwTFAEtGVFBXqRSnPPKuU/CcuGvnexnWOKp+K+aqaluy0nmcA9EXNLMzhDXqQasBRNXcQyrUR50fvgV62GsYolzSHxPQvk87olf7zhafZ90jXF+0KeJXHe0Q/+vKD97DnSYxFqnBQDfiLFPyHyEzYR/8fdAnnT+ieRhBw2ITAMNdQgBgwlOBrcSEUcBPHHLj1kQeEx37Rli+Hn8ImOA052hSK8vn5aaf1vAoAqyBJpNf5qcgr7wi5BKsyBBa2RWhVwVoLK/X9a8PmV8gafBQ7n7btRAkXRT52zAkMHoXCRN158NUX2TMkDXzlzBZ2NvnyyxrI3iGvtx+TrYFvi2G25ZJseSt/1J+49LOHiLKbVBV89tDny7RCP0Vx9hEYVEeCERcM5U7apvvqGee6+hm4tQ/J5J6tb7KfU+wIXB3umPLGAtugOGZ0fwgaUp2sHQDUmwez9babm/ve3bVFZWN1TU1s2oL5KqssqCtD4HP0hsNxQZ1J+DGFjILnzG9mLXOPZIfRVuAsMgbOJIv/yPgY27p/P9tK0geClmyleHkv0l636hUiCTiIAoY5pPO3zFtA7sYL2AlkDMwTg0PjByi012PEnB4lfD9H27euoeagplIdYVZ+C4QD27azgW61DLGhhjWtW9negzYK+kw7AWAAzXihIlWS/jtr0YkqqhLXQdeLQxgQbPeKy5k3BgM2DFSSoXVqbT2bTsy3MXcWo4d2XQ6QZAKD6jBiDEgsNPm56NPV1wZKwde9r7zKFG8DdtEOwEKrkbwKgAfUv06ipWbrZdhPAI4/MAJtiRA9TkzAJIOBMBjAzU+9w70Mth1n/0G52vlCpIj4LfqRa1kuF2jcnjNvBMTDygxbb3+p4vuh3v0qqnGvAwg3PMAdR+atfgxgDioifgCrg3aKabyAAZD4UsAdVGBMSyQTR8CScWbAETTzsOQxANFfSkfwgQkdtFNM4wUMoOOG9nU+4JPKeoiMVlEly+iSHYyoWjXtlDMGrLlmzT2VuNBBO8U0XsAAADzpMOtUdgJ16eiIEEZuExC+NS8MBpRiQKXObwdMB8040fYkBkDh7ZTbAYaisAPYsWfpYcYuYMeK+a4SA9bcsuaayrqpLh0040TbkxgAmdPWKu5LtBKABXxuYCwRzXpsPg0GwmIgP6c0ET/g0yIBOND2JAbATwll1B4PhqOKjg55DySxM2xu5kfMu4TJYTDgigE+l3LzyjVj8JegFeXOXUTT1glAO2STGAB/Oa5eCtAh0tg7IvqOocowCh1liWyijOa5wYAXBmgOYS5hTulMWmhFQNPODICptwPEIwFkhyk7YLqHTeeUMHUnAwM6bP2Te6aJVtZPbsl2HNj+knyFldsBINIM7lPr02yHWeo7dDZwcaMSSKHLZMrNFUiPGvV9O55BI8rFf2pARNOOEgAOCjhtGdgBDfJdE2fzCQokgSwj8FnQZC83DIDwucAfnfSog0ZAy9bhn+IhdGQAyERBZu8qzhz2Nzqn+GBDIJCygpzxHAyEvDIplBUSMVOiI37Qhg4G4EbLQgaQGVWvBmDuHOruScwUwuAabSAxw5EYQDAnoiR8q+O6aMONloUMIHdpqPKzAf17ZC5MtFCi/5MPtPEZ0I/ohLeQXwhoLsRB/ECPJtrodLsAWMgAcuO1WvW4JcIY6NSpiIw8Tk2bZwnBQIxzQJfxjzDrSsOuDEBkOQw7XJo4XViwspwfk4Abf0JXZypIAwZyFv64Vn0LRbpowouGXRkA3w1wcB+0gA76iWChOo46BoVncjkwgSwjyIuGkzOZJynFAB/THOFnrfzxdgS0oCeALlsrsv5bPXZlAMhUMc7usDKr/Ox7R3yHu8p2wtWVswAbiSAcGpNUOj+WubFNAGy6aEGGdj0ZwKab29eSSKzcdD+kieupHs+saGh2C1TjNY76stJcfEY+pz5j5QctKE9Es5x2PSr2ZAC8/Bhb41FPoNcD7yZrR8CtE5wRcLUgxwzyK4lbKfMuVgzQGGXF/axKF7ee74QLbTQgSbNSDKBy3N2S6NQxmWcId6xD95FpO0weTCSL/jHBsitLmBpNWVUYsI8HxiiJRG/1FXNfdchvq25ZmpViANhHJB6q/HwAgD349g4L5lR9WhJBwQSzuEKqelIiwNpw7zg2CeymrrkPWnXb+7ejQooBoEB1Xd0d9oKqvkP/SfaOgHdPrQmHOZgVOb3LmByKMEBI56s+qiMVrYAhK2pCRzWY81p0fwK2qkJeYpdmAI9942ZIAF06kKHLCqoDVrc6s9MP1mXonPSRm5xuZcw7/xjIM1ngmJN8egjf6q3GOd9VHPjTatPpU5oB5Aqvcqok7DNwQh2HIMLCFa58jhFQJZZeyiduuErLtvRkHE7gN21IwVzXtfoTLnzRKOehfhB4ztfbuonpNvopI5M3kmvEZADRmAd3GSLxSLIa2ym1qi3GmRbx3gv/Gq77yjaJsF83tjd5tW9/71cCgJrl6ltsr9zPdxyF7N+120+R1OXNqwi57cSJrQTqim9WnLruewMMHFh44N+hSllYw4qf/oQ5rutIfBDa9M0A6mtYOw2ScscgDC38oXUhJ5lTJzvBocfyeZ+b/NaKl0yY1UKV7yv1Pdv9CZyobSn+2vgip+s0LNEkp02f3az0mZ91resYnH9BSx2Va/Vb1iv/OMnGY0PDrLZRuYbh1XRy3ueYAKjBHrqME4r1LjnQykPC+zORPduf3Kqe5n5NdMnzG6761uX3Qii89dHr2+/xBKIog28JAOV1SgEwkJSeQbAI65I/uc6bUxey+i9WRzAGqsAiGutTss5Islkw0ecErFlRPnvIKvc9EmCS0YjWeR1w9QdmAjEAnDAKom/IDgU45TgFETXJCQMW8dhEZRoMEJpFbNZ3/puqsH5zpmERJ3+e/WGVyzMX2zsOAbLl6rfqsn/a33EC59ugWN1tsPKKyvM/zGXMaV0JtOh16k/UdiAGgMp0SgEIGrJfI8JEyEjz8yypTRBd/jfNDusfCJL/0bOscQ33JWTfWu/wy/4uT9D5WqzaJj4tQudl04xETbBjLuuI9MvBDbH6o3xgBqBbCtAqMmkaaFOtwUAxBnTP4zCrP2C1CYTFoMv9Pueati2Us1kut79cFZWVbNaiE1mGPk0yGEgbBiD6Y89f2+pPnrl03dfCMHgJLAHYGl1l+670q1EFlKLTVBYxBrSK/tm+hKa90Evrjg0dnfP+pKWVjELNOvA7eugQy1RUsuqGeh3VmzoNBrRgoJ9iXQzs3aulblSKyz6evLn9irANqJAAWFWlP/9jv0Af3LlT2/6pX1hMfoMBLwxgrx9zVmdSRXNKGEDu9NEanR3u7dpqtgZ1ItjUrQQD0PsxVzWnNX5O/LnBooQBoAEKP3wFmRS1uAijfrhRRoBYNGWSwUBgDGCOanVnJxrjtBYYwsKCoW0AVnVwET7yghaE+l1mPVP9OTo8TFLAGKuZNk111aY+g4HQGDi4Yycb7O0NXY9HBV/aeH17h0ce6dcZ6ZySGc++uu0hMgi2SmYPlG36/PmsdqavU4+B2jGFDAZkMYCbffZv1+ftBzhyhr+lsjDJ5KuSyeQnT9U4WzFawZ4hxzFtJ3qA6Kr6OlZVW+sHtLLJaw+xhq3U0YHBQH2vqK5mlVNq8mWrGxry382XCQzA6Keb+KFeV42xFROtqvmmXAIAWEuuaVtJDqXXqQHRuRY4CTUee0zZMQHol2OkCg33DzA2NsaGBwa4cRTqkVbd0zYMCN5SScwBDloUK5L8SStYNTFkzjDoXTklEH/PG2/qdPbh6CRCXbXppvaVqnGrhQEAyChUAUgATcQEStVTECs5CH2UJhmIW2MYKaXzqoYkBc4kaHzAGEpVcoDFv5uIX9cRX2tQdIj+Vt3KVYB8xRGoAkA8BqAUmAD6MkLEzomeVnXdk8oaJx2fnFER87InMGtIC2AGpaC+RUX8ukR/a2y0SQBo4Nyvt7WNjbPbrMZ0faZREsCKfqh3f5bgiVg0+ovrQnuoeqHCgRngb8qM6VxiCFVhhIUjI37qU0WGXfH4je3turqnlQEA6LOvabuTGtG2NWghpq6piU1bMN/6mchPnAwbIqLHChmVvp5IRDgABZUBqkMNMYMp06c75EjOI5zt13Wjj72XZEdb++RN7ZfZn6n+rk0FsACdWsNW9A3TtqDGXQG0ZQ1IkpgAVgqs8rqPhFq4TvMnGOIA/ui6OCQwAf5HDCFJNp6oiB+i/9Rq9Vb/4jmiXQJAgy3XtrWS/85DxY3r+B23JGCIXv2oJoUZREb8hMLKCrZUlbuv24go8wR0a2T7wx1dFEgUzKbVLZ+KdzCeIbBozdQGOkWozNPZEzQY7xDy+QDddQhvMJxiNEkNBoBLSFGD+/ZxvFbQSRioDFElMHXc42dJJ7rbJUJZRXr/Gt3toP5IJACrI1HZA9BeFIZBTIwB8gDDsU+j01ujHM0nGEDdrFmsjjxCdaoIGOMotvosrEWh91tt4VO7DcDeGLcHDLFmerbY/lzHd51bhFjt4foZ1YqgAz9prxMMF0du8Qe1D67hqv0NoiZ+GpNO0EiUYxOpBICOUQixxSR3PKTbKGghEZLA9KMWKPEYBNEPkpEqLQ45Fg7K5RO7CLU5ZhC2z1hA9r+1LTp/DJykHWdLKcRXZ1jY/ZSPnAEAuCVXty0bz7A7/QAaJm9Yt2EQft9ufVc6hembKTsZA1APGg4/PPCBMRB/FO69dsgpuOdlm25uX2t/FsX3SIyAxR15+5GOVxZc2NJL+s6Hit/p+I0bhwb27iP/9RpWBd91icT1ewrrhEMeWPXx26R0YABjxY2G3T0USGKMxrxW2iAMZt+zpYtO3tHsjCjB2Yf8/NdE1FxBM7FIABYEpA7cTt+XW7+j+KyfPZtNnXeEa1OI59ZPK37avPOmHnEkO5r+mptmTupf58632V76G+zeN+ldqT+ABFhPEkH9YbNdu4rz/LifMuK0hsT+SPV+e/9iZQAAJMqdAavj0BVnNB89yXqcRlEfRP+X51/Ilp18Kmus9ZZuuogBrNvyOvu3p5+gle51CyX5z8aFx7H7vvA3+d9hviz5euiYlWGan1RWpBpAYkAkn6htO1Fb/CchhB5EugvgBECUOwNW+xjofZtf50wARkJY9RHCOU1bebW0yl/5kWWc8K1+yXxCOljetIQtP3MJZwTf/M2/l41UgPGFSgd7znRyG8euAfR97WG8nAcmcou/ExixMwDcMNS6sm1p3xD3FNS+PWghAZNh32ub+e4AJkGaElbpX//F56VWfLd+LSbpoRxVAow99vbB/GMa+06K67c06H1+bmPq9110rnIukAERQAhliXQLBCDFNAFcsOH+6hRauSGiy4j77jXRSZOXnvfKUtLvYxr7xBA/Bjd2CcCaYWACZBRcEaWPgNV2Wj6h7992yTIhuD2DA5yon925g71IBj8rvfeY49jpc+exVvq0M45fbHzYymI+o8BAdq9/RRJWfqu7iWEAAAhOEMQElhomYA1P4ef3PvHfCwjY/rZ943r26wfuJYmGQoUVpd+Tse/39OzbZCQ8kYyF173vYp7joI1JFBWZ9POD/+eHjkbDSRnNA2cMxOTo4wzMxNNEMQCAZZjAxODYv0H0h87ulFaQIe/Fpzc5vSp4BuaAfH9Ooj+kCZMiwkBCiR+9T4QNoHgYuDskuUWSJECeHCYBA9aqXYyNK36/Vor47eXACJy2AO15zHdFGEgw8aOHiWQAACzPBGIwDKL9JCWs1k7OPdjPf5REf5MSi4HOOPz7/WAjsQwAnQATiGt3wA8Sdee9iPR2p3QHOfOYlFgM8LnLF7LEgphgCcDCWZxbhBYMcX9etPBYRxBk9H7HguahbgwkaqvPrbOJMwI6AQsmAGehg0PsdvJdFu+DORUugWfNjZN9+yH+R5lOITXkRYkGB3v2laVzkYUauPfCuzVJW30WbE6fqWAAADyH0MviOEDkhLgonznp/z10d0CUyc3/wA7HqgfvZb9/4B77o3L6voai+K5IU4cTbQNwQiTpVCtwfNLpXTk9e/adHeXU3cT3FXMSczPxgBYBmDoGAPhxUQICKJTzNuEM8mM3KQEYoG0+zEWdl3fo7GUqGQAQwqOnwFegTLcJRU5BOieLqXsSBvg2XxyRfCZBEvBBamwATv3DFks5GAcRzKOY4J0Mg044UvUMMOCsgVdCvIFySGkz9onGJNUMAJ2yjINRXEkuQqLu5yCqSQyAzvXDQciPP38YOL9GHofGezCLQQossqpj5bdXhsFnUsqmVgUoRiDuTsdtKqVoF1i/5Y3i7vLfHzvzHMfn5qEmDJC+z2/sKRHiB5ZKhgGgM7hKqaGaLYR4ht+lkp562fncPqL6IDKQSfoxgDmFuRXFdV36ezPRQkkxAHQLKgFuVOVbhSVymAhRe5wcf3C2//uf+TxFtvGOBTgx5OabLwzQHMJcwpxKi3OPn/6VHAOwOo9tmcpRdgZFd15nPUvz50101t8pwTbwI4oQ5IcJ8KhC37jJVxmntkv9GeYO5lBat/hkxieWewFkAFORZ/sjHT07Hum4g19MmuHXkaV28xwutocRsS86bM4k1MydNp194tzz2b7qaraN8jkFBYGqcC6pDN/91F/ygKC1VdXsV5tfYajXnpDvs5SvOP0LHTwqzlucp2R+Y2+fsVufuLl9BeZQyfTLoSPUz/JILVe1NY9k6CxBhrWmtcdY5bHaF+8IFPcHuwZdNsLGlqGTO7GT264oLHi5RATCql81zlZ03NLeVYzXUvyd+m1A2UHJDehSOkuwnHYKbovqbkJZ+GTyYWW/kqL//Jz0fieCturAO7f3Vj7ECUSoMJMIA7AXjZOuf3P7mnLCR8naAESDSM5Da2DNpfepHGjs+//5D77jaBQU9Vn03EuSEJUrwed8TmBulGDfXLtU0jYAUc+71nUM7tjQcddRF7WsHxtjzaQWNIvyJvH52MgIu5908o093VwdaJS879DeF+wqwLmnWK8vJxsAF/cr2YpNN7avxpyw46dcvpeNDcBtQLlaQGH3KE+zW76kvoPe/pGT38OZQSt9d0qwC8CdF05F8CsQXQgCBvB+ByPgAxRMVFTGqb2EP+si+FaV44pfPC6GAdgwwt2JM+zyNNoHbN3gX+EmDKOhcd+1YQbW/XG2Gl6jtqdl/dUwgKLhp8NFjf1DrG28RBhBUffK82eO8OtrWHspOvOEGVTDAATYM4xAgJg0PTaE7zlahgF4oMgwAg8EJfG1IXzpUTEMQBpVjKXdWOijq2nN2kWAG+Oej9EzDMAHsqysLde2tY6MscsJeWUXodjCQZI+cVKvqoKtLrWTelHg2DCAEFiGe/EojIUVbHkp7ByEQEX0RUnMZ2NsTSVZ9cvFbVcHkg0DUITVJVe3LRvLsM8ZqUARQgXVYLWvGGd3pDkOn6BrsTw2DEAx2mE07BviqsHlVPVixdWXa3Wd1PHVdE3cWrONp3YKGAagFp8FtUFFGK9ky0bH2KVpPoVY0KmIfvCz+BXsrswoW2tEfH1INwxAH24LarZJBhfRybNlxmZQgB7rNN5aerrerPRFuNH40zAAjch1q5q2FBcT8peNjbOLylU6wCpP4bbWQ69HiHc3fJl3ejBgGIAevPquFVuLRBCLR8EQsraDZt+VJLtAFxF6ZyURPDG8TrNll4zBMgwgGeMwCQqoDIMjbDEdV24dY+x0GqhmypQWo2InEXsXBZt4tqKCrautYp3GeDdpiBPxwDCARAyDPBBQHSg2fSMYA0kMM4jQOFOIWo2A+A6oaQJ1Utu9IHQydvYYUV5+LJOQ0zCAJIyCQhiw88CquLTAxkZZI1FoOKlhnHVWVJLTDdII6zIWeYWDlYCq/j+A3luv/BDsyQAAAABJRU5ErkJggg== provider_display_name: Pivotal support_url: TBA plan_updatable: true plans:
- description: 'Example: This plan will configure a lightweight kubernetes cluster.
Not recommended for production workloads.'
instance_groups:
- azs:
- az-1
- az-2
- az-3 instances: 1 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
- azs:
- az-1
- az-2
- az-3 instances: 3 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
- azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
disabled: true name: wavefront-proxy-errand
-
name: vrops-errand
-
name: vrops-registration
-
name: telemetry-agent pre_delete:
-
name: drain-cluster metadata: allow-privileged-containers: true master_instances: 1 max_worker_instances: 50 worker_instances: 3 name: Plan 1 plan_id: 8A0E21A8-8072-4D80-B365-D1F502085560 properties: addons-spec: |+
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Sinks [{"addr":"example.com:12345"}]
parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log
apiVersion: v1 kind: Service metadata: name: fluent-bit namespace: pks-system spec: selector: k8s-app: logging-agent ports: - protocol: TCP port: 24224 targetPort: forward-plugin type: ClusterIP
apiVersion: batch/v1 kind: Job metadata: labels: job: cert-generator name: cert-generator-v0.11 namespace: pks-system spec: backoffLimit: 0 template: metadata: labels: job: cert-generator spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/cert-generator:v0.12 imagePullPolicy: IfNotPresent name: cert-generator restartPolicy: Never serviceAccountName: cert-generator
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: fluent-bit resources: limits: memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true - command: - ghostunnel - server - --listen - :24224 - --target - localhost:24225 - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --allow-dns-san - event-controller - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel ports: - containerPort: 24224 name: forward-plugin volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /fluent-bit-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /fluent-bit-certs name: fluent-bit-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config - emptyDir: {} name: keystore - name: fluent-bit-certs secret: secretName: fluent-bit - name: pks-ca secret: secretName: pks-ca updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: event-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: event-controller spec: containers: - env: - name: FORWARDER_HOST value: localhost image: oratos/event-controller:v0.12 imagePullPolicy: IfNotPresent name: event-controller - command: - ghostunnel - client - --listen - localhost:24224 - --target - fluent-bit.pks-system.svc.cluster.local:24224 - --override-server-name - fluent-bit - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /event-controller-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /event-controller-certs name: event-controller-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: event-controller volumes: - emptyDir: {} name: keystore - name: event-controller-certs secret: secretName: event-controller - name: pks-ca secret: secretName: pks-ca
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.12 imagePullPolicy: IfNotPresent name: sink-controller serviceAccountName: sink-controller
-
- azs:
- description: 'Example: This plan will configure a medium sized kubernetes
cluster, suitable for more pods.'
instance_groups:
-
azs:
- az-1
- az-2
- az-3 instances: 3 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
-
azs:
- az-1
- az-2
- az-3 instances: 5 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
-
azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
disabled: true name: wavefront-proxy-errand
-
name: vrops-errand
-
name: vrops-registration
-
name: telemetry-agent pre_delete:
-
name: drain-cluster metadata: allow-privileged-containers: true master_instances: 3 max_worker_instances: 50 worker_instances: 5 name: multi-master plan_id: 58375a45-17f7-4291-acf1-455bfdc8e371 properties: addons-spec: |
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Sinks [{"addr":"example.com:12345"}]
parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log
apiVersion: v1 kind: Service metadata: name: fluent-bit namespace: pks-system spec: selector: k8s-app: logging-agent ports: - protocol: TCP port: 24224 targetPort: forward-plugin type: ClusterIP
apiVersion: batch/v1 kind: Job metadata: labels: job: cert-generator name: cert-generator-v0.11 namespace: pks-system spec: backoffLimit: 0 template: metadata: labels: job: cert-generator spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/cert-generator:v0.12 imagePullPolicy: IfNotPresent name: cert-generator restartPolicy: Never serviceAccountName: cert-generator
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: fluent-bit resources: limits: memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true - command: - ghostunnel - server - --listen - :24224 - --target - localhost:24225 - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --allow-dns-san - event-controller - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel ports: - containerPort: 24224 name: forward-plugin volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /fluent-bit-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /fluent-bit-certs name: fluent-bit-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config - emptyDir: {} name: keystore - name: fluent-bit-certs secret: secretName: fluent-bit - name: pks-ca secret: secretName: pks-ca updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: event-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: event-controller spec: containers: - env: - name: FORWARDER_HOST value: localhost image: oratos/event-controller:v0.12 imagePullPolicy: IfNotPresent name: event-controller - command: - ghostunnel - client - --listen - localhost:24224 - --target - fluent-bit.pks-system.svc.cluster.local:24224 - --override-server-name - fluent-bit - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /event-controller-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /event-controller-certs name: event-controller-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: event-controller volumes: - emptyDir: {} name: keystore - name: event-controller-certs secret: secretName: event-controller - name: pks-ca secret: secretName: pks-ca
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.12 imagePullPolicy: IfNotPresent name: sink-controller serviceAccountName: sink-controller
allow-privileged-containers: true disable_deny_escalating_exec: true max_worker_instances: 50
-
-
- null service_description: Default on-demand Kubernetes service. service_name: p.pks tags:
- pivotal
- kubernetes
- k8s service_deployment: releases:
- jobs:
- kube-apiserver
- kube-controller-manager
- kube-scheduler
- kubelet
- kube-proxy
- kubernetes-roles
- flanneld
- nginx
- kubernetes-api-route-registrar
- apply-specs
- secure-var-vcap name: kubo version: 0.25.8
- jobs:
- etcd name: cfcr-etcd version: 1.8.0
- jobs:
- docker name: docker version: 33.0.2
- jobs:
- pks-nsx-t-prepare-master-vm
- pks-nsx-t-ncp name: pks-nsx-t version: 1.19.0
- jobs:
- ncp
- nsx-node-agent
- openvswitch
- nsx-cni
- nsx-kube-proxy name: nsx-cf-cni version: 2.3.1.10693410
- jobs:
- fluentd name: pks-vrli version: 0.7.0
- jobs:
- syslog_forwarder name: syslog version: 11.4.0
- jobs:
- bpm name: bpm version: 0.13.0
- jobs:
- wavefront-proxy name: wavefront-proxy version: 0.10.0-dev.141
- jobs:
- pks-vrops name: pks-vrops version: 0.10.0+dev.5
- jobs:
- drain-cluster name: pks-helpers version: 50.0.0
- jobs:
- telemetry-dns-alias name: pks-telemetry version: 2.0.0-build.113
- jobs:
- sink-resources-images name: sink-resources-release version: 0.1.15 stemcell: os: ubuntu-xenial version: "170.21" service_instances_api: authentication: basic: password: ((pks_api_basicauth.password)) username: ((pks_api_basicauth.username)) root_ca_cert: ((pks_api_internal.ca)) url: https://localhost:9021/service_instances startup_banner: true username: ((odb_broker_basicauth.username)) stemcell: bosh-vsphere-esxi-ubuntu-xenial-go_agent update: max_in_flight: 1 vm_type: large name: pivotal-container-service-dda71dbb88455ace2ade releases:
- name: cf-mysql version: 36.14.0
- name: docker version: 33.0.2
- name: kubo version: 0.25.8
- name: cfcr-etcd version: 1.8.0
- name: kubo-service-adapter version: 1.3.0-build.126
- name: on-demand-service-broker version: 0.24.0
- name: pks-api version: 1.3.0-build.126
- name: pks-helpers version: 50.0.0
- name: pks-nsx-t version: 1.19.0
- name: nsx-cf-cni version: 2.3.1.10693410
- name: pks-vrli version: 0.7.0
- name: syslog version: 11.4.0
- name: pks-vrops version: 0.10.0+dev.5
- name: sink-resources-release version: 0.1.15
- name: pks-telemetry version: 2.0.0-build.113
- name: uaa version: "64.0"
- name: bpm version: 0.13.0
- name: backup-and-restore-sdk version: 1.8.0
- name: wavefront-proxy version: 0.10.0-dev.141 stemcells:
- alias: bosh-vsphere-esxi-ubuntu-xenial-go_agent os: ubuntu-xenial version: "170.21" update: canaries: 1 canary_watch_time: 30000-300000 max_errors: 2 max_in_flight: 1 serial: false update_watch_time: 30000-300000 variables:
- name: kubo_odb_ca options: common_name: ca is_ca: true type: certificate
- name: kubo_odb_ca_2018 options: common_name: ca duration: 1461 is_ca: true type: certificate
- name: pks_api_internal
options:
alternative_names:
- localhost
- 127.0.0.1 ca: kubo_odb_ca common_name: localhost type: certificate
- name: uaa_jwt_signing_key_1 type: rsa
- name: uaa_active_pks_saml_key_2018 options: common_name: ca duration: 1461 is_ca: true type: certificate
- name: pks_api_basicauth type: user
- name: odb_broker_basicauth type: user
- name: telemetry_ca_2018 options: common_name: ca duration: 1461 is_ca: true type: certificate
- name: telemetry_server_tls_2018 options: ca: telemetry_ca_2018 common_name: telemetry.pks.internal duration: 1461 type: certificate #+END_EXAMPLE ** pks tile: with vrli enabled :noexport: #+BEGIN_EXAMPLE ubuntu@opsman:~$ bosh -d pivotal-container-service-8f7873397b6d8f2b58f6 manifest Using environment '30.0.0.11' as client 'ops_manager'
Using deployment 'pivotal-container-service-8f7873397b6d8f2b58f6'
instance_groups:
- azs:
- az-1 env: bosh: password: $6$b7ce3e45788446e7$xSFtkreCw31eikORMpXZEIMQJraCEEns2odZRIw3CFT9gt0BjLzsAv1FJjCGM9v5qVSMt8L5oMUYf94lK9PE/. instances: 1 jobs:
- consumes: {} name: service-adapter properties: deployment: broker_deployment_name: pivotal-container-service-8f7873397b6d8f2b58f6 director_url: https://30.0.0.11:25555 kubo_odb_ca: ((kubo_odb_ca.certificate)) syslog: migration: disabled: true provides: {} release: kubo-service-adapter
- consumes: broker: from: proxy-broker name: pks-api properties: pks: db_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_pks_db_password.value)) fqdn: api.pks.local internal_tls: certificate: ((pks_api_internal.certificate)) private_key: ((pks_api_internal.private_key)) password: ((pks_api_basicauth.password)) pks_client_secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_api_uaa_client.value)) telemetry: authenticationMode: service_account enabled: true eventEmitterBaseUrl: http://localhost:8012 tls: certificate: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.cert_pem)) private_key: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.private_key_pem)) uaa_service_admin_client_id: service_admin_client uaa_service_admin_client_secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_services_admin_uaa_client.value)) username: ((pks_api_basicauth.username)) provides: pks_api: as: pks_api_http pks_api_shared: as: pks_api_shared_http shared: true pks_uaa_service_admin_client: as: pks_uaa_service_admin_client shared: true release: pks-api
- consumes: {} name: bosh-update-config properties: bosh: authentication: uaa: client_id: pivotal-container-service-8f7873397b6d8f2b58f6 client_secret: ((/opsmgr/director/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_client_secret.value)) url: https://30.0.0.11:8443 root_ca_cert: | -----BEGIN CERTIFICATE----- MIIDUDCCAjigAwIBAgIUQw3pICIuRKxgSrFmy8WrU0cMIgkwDQYJKoZIhvcNAQEL BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTgxMDE0MjMy MjU1WhcNMjIxMDE1MjMyMjU1WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2 b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMzUbJTQ70Kq38zJ PKdMPbfkdV7KPc3L2+MyolQwFNWZQAIJ+LPKld/8zRFZfosqSzphHy9DKXY441r1 VvhhZdmwPcAHWyOq6we2pt60KjLvrA52oGcNnXImviOnEYM7mWYYoOA3DhBaUaB/ 0FjXYevKqtBDLZPirkGlZiMDfuqjHpzT5fmPavNoq3XyU32BflHQdEFPiavUzo7H QGTVMaUebQwRT7Z8HLlSUT621ARWDwVfbj4GYYKyXNdniqCELsTBemPMQdBKVzUe VpTWfNGJH9uR467H5SPyp+DtihFbpoZIyw3OhsYLRYEwxsOz4576SIPzJ1pubueP VhwzkpcCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUJ2J9H598MNr2iKzqaL6N4pWhBCIw HwYDVR0jBBgwFoAUJ2J9H598MNr2iKzqaL6N4pWhBCIwHQYDVR0lBBYwFAYIKwYB BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG MA0GCSqGSIb3DQEBCwUAA4IBAQAx5QdKSJ9N/7T7g80cdpmJsBNjJmpCERTpp5XA h7TNI75+kjTj2rgoXpGUpQ8yPO/g4xZS0eNh1ZmQXksVoGOWEQUTp4Cey8QraOab d28kaWrY8UHQj6DQIvHRuzLGWf/pMsOcH1KU2GK9+YKUHRtpvVI9gUGkHamFdXS3 YsdabMzHQgpvVWl4tnXb+fmTfygMHyS5bVxcYgZ4+5MaCxg7QbnHMNfNKGA/MMHr 3dlzV08GOyX4cpbJGlv/AtZcHLbWNu2X+idjHP1wlHcF5g7v9SHleeijXXCaHtb4 3OFNePJv/g7KmMllI9xHzQSbAp0dlq+xjHzqBOGmajTDWPWB -----END CERTIFICATE----- url: https://30.0.0.11:25555 cloud_config: vm_extensions: - cloud_properties: vmx_options: disk.enableUUID: "1" name: disk_enable_uuid cloud_config_name: pivotal-container-service-8f7873397b6d8f2b58f6 provides: {} release: pks-api
- consumes: {} name: broker provides: broker: as: odb-broker release: on-demand-service-broker
- consumes: broker: from: odb-broker name: pks-nsx-t-osb-proxy properties: bosh: authentication: uaa: client_id: pivotal-container-service-8f7873397b6d8f2b58f6 client_secret: ((/opsmgr/director/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_client_secret.value)) cloud_config_dns: 192.168.115.1 cloud_config_prefix: pks log_level: INFO root_ca_cert: | -----BEGIN CERTIFICATE----- MIIDUDCCAjigAwIBAgIUQw3pICIuRKxgSrFmy8WrU0cMIgkwDQYJKoZIhvcNAQEL BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTgxMDE0MjMy MjU1WhcNMjIxMDE1MjMyMjU1WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2 b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMzUbJTQ70Kq38zJ PKdMPbfkdV7KPc3L2+MyolQwFNWZQAIJ+LPKld/8zRFZfosqSzphHy9DKXY441r1 VvhhZdmwPcAHWyOq6we2pt60KjLvrA52oGcNnXImviOnEYM7mWYYoOA3DhBaUaB/ 0FjXYevKqtBDLZPirkGlZiMDfuqjHpzT5fmPavNoq3XyU32BflHQdEFPiavUzo7H QGTVMaUebQwRT7Z8HLlSUT621ARWDwVfbj4GYYKyXNdniqCELsTBemPMQdBKVzUe VpTWfNGJH9uR467H5SPyp+DtihFbpoZIyw3OhsYLRYEwxsOz4576SIPzJ1pubueP VhwzkpcCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUJ2J9H598MNr2iKzqaL6N4pWhBCIw HwYDVR0jBBgwFoAUJ2J9H598MNr2iKzqaL6N4pWhBCIwHQYDVR0lBBYwFAYIKwYB BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG MA0GCSqGSIb3DQEBCwUAA4IBAQAx5QdKSJ9N/7T7g80cdpmJsBNjJmpCERTpp5XA h7TNI75+kjTj2rgoXpGUpQ8yPO/g4xZS0eNh1ZmQXksVoGOWEQUTp4Cey8QraOab d28kaWrY8UHQj6DQIvHRuzLGWf/pMsOcH1KU2GK9+YKUHRtpvVI9gUGkHamFdXS3 YsdabMzHQgpvVWl4tnXb+fmTfygMHyS5bVxcYgZ4+5MaCxg7QbnHMNfNKGA/MMHr 3dlzV08GOyX4cpbJGlv/AtZcHLbWNu2X+idjHP1wlHcF5g7v9SHleeijXXCaHtb4 3OFNePJv/g7KmMllI9xHzQSbAp0dlq+xjHzqBOGmajTDWPWB -----END CERTIFICATE----- url: https://30.0.0.11:25555 create_network_with_lb: false enabled: true fip_address_parameter: nsxt_fip_address generate_lb_name: true kubernetes_master_host_parameter: kubernetes_master_host lb_service_id_parameter: nsxt_lb_service_id log_level: INFO network_name_parameter: nsxt_network_name nsxt: ca_cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWZ6AhVAMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE4MTAxNTIzMTQwOFoXDTIzMTAxNDIzMTQwOFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCE NhsvKNYS3EYkYnmHoeJdtPdlW5GwsRIh+eep9xiMX8tcU+n1a8LokthRJe5vxYjA /qTOiKZ1NdDxrwHnLQfmzUQa/xZeH73nHG4X8Rx0/+E0BHkPBnt9Uc/Kj6NUBBxq O3RnnebUA7+SmRr/1iDapgf2PZd/oCdXj9NG3mWSkEwPipalpfaEfGzxB83tLvGp km/PCMdUDx+kVkDH1o4dL8dpYxOpnaz3g+E2i6+ZoLhn1+8AQ2i5WxIXT5DEM3TP /HPrz9Cr43+NQ9AIrIj97lRpoe1nq/gPG/0rAHaAYyAsVgdR2BE3JI4gMCC58fGK 46B4FxFz30wOEOjHcSWRAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAGMrSeaeg/Jw mGao84dR7Uymn+KLW0W3Mptv3J115xzpfym+ZzDqaC/F9aCOM4qJ+RWex67aIM6c JqXEadM2Ul/COjIy5oFwf7925tas9p+3TJ8dCvmBvUdqSe3rb2Y8KDU2PkJssY1t FaKZ3FCB/fpRTAtleA5UJk3uA2u5z3G9ESyojhNF6HRwsawgcFwAH8pQflN//ouO eOwP5b7UiuMEi0Zpw4TwpjzUVzWsEkovr+oSPB2IPinWFIBJQQS2vguknuR/ssil VLnLYIlMlLMB7eNkEDUFL0r+5zRTV46Rz+RU7yGRstk5ckdZ5ieuabA4zYIzdPbs Q4EX1moA/Wc= -----END CERTIFICATE----- floating_ip_pool_ids: - 0cecb28c-79f9-47ea-be3c-ce0bfc176871 host: nsxmanager.pks.vmware.local insecure: true ip_block_id: b557d4d9-1c52-48c6-bad4-7c5fe174ca36 lb_size_large_supported: false lb_size_medium_supported: true log_level: INFO nat_mode: true network_prefix: pks superuser_cert: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) superuser_key: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) t0_router_id: 7c38157e-6107-4e70-94fe-26f2ac256995 password: ((odb_broker_basicauth.password)) port: 3000 username: ((odb_broker_basicauth.username)) provides: {} release: pks-nsx-t
- consumes: {} name: mysql properties: cf_mysql: mysql: admin_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_mysql_admin_password.value)) cluster_health: password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_mysql_cluster_health_password.value)) galera_healthcheck: db_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_mysql_galera_healthcheck_db_password.value)) endpoint_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_mysql_galera_healthcheck_endpoint_password.value)) seeded_databases: - name: pks password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_pks_db_password.value)) username: pks - name: uaa password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_uaa_db_password.value)) username: uaa - name: telemetry password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_telemetry_db_password.value)) username: telemetry provides: {} release: cf-mysql
- consumes: {} name: uaa properties: encryption: active_key_label: key-1 encryption_keys: - label: key-1 passphrase: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_encryption_passphrase.value)) login: saml: activeKeyId: active-pks-saml-key keys: active-pks-saml-key: certificate: ((uaa_active_pks_saml_key.certificate)) key: ((uaa_active_pks_saml_key.private_key)) passphrase: "" signatureAlgorithm: SHA256 release_level_backup: true uaa: clients: admin: authorities: uaa.admin,clients.read,clients.write,clients.secret,scim.read,scim.write,clients.admin,pks.clusters.admin,pks.clusters.manage authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_uaa_management_admin_client.value)) pks_cli: access-token-validity: 7200 authorities: uaa.resource authorized-grant-types: password,refresh_token refresh-token-validity: 21600 scope: pks.clusters.admin,pks.clusters.manage secret: "" pks_client: access-token-validity: 86400 authorities: pks.clusters.admin,pks.clusters.manage,uaa.resource authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_api_uaa_client.value)) pks_cluster_client: authorities: uaa.resource authorized-grant-types: password,client_credentials,user_token,refresh_token scope: openid,roles secret: "" service_admin_client: authorities: clients.admin authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_services_admin_uaa_client.value)) jwt: policy: active_key_id: key-1 keys: key-1: signingKey: ((uaa_jwt_signing_key_1.private_key)) ldap: emailDomain: [] enabled: false externalGroupsWhitelist: - '*' groups: groupSearchFilter: member={0} profile_type: no-groups searchBase: null mailAttributeName: mail referral: follow searchBase: null searchFilter: cn={0} sslCertificate: null sslCertificateAlias: null url: null userDN: null userPassword: null port: 35684 scim: groups: pks.clusters.admin: Allows a user to admin PKS pks.clusters.manage: Allows a user to manage PKS clusters user: override: true users: - groups: - uaa.admin - pks.clusters.admin name: admin password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_admin_password.value)) sslCertificate: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.cert_pem)) sslPrivateKey: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.private_key_pem)) url: https://api.pks.local:8443 uaadb: address: 127.0.0.1 databases: - name: uaa tag: uaa db_scheme: mysql port: 3306 roles: - name: uaa password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_uaa_db_password.value)) tag: admin provides: {} release: uaa
- consumes: {} name: bbr-uaadb provides: {} release: uaa
- consumes: {} name: database-backup-restorer provides: {} release: backup-and-restore-sdk
- consumes: broker: from: proxy-broker name: upgrade-all-service-instances properties: service_instances_api: authentication: basic: password: ((pks_api_basicauth.password)) username: ((pks_api_basicauth.username)) root_ca_cert: ((pks_api_internal.ca)) url: https://localhost:9021/service_instances provides: {} release: on-demand-service-broker
- consumes: {} name: syslog_forwarder properties: syslog: migration: disabled: true provides: {} release: syslog-migration
- consumes: pks_api: from: pks_api_http name: delete-all-clusters provides: {} release: pks-api
- consumes: {}
name: pks-nsx-t-precheck
properties:
floating-ip-pool-ids:
- 0cecb28c-79f9-47ea-be3c-ce0bfc176871 ip-block-id: 3a045baf-2f4c-4dc4-81bf-10f77ebb893a network-automation: true nodes-ip-block-id: b557d4d9-1c52-48c6-bad4-7c5fe174ca36 nsx-t-ca-cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWZ6AhVAMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE4MTAxNTIzMTQwOFoXDTIzMTAxNDIzMTQwOFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCE NhsvKNYS3EYkYnmHoeJdtPdlW5GwsRIh+eep9xiMX8tcU+n1a8LokthRJe5vxYjA /qTOiKZ1NdDxrwHnLQfmzUQa/xZeH73nHG4X8Rx0/+E0BHkPBnt9Uc/Kj6NUBBxq O3RnnebUA7+SmRr/1iDapgf2PZd/oCdXj9NG3mWSkEwPipalpfaEfGzxB83tLvGp km/PCMdUDx+kVkDH1o4dL8dpYxOpnaz3g+E2i6+ZoLhn1+8AQ2i5WxIXT5DEM3TP /HPrz9Cr43+NQ9AIrIj97lRpoe1nq/gPG/0rAHaAYyAsVgdR2BE3JI4gMCC58fGK 46B4FxFz30wOEOjHcSWRAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAGMrSeaeg/Jw mGao84dR7Uymn+KLW0W3Mptv3J115xzpfym+ZzDqaC/F9aCOM4qJ+RWex67aIM6c JqXEadM2Ul/COjIy5oFwf7925tas9p+3TJ8dCvmBvUdqSe3rb2Y8KDU2PkJssY1t FaKZ3FCB/fpRTAtleA5UJk3uA2u5z3G9ESyojhNF6HRwsawgcFwAH8pQflN//ouO eOwP5b7UiuMEi0Zpw4TwpjzUVzWsEkovr+oSPB2IPinWFIBJQQS2vguknuR/ssil VLnLYIlMlLMB7eNkEDUFL0r+5zRTV46Rz+RU7yGRstk5ckdZ5ieuabA4zYIzdPbs Q4EX1moA/Wc= -----END CERTIFICATE----- nsx-t-host: nsxmanager.pks.vmware.local nsx-t-insecure: true nsx-t-superuser-certificate: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) nsx-t-superuser-key: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) t0-router-id: 7c38157e-6107-4e70-94fe-26f2ac256995 vcenter-cluster: kubo-az-1 vcenter-datacenter: kubo-dc vcenter-host: 192.168.111.137 vcenter-insecure: true vcenter-password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cloud_provider/vsphere/vcenter_master_creds.password)) vcenter-user: [email protected] provides: {} release: pks-nsx-t
- consumes: {} name: pks-nsx-t-ops-files properties: fip_address_parameter: nsxt_fip_address lb-created-by-proxy: false nsx-t-ca-cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWZ6AhVAMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE4MTAxNTIzMTQwOFoXDTIzMTAxNDIzMTQwOFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCE NhsvKNYS3EYkYnmHoeJdtPdlW5GwsRIh+eep9xiMX8tcU+n1a8LokthRJe5vxYjA /qTOiKZ1NdDxrwHnLQfmzUQa/xZeH73nHG4X8Rx0/+E0BHkPBnt9Uc/Kj6NUBBxq O3RnnebUA7+SmRr/1iDapgf2PZd/oCdXj9NG3mWSkEwPipalpfaEfGzxB83tLvGp km/PCMdUDx+kVkDH1o4dL8dpYxOpnaz3g+E2i6+ZoLhn1+8AQ2i5WxIXT5DEM3TP /HPrz9Cr43+NQ9AIrIj97lRpoe1nq/gPG/0rAHaAYyAsVgdR2BE3JI4gMCC58fGK 46B4FxFz30wOEOjHcSWRAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAGMrSeaeg/Jw mGao84dR7Uymn+KLW0W3Mptv3J115xzpfym+ZzDqaC/F9aCOM4qJ+RWex67aIM6c JqXEadM2Ul/COjIy5oFwf7925tas9p+3TJ8dCvmBvUdqSe3rb2Y8KDU2PkJssY1t FaKZ3FCB/fpRTAtleA5UJk3uA2u5z3G9ESyojhNF6HRwsawgcFwAH8pQflN//ouO eOwP5b7UiuMEi0Zpw4TwpjzUVzWsEkovr+oSPB2IPinWFIBJQQS2vguknuR/ssil VLnLYIlMlLMB7eNkEDUFL0r+5zRTV46Rz+RU7yGRstk5ckdZ5ieuabA4zYIzdPbs Q4EX1moA/Wc= -----END CERTIFICATE----- nsx-t-host: nsxmanager.pks.vmware.local nsx-t-insecure: true nsx-t-superuser-certificate: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) nsx-t-superuser-key: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) t0-router-id: 7c38157e-6107-4e70-94fe-26f2ac256995 provides: {} release: pks-nsx-t
- consumes: {} name: pks-wavefront-ops-files properties: wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: wavefront-alert-creation properties: wavefront-alert-targets: ignored wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: wavefront-alert-deletion properties: wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: pks-vrli-ops-files properties: fluentd_vrli_ca_cert: null fluentd_vrli_host: 127.0.0.1 fluentd_vrli_rate_limit_msec: 0 fluentd_vrli_skip_cert_verify: false fluentd_vrli_use_ssl: false provides: {} release: pks-vrli
- consumes: pks_api: from: pks_api_http name: telemetry properties: pks_db_host: localhost pks_db_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_telemetry_db_password.value)) pks_instance_id: pivotal-container-service-8f7873397b6d8f2b58f6 telemetry_enabled: true telemetry_url: http://35.188.203.192 tls_ca: certificate: ((telemetry_ca.certificate)) tls_server: certificate: ((telemetry_server_tls.certificate)) private_key: ((telemetry_server_tls.private_key)) vcenter_server: 192.168.111.137 provides: pks_telemetry: as: pks_telemetry shared: true release: pks-telemetry
- consumes: {} name: telemetry-pod-ops-files properties: interval: 600 telemetry_ca: | ((telemetry_ca.certificate)) provides: {} release: pks-telemetry
- consumes: {} name: event-emitter properties: aggregator_ca_cert: ((telemetry_ca.certificate)) aggregator_endpoint: https://telemetry.pks.internal:8011/api/send?objectType=Event&sourceID=PKSServer pks_cloud_provider: vSphere pks_tile_version: 1.2.0-build.47 server_port: 8012 provides: {} release: event-emitter lifecycle: service name: pivotal-container-service networks:
- default:
- dns
- gateway
name: deployment-network
persistent_disk_type: "10240"
properties:
bosh:
authentication:
uaa:
client_id: pivotal-container-service-8f7873397b6d8f2b58f6
client_secret: ((/opsmgr/director/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_client_secret.value))
url: https://30.0.0.11:8443
root_ca_cert: |
-----BEGIN CERTIFICATE-----
MIIDUDCCAjigAwIBAgIUQw3pICIuRKxgSrFmy8WrU0cMIgkwDQYJKoZIhvcNAQEL
BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTgxMDE0MjMy
MjU1WhcNMjIxMDE1MjMyMjU1WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2
b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMzUbJTQ70Kq38zJ
PKdMPbfkdV7KPc3L2+MyolQwFNWZQAIJ+LPKld/8zRFZfosqSzphHy9DKXY441r1
VvhhZdmwPcAHWyOq6we2pt60KjLvrA52oGcNnXImviOnEYM7mWYYoOA3DhBaUaB/
0FjXYevKqtBDLZPirkGlZiMDfuqjHpzT5fmPavNoq3XyU32BflHQdEFPiavUzo7H
QGTVMaUebQwRT7Z8HLlSUT621ARWDwVfbj4GYYKyXNdniqCELsTBemPMQdBKVzUe
VpTWfNGJH9uR467H5SPyp+DtihFbpoZIyw3OhsYLRYEwxsOz4576SIPzJ1pubueP
VhwzkpcCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUJ2J9H598MNr2iKzqaL6N4pWhBCIw
HwYDVR0jBBgwFoAUJ2J9H598MNr2iKzqaL6N4pWhBCIwHQYDVR0lBBYwFAYIKwYB
BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG
MA0GCSqGSIb3DQEBCwUAA4IBAQAx5QdKSJ9N/7T7g80cdpmJsBNjJmpCERTpp5XA
h7TNI75+kjTj2rgoXpGUpQ8yPO/g4xZS0eNh1ZmQXksVoGOWEQUTp4Cey8QraOab
d28kaWrY8UHQj6DQIvHRuzLGWf/pMsOcH1KU2GK9+YKUHRtpvVI9gUGkHamFdXS3
YsdabMzHQgpvVWl4tnXb+fmTfygMHyS5bVxcYgZ4+5MaCxg7QbnHMNfNKGA/MMHr
3dlzV08GOyX4cpbJGlv/AtZcHLbWNu2X+idjHP1wlHcF5g7v9SHleeijXXCaHtb4
3OFNePJv/g7KmMllI9xHzQSbAp0dlq+xjHzqBOGmajTDWPWB
-----END CERTIFICATE-----
url: https://30.0.0.11:25555
disable_cf_startup_checks: true
expose_operational_errors: false
password: ((odb_broker_basicauth.password))
port: 8080
service_adapter:
path: /var/vcap/jobs/service-adapter/bin/service-adapter
service_catalog:
bindable: true
global_properties:
authorization_mode: rbac
deployments_network: service-network
iaas: vsphere
oidc: false
oidc_ca: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.cert_pem))
oidc_client_id: pks_cluster_client
oidc_groups_claim: roles
oidc_groups_prefix: ""
oidc_issuer_url: https://api.pks.local:8443/oauth/token
oidc_username_claim: user_name
oidc_username_prefix: '-'
ops_files_paths:
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_pks_nsx_t.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/prepare_master_vm.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_fip_to_tls_certs.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/remove_flannel.yml
- /var/vcap/jobs/pks-wavefront-ops-files/manifests/add-wavefront-job.yml
- /var/vcap/jobs/pks-vrli-ops-files/manifests/add-fluentd-job.yml
- /var/vcap/jobs/telemetry-pod-ops-files/manifests/add-agent-job.yml proxy: null routing_mode: external vcenter_dc: kubo-dc vcenter_ds: iscsi-ds-0 vcenter_ip: 192.168.111.137 vcenter_vms: pcf_vms worker_max_in_flight: 1 id: DF8EECC4-7225-42D0-8459-4A6C584314CA metadata: display_name: Kubernetes documentation_url: TBA image_url: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAAKp2lDQ1BJQ0MgUHJvZmlsZQAASImVlwdUFEkax6t7ciINDEHCkJPkDJLjkCWDqAwzhCGMw8CQzMqigmtARQQURVeSAosKiAkQBZVFMIB5QRYV5VwMiIo618Ax3N69u3v39atXv/d19b++rq56798AkHuYPF4KLAFAKjeDH+TpQo+IjKLjRgAauQhAAlCYrHSec2CgL0Biof9rfBoE0Gx/12BW69/v/9eQZMelswCAAhGOZaezUhE+i7RGFo+fAQCKjeTVszJ4s7wdYWk+UiDC5bOcMM+Nsxw7z11zY0KCXBF+AACezGTyEwAg/YHk6ZmsBESHjEbYmMvmcBE2R9iBlchE5iEj98DS1NQ1s3wUYZ3Yf9JJ+ItmrEiTyUwQ8fy7zAXejZPOS2Hm/J/L8b8jNUWwMIca0siJfK+g2fmQNatOXuMjYm6sf8ACc9jzNc1yosArdIFZ6a5RC8xmuvkssCA51HmBmfzFZzkZjJAF5q8JEulzU/x9RfpxDBHHpbsHL3A8x4OxwLmJIeELnMkJ81/g9ORgn8UxrqI8XxAkqjme7yF6x9T0xdpYzMW5MhJDvBZriBDVw45zcxfluaGi8bwMF5EmLyVwsf4UT1E+PTNY9GwGssEWOInpHbioEyhaH8ABfoAJWBlx2bP7Criu4eXwOQmJGXRn5JTE0RlcluFSuqmxiRUAs2du/pN+oM2dJYh2czG39QgA9meFQuGFxZxPGwBnCgEgDi3mtNcDINYBQE8ZS8DPnM/NbnWAAUQgDqSBPFAG6kAHGABTYAnsgBNwB94gAISASLAKsEAiSAV8kAXWgc0gHxSCPeAAKAUV4DioBqdBE2gBF0EHuA5ugX5wHzwGw2AMvAGT4BOYgSAIB1EgKiQPqUCakD5kCllDDpA75AsFQZFQDJQAcSEBtA7aChVCRVApdAyqgX6FzkMd0A1oAHoIjUDj0HvoK4yCybA0rARrwUawNewM+8Ah8Eo4AU6Dc+E8eBdcAlfCp+BmuAO+Bd+Hh+E38BQKoEgoGkoVZYCyRrmiAlBRqHgUH7UBVYAqRlWi6lFtqG7UXdQwagL1BY1FU9F0tAHaDu2FDkWz0GnoDeid6FJ0NboZ3YW+ix5BT6J/YCgYRYw+xhbDwERgEjBZmHxMMeYk5hzmGuY+ZgzzCYvF0rDaWCusFzYSm4Rdi92JPYxtwLZjB7Cj2CkcDieP08fZ4wJwTFwGLh93CHcKdwV3BzeG+4wn4VXwpngPfBSei9+CL8bX4i/j7+Bf4mcIEgRNgi0hgMAm5BB2E04Q2gi3CWOEGaIkUZtoTwwhJhE3E0uI9cRrxCfEDyQSSY1kQ1pO4pA2kUpIjaQe0gjpC1mKrEd2JUeTBeRd5CpyO/kh+QOFQtGiOFGiKBmUXZQaylXKM8pnMaqYoRhDjC22UaxMrFnsjthbcYK4priz+CrxXPFi8TPit8UnJAgSWhKuEkyJDRJlEuclhiSmJKmSJpIBkqmSOyVrJW9IvpLCSWlJuUuxpfKkjktdlRqloqjqVFcqi7qVeoJ6jTomjZXWlmZIJ0kXSp+W7pOelJGSMZcJk8mWKZO5JDNMQ9G0aAxaCm03rYk2SPsqqyTrLBsnu0O2XvaO7LTcEjknuTi5ArkGuftyX+Xp8u7yyfJ75VvknyqgFfQUlitkKRxRuKYwsUR6id0S1pKCJU1LHinCinqKQYprFY8r9ipOKSkreSrxlA4pXVWaUKYpOyknKe9Xvqw8rkJVcVDhqOxXuaLymi5Dd6an0EvoXfRJVUVVL1WB6jHVPtUZNW21ULUtag1qT9WJ6tbq8er71TvVJzVUNPw01mnUaTzSJGhaayZqHtTs1pzW0tYK19qm1aL1SltOm6Gdq12n/USHouOok6ZTqXNPF6trrZuse1i3Xw/Ws9BL1CvTu60P61vqc/QP6w8sxSy1WcpdWrl0yIBs4GyQaVBnMGJIM/Q13GLYYvjWSMMoymivUbfRD2ML4xTjE8aPTaRMvE22mLSZvDfVM2WZlpneM6OYeZhtNGs1e2eubx5nfsT8gQXVws9im0WnxXdLK0u+Zb3luJWGVYxVudWQtbR1oPVO6x4bjI2LzUabizZfbC1tM2ybbP+0M7BLtqu1e7VMe1ncshPLRu3V7Jn2x+yHHegOMQ5HHYYdVR2ZjpWOz53UndhOJ51eOus6Jzmfcn7rYuzCdznnMu1q67retd0N5ebpVuDW5y7lHupe6v7MQ80jwaPOY9LTwnOtZ7sXxsvHa6/XEEOJwWLUMCa9rbzXe3f5kH2CfUp9nvvq+fJ92/xgP2+/fX5P/DX9uf4tASCAEbAv4GmgdmBa4IXl2OWBy8uWvwgyCVoX1B1MDV4dXBv8KcQlZHfI41CdUEFoZ5h4WHRYTdh0uFt4UfhwhFHE+ohbkQqRnMjWKFxUWNTJqKkV7isOrBiLtojOjx5cqb0ye+WNVQqrUlZdWi2+mrn6TAwmJjymNuYbM4BZyZyKZcSWx06yXFkHWW/YTuz97PE4+7iiuJfx9vFF8a8S7BP2JYwnOiYWJ05wXDmlnHdJXkkVSdPJAclVycKU8JSGVHxqTOp5rhQ3mdu1RnlN9poBnj4vnzecZpt2IG2S78M/mQ6lr0xvzZBGzE2vQEfwk2Ak0yGzLPNzVljWmWzJbG52b45ezo6cl7keub+sRa9lre1cp7pu87qR9c7rj22ANsRu6NyovjFv49gmz03Vm4mbkzf/tsV4S9GWj1vDt7blKeVtyhv9yfOnunyxfH7+0Da7bRXb0ds52/t2mO04tONHAbvgZqFxYXHht52snTd/Nvm55Gfhrvhdfbstdx/Zg93D3TO413FvdZFkUW7R6D6/fc376fsL9n88sPrAjWLz4oqDxIOCg8MlviWthzQO7Tn0rTSx9H6ZS1lDuWL5jvLpw+zDd444HamvUKoorPh6lHP0wTHPY82VWpXFx7HHM4+/OBF2ovsX619qTiqcLDz5vYpbNVwdVN1VY1VTU6tYu7sOrhPUjZ+KPtV/2u10a71B/bEGWkNhI2gUNL7+NebXwSafps4z1mfqz2qeLT9HPVfQDDXnNE+2JLYMt0a2Dpz3Pt/ZZtd27oLhhaqLqhfLLslc2n2ZeDnvsvBK7pWpdl77REdCx2jn6s7HVyOu3uta3tV3zedaz3WP61e7nbuv9Nj3XLxhe+P8TeubLbcsbzX3WvSe+83it3N9ln3Nt61ut/bb9LcNLBu4fMfxTsddt7vX7zHu3brvf39gMHTwwVD00PAD9oNXD1MevnuU+Wjm8aYnmCcFTyWeFj9TfFb5u+7vDcOWw5dG3EZ6nwc/fzzKGn3zR/of38byXlBeFL9UeVnzyvTVxXGP8f7XK16PveG9mZnI/5vk38rf6rw9+6fTn72TEZNj7/jvhO93fpD/UPXR/GPnVODUs0+pn2amCz7Lf67+Yv2l+2v415czWd9w30q+635v++Hz44kwVSjkMfnMOSuAQhocHw/A+yoAKJEAUPsR/yA274nnApr38XME/hPP++a5sASgHulmrZBrOwCNSNNyQrSRftYShTgB2MxM1P4R6fFmpvNaZMRZYj4LhR+UAMAhfuY7XyicOSwUfj+BFPsQgPa0eS8+G1jkD6Ue1zS6T3ygzWgT+Jf4O0btBkBVU4mLAAAB1WlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyI+CiAgICAgICAgIDx0aWZmOkNvbXByZXNzaW9uPjE8L3RpZmY6Q29tcHJlc3Npb24+CiAgICAgICAgIDx0aWZmOk9yaWVudGF0aW9uPjE8L3RpZmY6T3JpZW50YXRpb24+CiAgICAgICAgIDx0aWZmOlBob3RvbWV0cmljSW50ZXJwcmV0YXRpb24+MjwvdGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KAtiABQAAO3BJREFUeAHtfQmcXUWZb93e0kuW7iSQEBLosAYQCFtocICOOqLyU4KOT984jonPeU9nVBpmngOIksjuqHQct9E3Ehxn0fccgqKyDJAQCE3Ymn0JkA4JCQlJujtJL+n1ff+699w+9/apc+qcU3WWe6vy69x7z6nlq6/q++r7vvrqqwwzqaQw0HJVWzOrYs3o1Ngoa2QZtjhUB8dZZ0Ul6+F1jLCujlvau0LVZwonCgOZREFjgPHEwDnXtC2urGCNY2OsdXyczRhnWQLPZFirZ2GFGajtdaiOJlAntd1bUcHWjY6xniduau9U2IypSjMGDAPQjOCg1bfednPz4N5dzSD0McZOp4FqprrCreZBgfFfrpMYU1cFY8+CMTAjOfjHYEQlDAOICNFezbRc24YVffHoOLuIBgWE3uxVJmXvu4gpdFZm2HqSGDo7bmhflzL4SxJcwwBiGlaI8oT8ZWMg+IjF95i6PKlZqBEVxBCIMaw1qsMk9ETywDCASNDMWOvKtsa+IbaMmruIFOdlbJwMdCZNYCBDhsZxtpYerG+oYWvXrWzvmXhpvunCgGEAujBL9cIiP17JlpFx7NJyXeWDohfSARk778qMsrVm5yEoFr3LGQbgjSNfOWwr/eVUcLGvwiazCAOd9GK1kQxE6An+3DCA4LgrKLnk6rZlYxn2OULosoIX5odSDMBeUDHO7th0c/tapRWXaWWGAYQYeIj4oxl2Oatgy41OHwKRQYrCZjDG1lSOs9VGRQiCwGwZwwAC4O68669eNjwwYFb7ALjTUcRIBcGxahiAD9zR1t1yyn4d/TXTn0nJw0AXgbSKthTXJA+0ZEJkGIDHuMCo1z/E2sYh6putOw9sJeQ1qQcZUg3qa1i72U50HxPDAAT4MYQvQEyaHhtG4DlahgEUocgQfhFCSuGnYQTCUTQMwIaaJde0rTSivg0hpfY1xwg23dS+stS6FrQ/hgEQ5oxxL+j0SW25LoLcGAsJCWXNAHACb2SUXWfcdFNLyKEAh7txVSVbVc4nE8uSAeTcdW+j2bM81AwyhUsFA2vIzfiKctwxKDsGwMX9DLvNbOmVCu0q6gfZB2hOXFFuPgRlwwDgtjuSYbeXk7hfUVnJqmprCyikZurUgt/FP4YOHix4NDI4SLEFRwuelfIPrhaMsxXl4l5cFgygVK37IO7KmhpWXVfHGMXeqq6nT0rVDQ1aaHS4r4/XO9w/QBFHxxi5Q7PRoSEGJlFSqYx2C0qaAZTKqm+t5Fi9K+uyRF+8ssdNgGACYAajA4MMUkQpSA7lIA2ULAM49+ttbRRM87o06vpY1WtoFcdKXkWretKIXZbZgAmMkLQAyWGI/sAgUpdIGqDgpqsev7G9PXWwSwBccgwAFv6DQ6Trp+xc/pTp0znBT5kxnYv1EmOXuixgAId693OGcGj//lTBjxOHU2vYilLbKSgpBoB9fYqqe2caVn2I9SD6GiJ4fJZjAhMYIoaAz1QYGkkaoKjGl5WS30DJMABu6Mse1U0sLRmiFw9NmpgBEc2qUnEnTj0DSIPIjxUef7Uzm8QUYN7kMTC4r5tLBUlWE0pFJUg1A0BsfZo1t9MfPhOVYMira2piU5oaS1an141wbjPo7mED3d1JNSB2Eg5WpPlOg9QyAAThHK8g4k9YkA5Y7OtnzzarvWLuAKmgf8+e5PkcwGdgjK1Ia5DSVDIAvsU3Tu68CUrYtmuYO0ebE06CuhorKNhS7HtnF99WjBWQosbphqMr0rhVmDoGQGI/RP7lRfiP7SfE/Po5hxsxP+IRgHrQv2s3Vw8ibtqtuTWkDqxwy5C0d6lhAEkz9mHFn7ZgviH8mGc0GMGBbdsTIxGkzTiYCgaQO777EM212I19RtSPmeIFzSdMNeik48VL0+A0lHgGkBTih1V/6rwjytZpR0B3iXuMrcODO3YmYdcgFUwg0Qwgt813J82y5rhmGpx3YNWHnm9SejAA+wB2DWL2MEz8NmFiGQAn/gx7KM5tPjjvYNXH6m9S+jAA+wCkgVgdimibkObw0qT6CiSSAcRN/Fj1YeBLm49+hkI8khGKp+KBtZ7zl9YPZKLvFCSloBwdg81Gi7Tn4wUnB5G0ZxnP15LLnJAPMAAYCmOTBhLMBIrnSexDFjfxQ9xvIHE/Q0wgmQlkniPYHKHy3xYlxgx0npnYYeMwxQvgOEU16supBbGgKKFMIFEMIE7iT+Kqz1f0HN0UEFYsMzhkozTT7Iwq25/omUKs0kACmUBiGEDO2v8MTbPmkFPNd3GI+tNJ5I9z1c+L70QTnDiipw3feFNRIN9Xi0FEoEZAGthPKkFMtoFE7Q4kggHEudU39YgjWP1hs1XM5QB1EPpB6NYolAnRCxFlxwP/rhch/e/uYQd37hSCo/FFYpiAhXKNfXWvOi7ih2V/RvPRkYbbymrvWXxwQ5s7asxbwgAkBCvpMDIibFlv19Y4/AYSwQQo3Fm8CeG7CIJIPfwg8s88/rhIiB9Eb/0DpkH4hvjl55wdXxYe8akq4fQm5kIMOz6Lc3NfVVcC1ROrqZuMfojd9+lAkAcsNHXOHDZt/pEsQ2G09SZM0uxUNQSvFtNZqSCLXxU1Yy7UNjby0ULw0qgS9WDRkRe0NO/Y0HFXVG0WtxMbA8CRXloMryoGSNdvbuU/8khWp1Hft1YortTrVV91oSl99RI3sP6FBb56agOrrK7hQUvHo+Paixdc2NL79oaOjrDwBykPNhp54sE8MhS8M6IE4m889hhtIj8mIJ8vwKYh/IhGtaiZHO75rkLIQYBdoOeNNyN1HMqMs8viCCoSOQOIeq8fOh6MfardeS2it4xU0S0YRRPf/CzAgH08wjADuBHDOAhmEEmKyUdAtyJcgDtY/OlBZGG8QPxNtPJrIf5cz0D4hvgLhjnWH/bxgDAGRh0kYc5g7mAORZKyoe1uz9FIJE2ikUgZQJQWf5zbxwCqcu7hE2mcJhP9cYI3on5kkzRwQxZzzo2bX2aAuYM5hLkUUYp8ZyAyIyDi9hMSvxgFIhGmC2K/Ckt/nvCjANy0EQEGaEQtPUGiNb5DQOHcx4aGI1EHaIlZNP+ClgwZBddJgBc6SzD5yGez/MaeMTraG0EC8eMkn5qUXTOMiK8Gm4mphTMAiHD+xDicKESI8ihSZQVbGsUNRNoZAPf0G2ZbCNfQ/7UmdcQPUV8rqKbypGCAU4D8YEfGBMgo2FDNFuoOK6bdBsD1/pQQf17cl58PSZnGBo6gGMBYk40gK+t5VwLpEguN9kQ0E4WnoFYbAJx9CFH405pgpIHOHyphEmiXh0JBaAprxEBWK6AJIDEHcIPz8EG67nx4WCNEHJRFup2EJLobrI8tV7U1j1ayZ3SL/tZWX1BrPzi/0fGDjXGplgIz8Dp4hCPF3eQspN1PgFSBylF2Rsct7V068K1NBRjJ6N/vN8SvY0qYOrEgeKkE1hahdj8BUgU4LWkaFi0qAN/yy7DlmmDm1VruvRXV1b6b4YOLvWGTDAZcMQC1UDxPsEVYM20aO0QXmOo8O0AgNOvaGhT3zhUx4pdRiP4W8QfjviB/iP3G0iceRfPGwgAYQFYdEM+XSM4OaFIFlKsAUYj+iOITiPi5Rxjpd4b4rfltPj0wwOcKaN9FYsRcxJzUmjSpAkpVADros5wYplarP87z+z3SmxVzlAs7WsfbVJ5ADNDkFs2iqro6LlnqjCdAzTdT/ICtFD+gUxV2RP3xXX8UDj+I2uJ3u88ifrPo+x5SU8AJA5wJjAn9xHCCUGuwUcUOQspUgL4hdpvOLT+czkLkXn8J5G+2+fzhzOR2xQBfSbLzyikf5qjq06cF7ZAqwGmt4GHwH0oYAHz9CYTlwcHwLskP9/i5rCOns5mV3xu3Joc/DOTnlINdANuDfqVUf63z3MtzNBegaGERJQxgZJRdV1it2l/+jX45zUZsuFULoKmt/DBgzS0HJhCFUVAVzYU2Auo2/EHvn3bkPPkJRgNCappJBgORYIDPNTCBojlX3VDPRgYG2eihQ1rgoHaVGARVSADaVn/s9/vR+y0bbV5E04J6U6nBwAQGrLlmzb2JN4zPXcxhjSk07YWCLhfkY5muDk4/+iiG7RW5RCzYEsvkCphcBgNqMVAkesJTsLJ2CjvU06u2nYnaGsN6CBYJLhM1e33Tve2HW3qnzpN0rnDQw7zgN+8NBrRhgEL82tPBHTtZ/5499kfqvofcFgysAvQPkcMPbUmo68lETRCbcEW3VDI6vxSaTKZoMJC3Cdiaw1zWpgoQDXJatLXn52sgCUD36o9tFJmrmqB3cV5byHD99N/kNRhQjwGiKhCW/UgxnIPgJKQlhZACAkkAOld/EL4M8XMHH2DTEL+WOWUqDYEBmpPZaTmxvsrP6wDthpACJiCUbFfn6g8xqYkuavT2pCKwDeFLjpjJFisGOIVlJysuG+ne/LqeG4cCSgG+JQCdqz8Mf17ED7HfN9eKdQaYxssZA5ir1ozF3MYc15ICSgG+GQDZ3C7X0QGOHAnDH/Zdrb1XHXCYOg0GVGKgeL7W0xz3WuSCth+ENn0xAHj9keitxfIvt+Vn1v6gk8OUixkDtq1qubkeAF6iTU6jPor6YgBU73U+6pbOiqi+3oY/Q/zSCDUZk4cBPn2zcxhzXeN1Y75oVJoB4EpvwmqzDsw2zJ3jWm1W7yfkZW0prnnNS4OBRGKA5m5+HhOAXnM+RB+az7v+amnvXGkGMJZhnwsBlLAoOGG1xOWLJoyXEIXmRUowYJ/DmPO6pIDhgQFpWs3KJB4I5IE+K+h6Lw1p1qITPYwiZuXXgHZTZZwY4FQ3zrAtuPeVV7VAUjnGFsrcJSAlAYxqsvzjiiUvi6gUh9KCQlOpwYAeDFhzGnNf1zVjsjQrxQBYhZ5oP9gScU1kOTVbfq4YMi9TiAE+p3O7Ap40ELR/kjTryQC48U/D1h/0H7fVnxtMLFYZFAmmnMFAQjGAQ0OY46ABLbYAotmc4d4VA54MQJfxz8sKWuxA4doL89JgIGUYsM9vL1oI2jUZ2nVlAPD7J0YlvaUgCyhiprlb/s3SL4tLky/tGMhwWgh00Y1H10G7oGG3bK4MgMIPKyd+AKPNH9qtp+adwUCCMaCLJrxo2JUBEL6U+/1D56md2SQciqyzhPC1eWEwUGIYyM540ISbTSxEp11pWMgAsPdPjS4O0bBjUZltD2P5d0SdeViKGLBNdhnaCICCxTladiwqZADjlXrE/ylNYpUEvNAkg4FyxADmvhtthMGJGy0LGcDoGLs0TKNOZXEIwk3MsVtGncqbZwYDpYgBa96DNrwPxfnHgBstOzIAbv3PsFb/TbmX0NE59xbNW4OBdGFAB42Qz0GraDfAkQF4WQ6DoBThvryMf0HqNWUMBkoFA1ADQCM6IgiLaNqRARBCL1KNVC/OZrOFqG7a1GcwkAoMWDTgRSsBO+NI084MIKPeAFgzY7oQbmP8E6LGvCgjDFjuwW60EhgdApqexAAopNBi1WG/INK4cbXxwL0yBQ0GSgcD3BhI3QGtKFcDsuHCJm3rT2IAOlx/3YifD5/hAKUzi01PwmEgRwueNBOgFSfansQAxsbV6/+eIo3Z/g8wnKZISWIgRwueNBOg8060PYkBYMsgQN2uRdy4Ge+vkQBc8WdelhEGiBZAE240ExQbTrRdwABarm1rDVq5qJxXR8Zt4ZJFdZjnBgPlhAGLJrxoJwhOimm8gAGQEWKSkSBII/Yy7sd+KacR/+3oMt8NBvI04Uk7AXBVTOMFDGBUg/4/xWX7j8NvxP8Aw2iKlDQGLEOgF+0EQEIxjVfZ66DFWKkEAN9mN99/vv9PjVoOEHZYZL6fdtRCdtGRRzlm/dWrL7Dd+/Y6vlPxMEMK1Z+ddhabW9/gWt3bBw+wO194OnAfXSuP4OUHFr2HndQ0S3lLozTo+4cOsZ5Dg/R3iHUPDrDNe3axocFB5W2lrUL4A0AMsOgH0YNVpWIazzMAHvqbsWZVDaEemVhnQYkf9R83o4ldfPQx+Dop3f/WFrab6WEAIP6vnHcR+yAxILf01oH97J+ffTK1xI++nXnYHHahgMm69T3IOyx8L+/bwzbufJs9tPUN1ru/N0g1qS8DmsgygSwNDShkAIScZpwLWLeyvQeImlABqtQSPyr30mHCED/qjyPJEv+W/T3sygf/wPr7++IAM5Vt0urETp45m33hlNPZLz58GfuLM89l1SRFlmOyaMOLhoLgZnBkQtLPM4CxMdYapDK3MlX1dW6v88YO90zJeStL/Jt7utnfPvhHNjgwkBzgUwZJJS2Bnz7hZHb7JZ9gx8ydlzLoFYALbkjJk4ay2Xz9b6f1CQbA2Om+avHIDFdGHYEOPZrV9lqW+F8mu8PXHvqj0WUVjUTjlFr27Qvez+aTKlKOCTSk2i14zEbreQZADKdZJYKliD8lOwCyxP/83nfZ1Q/dw4bJqGWSOgzUVlax71z4ATa7aaa6SpNek402pGjJR3/stJ43AlJ5pTsANVOneoIEQ4el63hmjimDLPE/vXsXW/XwfWx0ZCQmSKNv9qpHH2K7Du730XCG1VVXs4bqGv43Y8oUdsqsw9jZh89lM2vd1cWpVOarZ7Swb5JdpRwSp41cR0FLQ31KbUl5WucMgJ8AVIzVyrpazxpLhfg3vbOD3fjIA2VF/Bjct3t7WDcZO8OkB3KF55Dx78tnLGFnuIj6Zx4+hx1FzOKt3e+EaTIVZTlt0AKJJENL2Zzy/4Pmn7ipvZOrAJUVrFG+qFxOt/1/uRrizSW78j+6Yzu7YcN/lR3xqx6dXbT9940H/sB+8vwzrlUvPyW/eLnmK6WXOmjJonnOAOxWQVWIU623qIJLph5Z4l+3/S12C638Y6OjMtWaPBIYuPvFTvbbNzcLcy6ZcwSr93C+EhZO6QsdtGTRPGcAJG7MUIkbWYAtZweVbYetS5b47yNHo+9ufJCNEyZNUouBXzz/FIOnoCjNmaZ0uoqaifV5MW3I0pQs0BbNZxmAYgOgDpFFtmNh8skS/x+63mD/+Nh6In7xJA0DR7mXhf8EVCtRmj9NHF5OVCbtz1XTFM1crktxBqAaOdV17hZdqz0XJm9liexTlvjXvvEa+/HjD9PuhSF+nYPzRm+3sPoFZcAAiqeXLE0JkSZ4wXcBSNxoFbwP9rhCC18JBotEKVni//Xml9kvnu6gvUuJSk2WUBjY7eJCPYX8AlQmjP8C2n04ng49YTsSf7PIAWeY1Lt9dEhpHx1Q2tF3gD33zttsoL9fZdPydSmmKYvm1WIy151qLxdg+W5rzylL/L985QX2H51PKIPn1AXNbCn9FacRmnQ/6ni4+LHv34uPOoZdOH/ySckdfQfZ/6MDSklPbjx26wE1h4ROOGI++wAdJnvvvPlsRs0UKZRAMnn47W3sd6+9GKm3py6aqsqdApTqvOpMxIVidQSSJf6fv/gs+8/nn1ba/WPoJKPTacIh2lFQwQCObXSu/9XuvcQAlHZFS2WH1dUL630rpO/BTML9V+ig0Tm0o+A3HUtl8YdzCv/x2kvst688z4bVntbLggTi0CxqgvarGE4BKjZky55gKtZz/A5GmPyyxP8TIvy7iQGYFC0GFs4Qu6bsDHpMmGjqk6edzT5LMQ4qOIEF71NdVRVbcfJp7KMLj2PXkEfkjj27g1fmVBLEAR6QS7I0ZeWX+iTaj1dZt3VQCmBFmWSJfzWJ/Ib4FSHdRzU4AnwBieVOqZeCiPT1H3R65foMY/6lJRewz510amjitzc0mySVH77vw+z8Y0+0Pw7/PSLaqBgbVesFqPrkUnhMFtYgQ/zQP7/z9OPsftL7TYoeA5eedBqrrqh0bPhnL3T6lowzZEC78r1L2SW0WnulMVp59wz0s1e795Hh7yAZAr2dvKqp/mvOOZ8hepLOpJq2QPtVJGYo9a305bDgZunRgEkZ4scEuOXJx9jGN17VAIGp0gsDp8w/mq/STvneoLMH62knxm/6bxS6bSnVK0oITXbv1jfZfeTf8Q65JBds8dJKXEu7AkvmN7OPHXM8W+QSHu2rp5/NtlAsiDdotyB0Am0USQGgLaWHgoj2tewCyHY+SiOgLPFfv+kR9sSW12W7YPIpwkAFOad/8tSz2GdOPKV43udbaKct2ALizL8Rf0EcAdQpSht3bmffeXyD2KJPhAjHpIeJ8eDvJAqPdl3LBQynE4sT7Ao3nN/K/ud9v2UHKBZkmMRpI0wFkmVjZQCAMQomIEP8gGU3iX5Pb+vCV5M0YwBi+RQ6DnzCbIo5SKsztuKmORCVBcZPX3iGbdm10/op9VlJhrpvnHuBUOf/1Wsvs395xp9fx8tvv0UE/jt264V/ypwckqaR/eLzJHGs3rhOCkanTKCJqFKsDGAc2xyaLwaRJX4gHBF+v3DWeeyfaEUwyRsDN/zJ+9iQz7MQ9USUiAPgtII6tQgd/B+eejyQSvb+405iR06d5lQtQ/yGX3Y+7tuegMr2U7DXb2x8iP3sTz9KtorJdvT3k3/H7eStiHxBEt8dy0AH0J+UMwCZQCAF3QK309RXP8RvwYRtncdILHyODvuY5I6Bo6frPZTTQbr095/qCERIWEU/efwixw7AznPr4+HOcuwhIyGkkr+h1b44QRX4NBkyf0rqZKAkkAA0BAaxRQUOBKmCQpqIH0ElZUJ3O/XgarLoltuRUyc8xPlscHSEdVCglT5Sy4KkRfOOYkc0OEeleoQOGvUpiLBzD+0SHRx2jtn/IfIwhF0jUNJEE06wBITQqapgzzKaRJ2vLj7H0dNOBkroolefd1E+NrtMGZNHLQYQB7CNxvCXH/sU+wStstWkNvhJy1z25de+/oqfqoR5cRQcbsFOqYaC4s6fdbjTK89numjCqeEKYjZHO72I6hlndgKRJwwMIt0PdWLb5/rHH3Hd40VoqktOPj0MCKasAgzAqAaPu599aBlrlnTdhfi/WBBabCft7W9+Z7sCyLJVPLhti7Cu0w+bK3wnfEGwc5oQZlD7oqq6gU499cV8HVOEPe6mk11t6+5le3v2sR/SqoJVRpT+6j1nsKdIDN1J0X5NmowBEJNfIyBqmUKGs+mE+/qq6smVCp7A4+77rReT3t1J3pnkDOSS4OvfQMFHnRKOc3Mjm9PLAM9epVuM+oaHHds7g+IX/u5Fn5WCFjQsiCIoqkb6BreKXkb2HB2OgAm8S/pkG4Xttq6c+i+6P/CCeQvYWTRQTgl2hJWkCvz1vXexURpkkwox8LX194cKCpqpyNDNP1PYEdMb2Zlz5rEldAHIKTNnC7ftYFz74qln8HsEH3Vx1FrkInpvIgOvygQ1YDvt+Z/oELL86CBxCyIkfuAhdhsAgECfIbbpTFitvkpBJy3i520R07mVjt6KDDnIA1Xif9HWoEnqMYCISrgMdCtF+b2TDl1dff/dbPk9d5Hxz92T7n+f1cIOnym+sPRkCjUuSr0BzhGI6rKedx9yvgFquuQRY6se0IBmMrCayn8mggHAH0ClWJbvXe4LLum8nIjfyTsLd/f9A201uaWPNB/LziCrrkn6MbCPztvfsO4+7o4taq2KVIgvnHqm6DVrotuEnNIhOmo9pOHSFgQMcUo4MQiHJ9kEGuC+MbIFFOSTh06ysaGD/k9qoWpdEgACOFxBxO92SedT5AOO24Td0t+ffR5rEGwruZUz74Jh4BGy1P/AJXDJ+RTMQyQFTBdcKLqLmL2OtJeiBokSvB1lkxcNBKUtt/aVMwC3xtzegfN5IcCtvNO7Vyj4xd/RJZ2HXAbIKvejJzcy2AhECZ5r17RgazBqIU0EkfvztMDp1ot7Xn6e4ZZlUTqP9vqd0jSB6K2LARxwCQhSI2noxLSKevUH7hLDAACMSjXgOQrQcBURv+w9fYjqcgNtDbrZIk8/7HD2Ubq6Og2pThg3Lx0MzMLxv7/6kvV10ucSMhw6paglALg2i9IhgaNQcX6Vc7+4brffiWIA4IKqFthf0Oox4tNyj2Ocv6YDIm7pC3QzzbzZwRw83OpV/Q7756WQOl2s9ic4WN7d+gznHB3JLXzZkIt0YMGict5bdcp+VtCS576pKltTLt+IwCAiU00cIlAxXP/67BOsyyXkFLaisDVYJdhnLq4vrt+lwgAGBvoYAqU6JRjZnIJkwNHLKeGwl450uCB+IYyOshfHyMz9MLTl2G+i/QoKvCJWshxLuT9Uck2WKjHAHVTHt4D/Wx3rhZMOheaRMfCLZ5/vWD4pD6dXi8XSpMAoAwdEY5GVHeWdjGz7Bavu3HrnswEycLjlEUkAIkZUUJePua6EtmyNg/YTpQIAtiwndNPEbT3Q9HX3vr3c48ytehz2OIu2B5Oa3PTSpMLsBBfoo6nWmZlhljiJ2AcEEsBsurDGz7acEzxOz0QM4E2KYOSdMOPjm+8VbIR1eQPpL8ewgpNW/lpUn/uPLz/HYEh0S18jB6GpgvPmbuV0v4N64hSsQne7OurHqUxRfEBY9UdHRiY1u1Mw/6C+NQjE9UmVSD5ooujFItvCiwpdyLXQFNF+Rcct7V2SfY0wW3wc0eokQk/dTF6C/SNiF2D4m3+d7AFwaVWVKuE4ErK648i1GW7MpZDmTmsUdmOLYIV9Ya+YcS9onCmsL8iLpXQBiyi9RPEFvVN8cx20r0UFGO4XO0Z4IySbA/M37jkMz8HvPb3JFeRTye300lPOcM3j9LJPwFhAuNUuobGc6ip+dtbhzttjxfnS8PvjgqAegF3kI/Dqu7uEXfv4cc5BQoQFPF5cLPAQBVlvcZEA/M5vFTTl1BXOAGixW+f0MvAzgdXWT31cM4qPOeZB7XjzNbb+7bfyv52+fJ6OqyL4pJ/U7eKc1EBRaAMnYpznC2LqB64zpoKzaZvvQgrCKUqvUVQepwSvz92CO/xajjiSNdHhIxVp7qzZwpBjT1H8QjcHNBg3fen+CmjK3meL5rVIAMpcFilYSNxSAJC2+olH+SWRdgTav/OtwZYLaWtQfu/djQGc4HKazd6u0/dzm49ngU6hOVUW4zNs711x5nlCbegdOtz19LY3hRDeLzinD8Xo4yeeLCzn58X7jxIbgX/75mZhVXxO+wyEo4ymiqDiDICQ0ln0PNRPldsVCRAC+Im1Gzc96oqTubQ1+NcUSkw29VCoaVF6HwWVDJIQBfdLp4kPyQSpM44yIP5rL/gAg+elKP2YTg+OjTr7B6DM7+jePsT+c0ofpt2bGhfvPacyxc9qaUfhkoXODACuwZ3bu4qL5H87Q5V/7fhFJU2hAYvmswwgw9Rct5oDXa3DQjKkgFd3bGN3upxBR9dx2eeShcfnsOD+0UvXTeMiUKd0HompEC/9JMSfu4LOKiBwRpoTbk3+8Ycv47EBRP3ABadPbX1D9Jo/P0gSwgYaM6eEcGN/e+6Foc51XH7Oe5nouO//pfsDRASblWj9swC1NMXta5zmOQMgw/M6J0SFeaYeYAhv8aY1z2xi2zxCPf/dWeeyaRJbg9i+ekTg5gpD4DeJmOvq5YiZx7+/8IOs1eE68HgxJm4dhIA4fzOmzWC4pvszdFvvDz/ycXYzXeHlFs4NNf4A17RL0NAdFDloVCAF4B6Cz5Mzl98dHByy+tTic3ggGafeIe7E2pecL5MNekBLNS0BbovmeVhwkqRkPBac+it8NkpikK9rwoQ12Ywl4AESA+9SVahXINpv0dbgP33gEmHUGoS5uva8VnbVA3eTG6g7sPfSMeT3Ca6sOoqiyfzs4ktZOzEdXFbitKJMIWPhh48/if3ZcSeyxqIz8JiIoqi4oZBgK3z7hy61/fL3FXYT/PlN/0hHhGUvCIFDF+4SRBQhp3TZsSewkymwyC0UvvtdgUHRXg6M/Upa+d2uFf8u3V7kNFaQubNGP/c5YW/P+g5aUp0smucM4Imb2jvPuaZNaRujAxQkYfp0hXUCcf4njEIAeFWID/jPdF34X71nsbDqU0h8//h7zmS/ee4pYR68eHnnNgpvNUiebs4BLEDUK8m42EcOR9hT3kM7B8OkNsDecAQ5yIDAERyjOOFyy+seW89+SoxKZ3JqW2d77c88wRDGzU/6Pa3GWO2xXeuUTqS7/m4nRgv8PkiMdjMxgp7BftZPgUPq6FhxE6lU86ZNI/XuGB5o1G0G/oEY+iuCKMHZtv0TP8pxWnICPsQz0DyKcwaQqwcPxLM6l0n2A1bL+jliI45sPYX5ksEEfvtSJ99qQ/w6UcI11E/S6UKEuxIlGLFWkkTR3vpBV9YGhyO3VcdeP+Lp//2GB1jfIecoNfa8afmOPq0m4t8QIJw3HLq+9ciD7NtLL2YLXbb/TqaxxF/Q9ND2rezHmza4FA9G/KhQww4AJ37UnV8+CLwuPFCVdOgtgA1SYwDJUVW3eD0Q7W967GGGiSlKEG+/Sas37rp3SziCfDtJFCoSptiqjg1sF61mQfVNFXCoqgNW/LvpotbP3v2bQMRvwTFAEtGVFBXqRSnPPKuU/CcuGvnexnWOKp+K+aqaluy0nmcA9EXNLMzhDXqQasBRNXcQyrUR50fvgV62GsYolzSHxPQvk87olf7zhafZ90jXF+0KeJXHe0Q/+vKD97DnSYxFqnBQDfiLFPyHyEzYR/8fdAnnT+ieRhBw2ITAMNdQgBgwlOBrcSEUcBPHHLj1kQeEx37Rli+Hn8ImOA052hSK8vn5aaf1vAoAqyBJpNf5qcgr7wi5BKsyBBa2RWhVwVoLK/X9a8PmV8gafBQ7n7btRAkXRT52zAkMHoXCRN158NUX2TMkDXzlzBZ2NvnyyxrI3iGvtx+TrYFvi2G25ZJseSt/1J+49LOHiLKbVBV89tDny7RCP0Vx9hEYVEeCERcM5U7apvvqGee6+hm4tQ/J5J6tb7KfU+wIXB3umPLGAtugOGZ0fwgaUp2sHQDUmwez9babm/ve3bVFZWN1TU1s2oL5KqssqCtD4HP0hsNxQZ1J+DGFjILnzG9mLXOPZIfRVuAsMgbOJIv/yPgY27p/P9tK0geClmyleHkv0l636hUiCTiIAoY5pPO3zFtA7sYL2AlkDMwTg0PjByi012PEnB4lfD9H27euoeagplIdYVZ+C4QD27azgW61DLGhhjWtW9negzYK+kw7AWAAzXihIlWS/jtr0YkqqhLXQdeLQxgQbPeKy5k3BgM2DFSSoXVqbT2bTsy3MXcWo4d2XQ6QZAKD6jBiDEgsNPm56NPV1wZKwde9r7zKFG8DdtEOwEKrkbwKgAfUv06ipWbrZdhPAI4/MAJtiRA9TkzAJIOBMBjAzU+9w70Mth1n/0G52vlCpIj4LfqRa1kuF2jcnjNvBMTDygxbb3+p4vuh3v0qqnGvAwg3PMAdR+atfgxgDioifgCrg3aKabyAAZD4UsAdVGBMSyQTR8CScWbAETTzsOQxANFfSkfwgQkdtFNM4wUMoOOG9nU+4JPKeoiMVlEly+iSHYyoWjXtlDMGrLlmzT2VuNBBO8U0XsAAADzpMOtUdgJ16eiIEEZuExC+NS8MBpRiQKXObwdMB8040fYkBkDh7ZTbAYaisAPYsWfpYcYuYMeK+a4SA9bcsuaayrqpLh0040TbkxgAmdPWKu5LtBKABXxuYCwRzXpsPg0GwmIgP6c0ET/g0yIBOND2JAbATwll1B4PhqOKjg55DySxM2xu5kfMu4TJYTDgigE+l3LzyjVj8JegFeXOXUTT1glAO2STGAB/Oa5eCtAh0tg7IvqOocowCh1liWyijOa5wYAXBmgOYS5hTulMWmhFQNPODICptwPEIwFkhyk7YLqHTeeUMHUnAwM6bP2Te6aJVtZPbsl2HNj+knyFldsBINIM7lPr02yHWeo7dDZwcaMSSKHLZMrNFUiPGvV9O55BI8rFf2pARNOOEgAOCjhtGdgBDfJdE2fzCQokgSwj8FnQZC83DIDwucAfnfSog0ZAy9bhn+IhdGQAyERBZu8qzhz2Nzqn+GBDIJCygpzxHAyEvDIplBUSMVOiI37Qhg4G4EbLQgaQGVWvBmDuHOruScwUwuAabSAxw5EYQDAnoiR8q+O6aMONloUMIHdpqPKzAf17ZC5MtFCi/5MPtPEZ0I/ohLeQXwhoLsRB/ECPJtrodLsAWMgAcuO1WvW4JcIY6NSpiIw8Tk2bZwnBQIxzQJfxjzDrSsOuDEBkOQw7XJo4XViwspwfk4Abf0JXZypIAwZyFv64Vn0LRbpowouGXRkA3w1wcB+0gA76iWChOo46BoVncjkwgSwjyIuGkzOZJynFAB/THOFnrfzxdgS0oCeALlsrsv5bPXZlAMhUMc7usDKr/Ox7R3yHu8p2wtWVswAbiSAcGpNUOj+WubFNAGy6aEGGdj0ZwKab29eSSKzcdD+kieupHs+saGh2C1TjNY76stJcfEY+pz5j5QctKE9Es5x2PSr2ZAC8/Bhb41FPoNcD7yZrR8CtE5wRcLUgxwzyK4lbKfMuVgzQGGXF/axKF7ee74QLbTQgSbNSDKBy3N2S6NQxmWcId6xD95FpO0weTCSL/jHBsitLmBpNWVUYsI8HxiiJRG/1FXNfdchvq25ZmpViANhHJB6q/HwAgD349g4L5lR9WhJBwQSzuEKqelIiwNpw7zg2CeymrrkPWnXb+7ejQooBoEB1Xd0d9oKqvkP/SfaOgHdPrQmHOZgVOb3LmByKMEBI56s+qiMVrYAhK2pCRzWY81p0fwK2qkJeYpdmAI9942ZIAF06kKHLCqoDVrc6s9MP1mXonPSRm5xuZcw7/xjIM1ngmJN8egjf6q3GOd9VHPjTatPpU5oB5Aqvcqok7DNwQh2HIMLCFa58jhFQJZZeyiduuErLtvRkHE7gN21IwVzXtfoTLnzRKOehfhB4ztfbuonpNvopI5M3kmvEZADRmAd3GSLxSLIa2ym1qi3GmRbx3gv/Gq77yjaJsF83tjd5tW9/71cCgJrl6ltsr9zPdxyF7N+120+R1OXNqwi57cSJrQTqim9WnLruewMMHFh44N+hSllYw4qf/oQ5rutIfBDa9M0A6mtYOw2ScscgDC38oXUhJ5lTJzvBocfyeZ+b/NaKl0yY1UKV7yv1Pdv9CZyobSn+2vgip+s0LNEkp02f3az0mZ91resYnH9BSx2Va/Vb1iv/OMnGY0PDrLZRuYbh1XRy3ueYAKjBHrqME4r1LjnQykPC+zORPduf3Kqe5n5NdMnzG6761uX3Qii89dHr2+/xBKIog28JAOV1SgEwkJSeQbAI65I/uc6bUxey+i9WRzAGqsAiGutTss5Islkw0ecErFlRPnvIKvc9EmCS0YjWeR1w9QdmAjEAnDAKom/IDgU45TgFETXJCQMW8dhEZRoMEJpFbNZ3/puqsH5zpmERJ3+e/WGVyzMX2zsOAbLl6rfqsn/a33EC59ugWN1tsPKKyvM/zGXMaV0JtOh16k/UdiAGgMp0SgEIGrJfI8JEyEjz8yypTRBd/jfNDusfCJL/0bOscQ33JWTfWu/wy/4uT9D5WqzaJj4tQudl04xETbBjLuuI9MvBDbH6o3xgBqBbCtAqMmkaaFOtwUAxBnTP4zCrP2C1CYTFoMv9Pueati2Us1kut79cFZWVbNaiE1mGPk0yGEgbBiD6Y89f2+pPnrl03dfCMHgJLAHYGl1l+670q1EFlKLTVBYxBrSK/tm+hKa90Evrjg0dnfP+pKWVjELNOvA7eugQy1RUsuqGeh3VmzoNBrRgoJ9iXQzs3aulblSKyz6evLn9irANqJAAWFWlP/9jv0Af3LlT2/6pX1hMfoMBLwxgrx9zVmdSRXNKGEDu9NEanR3u7dpqtgZ1ItjUrQQD0PsxVzWnNX5O/LnBooQBoAEKP3wFmRS1uAijfrhRRoBYNGWSwUBgDGCOanVnJxrjtBYYwsKCoW0AVnVwET7yghaE+l1mPVP9OTo8TFLAGKuZNk111aY+g4HQGDi4Yycb7O0NXY9HBV/aeH17h0ce6dcZ6ZySGc++uu0hMgi2SmYPlG36/PmsdqavU4+B2jGFDAZkMYCbffZv1+ftBzhyhr+lsjDJ5KuSyeQnT9U4WzFawZ4hxzFtJ3qA6Kr6OlZVW+sHtLLJaw+xhq3U0YHBQH2vqK5mlVNq8mWrGxry382XCQzA6Keb+KFeV42xFROtqvmmXAIAWEuuaVtJDqXXqQHRuRY4CTUee0zZMQHol2OkCg33DzA2NsaGBwa4cRTqkVbd0zYMCN5SScwBDloUK5L8SStYNTFkzjDoXTklEH/PG2/qdPbh6CRCXbXppvaVqnGrhQEAyChUAUgATcQEStVTECs5CH2UJhmIW2MYKaXzqoYkBc4kaHzAGEpVcoDFv5uIX9cRX2tQdIj+Vt3KVYB8xRGoAkA8BqAUmAD6MkLEzomeVnXdk8oaJx2fnFER87InMGtIC2AGpaC+RUX8ukR/a2y0SQBo4Nyvt7WNjbPbrMZ0faZREsCKfqh3f5bgiVg0+ovrQnuoeqHCgRngb8qM6VxiCFVhhIUjI37qU0WGXfH4je3turqnlQEA6LOvabuTGtG2NWghpq6piU1bMN/6mchPnAwbIqLHChmVvp5IRDgABZUBqkMNMYMp06c75EjOI5zt13Wjj72XZEdb++RN7ZfZn6n+rk0FsACdWsNW9A3TtqDGXQG0ZQ1IkpgAVgqs8rqPhFq4TvMnGOIA/ui6OCQwAf5HDCFJNp6oiB+i/9Rq9Vb/4jmiXQJAgy3XtrWS/85DxY3r+B23JGCIXv2oJoUZREb8hMLKCrZUlbuv24go8wR0a2T7wx1dFEgUzKbVLZ+KdzCeIbBozdQGOkWozNPZEzQY7xDy+QDddQhvMJxiNEkNBoBLSFGD+/ZxvFbQSRioDFElMHXc42dJJ7rbJUJZRXr/Gt3toP5IJACrI1HZA9BeFIZBTIwB8gDDsU+j01ujHM0nGEDdrFmsjjxCdaoIGOMotvosrEWh91tt4VO7DcDeGLcHDLFmerbY/lzHd51bhFjt4foZ1YqgAz9prxMMF0du8Qe1D67hqv0NoiZ+GpNO0EiUYxOpBICOUQixxSR3PKTbKGghEZLA9KMWKPEYBNEPkpEqLQ45Fg7K5RO7CLU5ZhC2z1hA9r+1LTp/DJykHWdLKcRXZ1jY/ZSPnAEAuCVXty0bz7A7/QAaJm9Yt2EQft9ufVc6hembKTsZA1APGg4/PPCBMRB/FO69dsgpuOdlm25uX2t/FsX3SIyAxR15+5GOVxZc2NJL+s6Hit/p+I0bhwb27iP/9RpWBd91icT1ewrrhEMeWPXx26R0YABjxY2G3T0USGKMxrxW2iAMZt+zpYtO3tHsjCjB2Yf8/NdE1FxBM7FIABYEpA7cTt+XW7+j+KyfPZtNnXeEa1OI59ZPK37avPOmHnEkO5r+mptmTupf58632V76G+zeN+ldqT+ABFhPEkH9YbNdu4rz/LifMuK0hsT+SPV+e/9iZQAAJMqdAavj0BVnNB89yXqcRlEfRP+X51/Ilp18Kmus9ZZuuogBrNvyOvu3p5+gle51CyX5z8aFx7H7vvA3+d9hviz5euiYlWGan1RWpBpAYkAkn6htO1Fb/CchhB5EugvgBECUOwNW+xjofZtf50wARkJY9RHCOU1bebW0yl/5kWWc8K1+yXxCOljetIQtP3MJZwTf/M2/l41UgPGFSgd7znRyG8euAfR97WG8nAcmcou/ExixMwDcMNS6sm1p3xD3FNS+PWghAZNh32ub+e4AJkGaElbpX//F56VWfLd+LSbpoRxVAow99vbB/GMa+06K67c06H1+bmPq9110rnIukAERQAhliXQLBCDFNAFcsOH+6hRauSGiy4j77jXRSZOXnvfKUtLvYxr7xBA/Bjd2CcCaYWACZBRcEaWPgNV2Wj6h7992yTIhuD2DA5yon925g71IBj8rvfeY49jpc+exVvq0M45fbHzYymI+o8BAdq9/RRJWfqu7iWEAAAhOEMQElhomYA1P4ef3PvHfCwjY/rZ943r26wfuJYmGQoUVpd+Tse/39OzbZCQ8kYyF173vYp7joI1JFBWZ9POD/+eHjkbDSRnNA2cMxOTo4wzMxNNEMQCAZZjAxODYv0H0h87ulFaQIe/Fpzc5vSp4BuaAfH9Ooj+kCZMiwkBCiR+9T4QNoHgYuDskuUWSJECeHCYBA9aqXYyNK36/Vor47eXACJy2AO15zHdFGEgw8aOHiWQAACzPBGIwDKL9JCWs1k7OPdjPf5REf5MSi4HOOPz7/WAjsQwAnQATiGt3wA8Sdee9iPR2p3QHOfOYlFgM8LnLF7LEgphgCcDCWZxbhBYMcX9etPBYRxBk9H7HguahbgwkaqvPrbOJMwI6AQsmAGehg0PsdvJdFu+DORUugWfNjZN9+yH+R5lOITXkRYkGB3v2laVzkYUauPfCuzVJW30WbE6fqWAAADyH0MviOEDkhLgonznp/z10d0CUyc3/wA7HqgfvZb9/4B77o3L6voai+K5IU4cTbQNwQiTpVCtwfNLpXTk9e/adHeXU3cT3FXMSczPxgBYBmDoGAPhxUQICKJTzNuEM8mM3KQEYoG0+zEWdl3fo7GUqGQAQwqOnwFegTLcJRU5BOieLqXsSBvg2XxyRfCZBEvBBamwATv3DFks5GAcRzKOY4J0Mg044UvUMMOCsgVdCvIFySGkz9onGJNUMAJ2yjINRXEkuQqLu5yCqSQyAzvXDQciPP38YOL9GHofGezCLQQossqpj5bdXhsFnUsqmVgUoRiDuTsdtKqVoF1i/5Y3i7vLfHzvzHMfn5qEmDJC+z2/sKRHiB5ZKhgGgM7hKqaGaLYR4ht+lkp562fncPqL6IDKQSfoxgDmFuRXFdV36ezPRQkkxAHQLKgFuVOVbhSVymAhRe5wcf3C2//uf+TxFtvGOBTgx5OabLwzQHMJcwpxKi3OPn/6VHAOwOo9tmcpRdgZFd15nPUvz50101t8pwTbwI4oQ5IcJ8KhC37jJVxmntkv9GeYO5lBat/hkxieWewFkAFORZ/sjHT07Hum4g19MmuHXkaV28xwutocRsS86bM4k1MydNp194tzz2b7qaraN8jkFBYGqcC6pDN/91F/ygKC1VdXsV5tfYajXnpDvs5SvOP0LHTwqzlucp2R+Y2+fsVufuLl9BeZQyfTLoSPUz/JILVe1NY9k6CxBhrWmtcdY5bHaF+8IFPcHuwZdNsLGlqGTO7GT264oLHi5RATCql81zlZ03NLeVYzXUvyd+m1A2UHJDehSOkuwnHYKbovqbkJZ+GTyYWW/kqL//Jz0fieCturAO7f3Vj7ECUSoMJMIA7AXjZOuf3P7mnLCR8naAESDSM5Da2DNpfepHGjs+//5D77jaBQU9Vn03EuSEJUrwed8TmBulGDfXLtU0jYAUc+71nUM7tjQcddRF7WsHxtjzaQWNIvyJvH52MgIu5908o093VwdaJS879DeF+wqwLmnWK8vJxsAF/cr2YpNN7avxpyw46dcvpeNDcBtQLlaQGH3KE+zW76kvoPe/pGT38OZQSt9d0qwC8CdF05F8CsQXQgCBvB+ByPgAxRMVFTGqb2EP+si+FaV44pfPC6GAdgwwt2JM+zyNNoHbN3gX+EmDKOhcd+1YQbW/XG2Gl6jtqdl/dUwgKLhp8NFjf1DrG28RBhBUffK82eO8OtrWHspOvOEGVTDAATYM4xAgJg0PTaE7zlahgF4oMgwAg8EJfG1IXzpUTEMQBpVjKXdWOijq2nN2kWAG+Oej9EzDMAHsqysLde2tY6MscsJeWUXodjCQZI+cVKvqoKtLrWTelHg2DCAEFiGe/EojIUVbHkp7ByEQEX0RUnMZ2NsTSVZ9cvFbVcHkg0DUITVJVe3LRvLsM8ZqUARQgXVYLWvGGd3pDkOn6BrsTw2DEAx2mE07BviqsHlVPVixdWXa3Wd1PHVdE3cWrONp3YKGAagFp8FtUFFGK9ky0bH2KVpPoVY0KmIfvCz+BXsrswoW2tEfH1INwxAH24LarZJBhfRybNlxmZQgB7rNN5aerrerPRFuNH40zAAjch1q5q2FBcT8peNjbOLylU6wCpP4bbWQ69HiHc3fJl3ejBgGIAevPquFVuLRBCLR8EQsraDZt+VJLtAFxF6ZyURPDG8TrNll4zBMgwgGeMwCQqoDIMjbDEdV24dY+x0GqhmypQWo2InEXsXBZt4tqKCrautYp3GeDdpiBPxwDCARAyDPBBQHSg2fSMYA0kMM4jQOFOIWo2A+A6oaQJ1Utu9IHQydvYYUV5+LJOQ0zCAJIyCQhiw88CquLTAxkZZI1FoOKlhnHVWVJLTDdII6zIWeYWDlYCq/j+A3luv/BDsyQAAAABJRU5ErkJggg== provider_display_name: Pivotal support_url: TBA plan_updatable: true plans:
- description: 'Example: This plan will configure a lightweight kubernetes cluster.
Not recommended for production workloads.'
instance_groups:
- azs:
- az-1
- az-2
- az-3 instances: 1 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
- azs:
- az-1
- az-2
- az-3 instances: 3 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
- azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
name: wavefront-proxy-errand
-
name: telemetry-pod pre_delete:
-
name: drain-cluster metadata: master_instances: 1 worker_instances: 3 name: Plan 1 plan_id: 8A0E21A8-8072-4D80-B365-D1F502085560 properties: addons-spec: |+
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: namespace-sink-modifier rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group
resources: ["configmaps"]
verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks
- apiGroups: ["apps.pivotal.io"] resources: ["sinks"] verbs: ["get", "list", "watch"]
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: service-account-namespace-sink-modifier subjects:
- kind: ServiceAccount name: default namespace: pks-system roleRef: kind: ClusterRole name: namespace-sink-modifier apiGroup: rbac.authorization.k8s.io
https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Addr example.com:12345
parsers.conf: | [PARSER] Name apache Format regex Regex ^(?
[^ ]) [^ ] (? [^ ]) [(?)] "(? \S+)(?: +(? [^"]?)(?: +\S)?)?" (? [^ ]) (?[^ ] )(?: "(?[^"])" "(? [^"] )")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z[PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.2 imagePullPolicy: Always name: fluent-bit resources: limits: memory: 100Mi requests: cpu: 100m memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true serviceAccountName: default terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.4 imagePullPolicy: Always name: sink-controller
- apiGroups: [""] # "" indicates the core API group
resources: ["configmaps"]
verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
-
- azs:
- description: 'Example: This plan will configure a medium sized kubernetes
cluster, suitable for more pods.'
instance_groups:
-
azs:
- az-1
- az-2
- az-3 instances: 3 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
-
azs:
- az-1
- az-2
- az-3 instances: 5 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
-
azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
name: wavefront-proxy-errand
-
name: telemetry-pod pre_delete:
-
name: drain-cluster metadata: master_instances: 3 worker_instances: 5 name: multi-master plan_id: 58375a45-17f7-4291-acf1-455bfdc8e371 properties: addons-spec: |
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: namespace-sink-modifier rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group
resources: ["configmaps"]
verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks
- apiGroups: ["apps.pivotal.io"] resources: ["sinks"] verbs: ["get", "list", "watch"]
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: service-account-namespace-sink-modifier subjects:
- kind: ServiceAccount name: default namespace: pks-system roleRef: kind: ClusterRole name: namespace-sink-modifier apiGroup: rbac.authorization.k8s.io
https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Addr example.com:12345
parsers.conf: | [PARSER] Name apache Format regex Regex ^(?
[^ ]) [^ ] (? [^ ]) [(?)] "(? \S+)(?: +(? [^"]?)(?: +\S)?)?" (? [^ ]) (?[^ ] )(?: "(?[^"])" "(? [^"] )")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z[PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.2 imagePullPolicy: Always name: fluent-bit resources: limits: memory: 100Mi requests: cpu: 100m memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true serviceAccountName: default terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.4 imagePullPolicy: Always name: sink-controller
- apiGroups: [""] # "" indicates the core API group
resources: ["configmaps"]
verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
allow-privileged-containers: true disable_deny_escalating_exec: true
-
-
- null service_description: Default on-demand Kubernetes service. service_name: p.pks tags:
- pivotal
- kubernetes
- k8s service_deployment: releases:
- jobs:
- kube-apiserver
- kube-controller-manager
- kube-scheduler
- kubelet
- kube-proxy
- kubernetes-roles
- flanneld
- nginx
- kubernetes-api-route-registrar
- apply-specs
- secure-var-vcap name: kubo version: 0.21.0
- jobs:
- etcd name: cfcr-etcd version: 1.4.1
- jobs:
- docker name: docker version: 32.0.3
- jobs:
- pks-nsx-t-prepare-master-vm
- pks-nsx-t-ncp name: pks-nsx-t version: 1.11.0
- jobs:
- ncp
- nsx-node-agent
- openvswitch
- nsx-cni
- nsx-kube-proxy name: nsx-cf-cni version: 2.3.0.10066840
- jobs:
- fluentd name: pks-vrli version: 0.6.0
- jobs:
- telemetry-pod name: pks-telemetry version: 0.9.2
- jobs:
- syslog_forwarder name: syslog-migration version: 11.1.1
- jobs:
- bpm name: bpm version: 0.6.0
- jobs:
- wavefront-proxy name: wavefront-proxy version: 0.8.0
- jobs:
- drain-cluster name: pks-helpers version: 50.0.0 stemcell: os: ubuntu-xenial version: "97.19" startup_banner: true username: ((odb_broker_basicauth.username)) stemcell: bosh-vsphere-esxi-ubuntu-xenial-go_agent update: max_in_flight: 1 vm_type: large name: pivotal-container-service-8f7873397b6d8f2b58f6 releases:
- name: cf-mysql version: 36.14.0
- name: docker version: 32.0.3
- name: kubo version: 0.21.0
- name: cfcr-etcd version: 1.4.1
- name: kubo-service-adapter version: 1.2.0-build.166
- name: on-demand-service-broker version: 0.22.0
- name: pks-api version: 1.2.0-build.166
- name: pks-helpers version: 50.0.0
- name: pks-nsx-t version: 1.11.0
- name: nsx-cf-cni version: 2.3.0.10066840
- name: pks-vrli version: 0.6.0
- name: syslog-migration version: 11.1.1
- name: pks-telemetry version: 0.9.2
- name: event-emitter version: 0.13.0
- name: uaa version: "60.2"
- name: bpm version: 0.6.0
- name: backup-and-restore-sdk version: 1.8.0
- name: wavefront-proxy version: 0.8.0 stemcells:
- alias: bosh-vsphere-esxi-ubuntu-xenial-go_agent os: ubuntu-xenial version: "97.19" update: canaries: 1 canary_watch_time: 30000-300000 max_errors: 2 max_in_flight: 1 serial: false update_watch_time: 30000-300000 variables:
- name: kubo_odb_ca options: common_name: ca is_ca: true type: certificate
- name: pks_api_internal
options:
alternative_names:
- localhost
- 127.0.0.1 ca: kubo_odb_ca common_name: localhost type: certificate
- name: uaa_jwt_signing_key_1 type: rsa
- name: uaa_active_pks_saml_key options: common_name: ca is_ca: true type: certificate
- name: pks_api_basicauth type: user
- name: odb_broker_basicauth type: user
- name: telemetry_ca options: common_name: ca is_ca: true type: certificate
- name: telemetry_server_tls options: ca: telemetry_ca common_name: telemetry.pks.internal type: certificate
Succeeded ubuntu@opsman:~$ #+END_EXAMPLE ** pks tile: with vrli disabled :noexport: #+BEGIN_EXAMPLE ubuntu@opsman:~$ bosh -d pivotal-container-service-8f7873397b6d8f2b58f6 manifest Using environment '30.0.0.11' as client 'ops_manager'
Using deployment 'pivotal-container-service-8f7873397b6d8f2b58f6'
instance_groups:
- azs:
- az-1 env: bosh: password: $6$b7ce3e45788446e7$xSFtkreCw31eikORMpXZEIMQJraCEEns2odZRIw3CFT9gt0BjLzsAv1FJjCGM9v5qVSMt8L5oMUYf94lK9PE/. instances: 1 jobs:
- consumes: {} name: service-adapter properties: deployment: broker_deployment_name: pivotal-container-service-8f7873397b6d8f2b58f6 director_url: https://30.0.0.11:25555 kubo_odb_ca: ((kubo_odb_ca.certificate)) syslog: migration: disabled: true provides: {} release: kubo-service-adapter
- consumes: broker: from: proxy-broker name: pks-api properties: pks: db_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_pks_db_password.value)) fqdn: api.pks.local internal_tls: certificate: ((pks_api_internal.certificate)) private_key: ((pks_api_internal.private_key)) password: ((pks_api_basicauth.password)) pks_client_secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_api_uaa_client.value)) telemetry: authenticationMode: service_account enabled: true eventEmitterBaseUrl: http://localhost:8012 tls: certificate: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.cert_pem)) private_key: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.private_key_pem)) uaa_service_admin_client_id: service_admin_client uaa_service_admin_client_secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_services_admin_uaa_client.value)) username: ((pks_api_basicauth.username)) provides: pks_api: as: pks_api_http pks_api_shared: as: pks_api_shared_http shared: true pks_uaa_service_admin_client: as: pks_uaa_service_admin_client shared: true release: pks-api
- consumes: {} name: bosh-update-config properties: bosh: authentication: uaa: client_id: pivotal-container-service-8f7873397b6d8f2b58f6 client_secret: ((/opsmgr/director/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_client_secret.value)) url: https://30.0.0.11:8443 root_ca_cert: | -----BEGIN CERTIFICATE----- MIIDUDCCAjigAwIBAgIUQw3pICIuRKxgSrFmy8WrU0cMIgkwDQYJKoZIhvcNAQEL BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTgxMDE0MjMy MjU1WhcNMjIxMDE1MjMyMjU1WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2 b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMzUbJTQ70Kq38zJ PKdMPbfkdV7KPc3L2+MyolQwFNWZQAIJ+LPKld/8zRFZfosqSzphHy9DKXY441r1 VvhhZdmwPcAHWyOq6we2pt60KjLvrA52oGcNnXImviOnEYM7mWYYoOA3DhBaUaB/ 0FjXYevKqtBDLZPirkGlZiMDfuqjHpzT5fmPavNoq3XyU32BflHQdEFPiavUzo7H QGTVMaUebQwRT7Z8HLlSUT621ARWDwVfbj4GYYKyXNdniqCELsTBemPMQdBKVzUe VpTWfNGJH9uR467H5SPyp+DtihFbpoZIyw3OhsYLRYEwxsOz4576SIPzJ1pubueP VhwzkpcCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUJ2J9H598MNr2iKzqaL6N4pWhBCIw HwYDVR0jBBgwFoAUJ2J9H598MNr2iKzqaL6N4pWhBCIwHQYDVR0lBBYwFAYIKwYB BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG MA0GCSqGSIb3DQEBCwUAA4IBAQAx5QdKSJ9N/7T7g80cdpmJsBNjJmpCERTpp5XA h7TNI75+kjTj2rgoXpGUpQ8yPO/g4xZS0eNh1ZmQXksVoGOWEQUTp4Cey8QraOab d28kaWrY8UHQj6DQIvHRuzLGWf/pMsOcH1KU2GK9+YKUHRtpvVI9gUGkHamFdXS3 YsdabMzHQgpvVWl4tnXb+fmTfygMHyS5bVxcYgZ4+5MaCxg7QbnHMNfNKGA/MMHr 3dlzV08GOyX4cpbJGlv/AtZcHLbWNu2X+idjHP1wlHcF5g7v9SHleeijXXCaHtb4 3OFNePJv/g7KmMllI9xHzQSbAp0dlq+xjHzqBOGmajTDWPWB -----END CERTIFICATE----- url: https://30.0.0.11:25555 cloud_config: vm_extensions: - cloud_properties: vmx_options: disk.enableUUID: "1" name: disk_enable_uuid cloud_config_name: pivotal-container-service-8f7873397b6d8f2b58f6 provides: {} release: pks-api
- consumes: {} name: broker provides: broker: as: odb-broker release: on-demand-service-broker
- consumes: broker: from: odb-broker name: pks-nsx-t-osb-proxy properties: bosh: authentication: uaa: client_id: pivotal-container-service-8f7873397b6d8f2b58f6 client_secret: ((/opsmgr/director/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_client_secret.value)) cloud_config_dns: 192.168.115.1 cloud_config_prefix: pks log_level: INFO root_ca_cert: | -----BEGIN CERTIFICATE----- MIIDUDCCAjigAwIBAgIUQw3pICIuRKxgSrFmy8WrU0cMIgkwDQYJKoZIhvcNAQEL BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTgxMDE0MjMy MjU1WhcNMjIxMDE1MjMyMjU1WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2 b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMzUbJTQ70Kq38zJ PKdMPbfkdV7KPc3L2+MyolQwFNWZQAIJ+LPKld/8zRFZfosqSzphHy9DKXY441r1 VvhhZdmwPcAHWyOq6we2pt60KjLvrA52oGcNnXImviOnEYM7mWYYoOA3DhBaUaB/ 0FjXYevKqtBDLZPirkGlZiMDfuqjHpzT5fmPavNoq3XyU32BflHQdEFPiavUzo7H QGTVMaUebQwRT7Z8HLlSUT621ARWDwVfbj4GYYKyXNdniqCELsTBemPMQdBKVzUe VpTWfNGJH9uR467H5SPyp+DtihFbpoZIyw3OhsYLRYEwxsOz4576SIPzJ1pubueP VhwzkpcCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUJ2J9H598MNr2iKzqaL6N4pWhBCIw HwYDVR0jBBgwFoAUJ2J9H598MNr2iKzqaL6N4pWhBCIwHQYDVR0lBBYwFAYIKwYB BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG MA0GCSqGSIb3DQEBCwUAA4IBAQAx5QdKSJ9N/7T7g80cdpmJsBNjJmpCERTpp5XA h7TNI75+kjTj2rgoXpGUpQ8yPO/g4xZS0eNh1ZmQXksVoGOWEQUTp4Cey8QraOab d28kaWrY8UHQj6DQIvHRuzLGWf/pMsOcH1KU2GK9+YKUHRtpvVI9gUGkHamFdXS3 YsdabMzHQgpvVWl4tnXb+fmTfygMHyS5bVxcYgZ4+5MaCxg7QbnHMNfNKGA/MMHr 3dlzV08GOyX4cpbJGlv/AtZcHLbWNu2X+idjHP1wlHcF5g7v9SHleeijXXCaHtb4 3OFNePJv/g7KmMllI9xHzQSbAp0dlq+xjHzqBOGmajTDWPWB -----END CERTIFICATE----- url: https://30.0.0.11:25555 create_network_with_lb: false enabled: true fip_address_parameter: nsxt_fip_address generate_lb_name: true kubernetes_master_host_parameter: kubernetes_master_host lb_service_id_parameter: nsxt_lb_service_id log_level: INFO network_name_parameter: nsxt_network_name nsxt: ca_cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWZ6AhVAMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE4MTAxNTIzMTQwOFoXDTIzMTAxNDIzMTQwOFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCE NhsvKNYS3EYkYnmHoeJdtPdlW5GwsRIh+eep9xiMX8tcU+n1a8LokthRJe5vxYjA /qTOiKZ1NdDxrwHnLQfmzUQa/xZeH73nHG4X8Rx0/+E0BHkPBnt9Uc/Kj6NUBBxq O3RnnebUA7+SmRr/1iDapgf2PZd/oCdXj9NG3mWSkEwPipalpfaEfGzxB83tLvGp km/PCMdUDx+kVkDH1o4dL8dpYxOpnaz3g+E2i6+ZoLhn1+8AQ2i5WxIXT5DEM3TP /HPrz9Cr43+NQ9AIrIj97lRpoe1nq/gPG/0rAHaAYyAsVgdR2BE3JI4gMCC58fGK 46B4FxFz30wOEOjHcSWRAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAGMrSeaeg/Jw mGao84dR7Uymn+KLW0W3Mptv3J115xzpfym+ZzDqaC/F9aCOM4qJ+RWex67aIM6c JqXEadM2Ul/COjIy5oFwf7925tas9p+3TJ8dCvmBvUdqSe3rb2Y8KDU2PkJssY1t FaKZ3FCB/fpRTAtleA5UJk3uA2u5z3G9ESyojhNF6HRwsawgcFwAH8pQflN//ouO eOwP5b7UiuMEi0Zpw4TwpjzUVzWsEkovr+oSPB2IPinWFIBJQQS2vguknuR/ssil VLnLYIlMlLMB7eNkEDUFL0r+5zRTV46Rz+RU7yGRstk5ckdZ5ieuabA4zYIzdPbs Q4EX1moA/Wc= -----END CERTIFICATE----- floating_ip_pool_ids: - 0cecb28c-79f9-47ea-be3c-ce0bfc176871 host: nsxmanager.pks.vmware.local insecure: true ip_block_id: b557d4d9-1c52-48c6-bad4-7c5fe174ca36 lb_size_large_supported: false lb_size_medium_supported: true log_level: INFO nat_mode: true network_prefix: pks superuser_cert: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) superuser_key: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) t0_router_id: 7c38157e-6107-4e70-94fe-26f2ac256995 password: ((odb_broker_basicauth.password)) port: 3000 username: ((odb_broker_basicauth.username)) provides: {} release: pks-nsx-t
- consumes: {} name: mysql properties: cf_mysql: mysql: admin_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_mysql_admin_password.value)) cluster_health: password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_mysql_cluster_health_password.value)) galera_healthcheck: db_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_mysql_galera_healthcheck_db_password.value)) endpoint_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_mysql_galera_healthcheck_endpoint_password.value)) seeded_databases: - name: pks password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_pks_db_password.value)) username: pks - name: uaa password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_uaa_db_password.value)) username: uaa - name: telemetry password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_telemetry_db_password.value)) username: telemetry provides: {} release: cf-mysql
- consumes: {} name: uaa properties: encryption: active_key_label: key-1 encryption_keys: - label: key-1 passphrase: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_encryption_passphrase.value)) login: saml: activeKeyId: active-pks-saml-key keys: active-pks-saml-key: certificate: ((uaa_active_pks_saml_key.certificate)) key: ((uaa_active_pks_saml_key.private_key)) passphrase: "" signatureAlgorithm: SHA256 release_level_backup: true uaa: clients: admin: authorities: uaa.admin,clients.read,clients.write,clients.secret,scim.read,scim.write,clients.admin,pks.clusters.admin,pks.clusters.manage authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_uaa_management_admin_client.value)) pks_cli: access-token-validity: 7200 authorities: uaa.resource authorized-grant-types: password,refresh_token refresh-token-validity: 21600 scope: pks.clusters.admin,pks.clusters.manage secret: "" pks_client: access-token-validity: 86400 authorities: pks.clusters.admin,pks.clusters.manage,uaa.resource authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_api_uaa_client.value)) pks_cluster_client: authorities: uaa.resource authorized-grant-types: password,client_credentials,user_token,refresh_token scope: openid,roles secret: "" service_admin_client: authorities: clients.admin authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pks_services_admin_uaa_client.value)) jwt: policy: active_key_id: key-1 keys: key-1: signingKey: ((uaa_jwt_signing_key_1.private_key)) ldap: emailDomain: [] enabled: false externalGroupsWhitelist: - '*' groups: groupSearchFilter: member={0} profile_type: no-groups searchBase: null mailAttributeName: mail referral: follow searchBase: null searchFilter: cn={0} sslCertificate: null sslCertificateAlias: null url: null userDN: null userPassword: null port: 35684 scim: groups: pks.clusters.admin: Allows a user to admin PKS pks.clusters.manage: Allows a user to manage PKS clusters user: override: true users: - groups: - uaa.admin - pks.clusters.admin name: admin password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_admin_password.value)) sslCertificate: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.cert_pem)) sslPrivateKey: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.private_key_pem)) url: https://api.pks.local:8443 uaadb: address: 127.0.0.1 databases: - name: uaa tag: uaa db_scheme: mysql port: 3306 roles: - name: uaa password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_uaa_db_password.value)) tag: admin provides: {} release: uaa
- consumes: {} name: bbr-uaadb provides: {} release: uaa
- consumes: {} name: database-backup-restorer provides: {} release: backup-and-restore-sdk
- consumes: broker: from: proxy-broker name: upgrade-all-service-instances properties: service_instances_api: authentication: basic: password: ((pks_api_basicauth.password)) username: ((pks_api_basicauth.username)) root_ca_cert: ((pks_api_internal.ca)) url: https://localhost:9021/service_instances provides: {} release: on-demand-service-broker
- consumes: {} name: syslog_forwarder properties: syslog: migration: disabled: true provides: {} release: syslog-migration
- consumes: pks_api: from: pks_api_http name: delete-all-clusters provides: {} release: pks-api
- consumes: {}
name: pks-nsx-t-precheck
properties:
floating-ip-pool-ids:
- 0cecb28c-79f9-47ea-be3c-ce0bfc176871 ip-block-id: 3a045baf-2f4c-4dc4-81bf-10f77ebb893a network-automation: true nodes-ip-block-id: b557d4d9-1c52-48c6-bad4-7c5fe174ca36 nsx-t-ca-cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWZ6AhVAMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE4MTAxNTIzMTQwOFoXDTIzMTAxNDIzMTQwOFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCE NhsvKNYS3EYkYnmHoeJdtPdlW5GwsRIh+eep9xiMX8tcU+n1a8LokthRJe5vxYjA /qTOiKZ1NdDxrwHnLQfmzUQa/xZeH73nHG4X8Rx0/+E0BHkPBnt9Uc/Kj6NUBBxq O3RnnebUA7+SmRr/1iDapgf2PZd/oCdXj9NG3mWSkEwPipalpfaEfGzxB83tLvGp km/PCMdUDx+kVkDH1o4dL8dpYxOpnaz3g+E2i6+ZoLhn1+8AQ2i5WxIXT5DEM3TP /HPrz9Cr43+NQ9AIrIj97lRpoe1nq/gPG/0rAHaAYyAsVgdR2BE3JI4gMCC58fGK 46B4FxFz30wOEOjHcSWRAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAGMrSeaeg/Jw mGao84dR7Uymn+KLW0W3Mptv3J115xzpfym+ZzDqaC/F9aCOM4qJ+RWex67aIM6c JqXEadM2Ul/COjIy5oFwf7925tas9p+3TJ8dCvmBvUdqSe3rb2Y8KDU2PkJssY1t FaKZ3FCB/fpRTAtleA5UJk3uA2u5z3G9ESyojhNF6HRwsawgcFwAH8pQflN//ouO eOwP5b7UiuMEi0Zpw4TwpjzUVzWsEkovr+oSPB2IPinWFIBJQQS2vguknuR/ssil VLnLYIlMlLMB7eNkEDUFL0r+5zRTV46Rz+RU7yGRstk5ckdZ5ieuabA4zYIzdPbs Q4EX1moA/Wc= -----END CERTIFICATE----- nsx-t-host: nsxmanager.pks.vmware.local nsx-t-insecure: true nsx-t-superuser-certificate: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) nsx-t-superuser-key: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) t0-router-id: 7c38157e-6107-4e70-94fe-26f2ac256995 vcenter-cluster: kubo-az-1 vcenter-datacenter: kubo-dc vcenter-host: 192.168.111.137 vcenter-insecure: true vcenter-password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cloud_provider/vsphere/vcenter_master_creds.password)) vcenter-user: [email protected] provides: {} release: pks-nsx-t
- consumes: {} name: pks-nsx-t-ops-files properties: fip_address_parameter: nsxt_fip_address lb-created-by-proxy: false nsx-t-ca-cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWZ6AhVAMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE4MTAxNTIzMTQwOFoXDTIzMTAxNDIzMTQwOFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCE NhsvKNYS3EYkYnmHoeJdtPdlW5GwsRIh+eep9xiMX8tcU+n1a8LokthRJe5vxYjA /qTOiKZ1NdDxrwHnLQfmzUQa/xZeH73nHG4X8Rx0/+E0BHkPBnt9Uc/Kj6NUBBxq O3RnnebUA7+SmRr/1iDapgf2PZd/oCdXj9NG3mWSkEwPipalpfaEfGzxB83tLvGp km/PCMdUDx+kVkDH1o4dL8dpYxOpnaz3g+E2i6+ZoLhn1+8AQ2i5WxIXT5DEM3TP /HPrz9Cr43+NQ9AIrIj97lRpoe1nq/gPG/0rAHaAYyAsVgdR2BE3JI4gMCC58fGK 46B4FxFz30wOEOjHcSWRAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAGMrSeaeg/Jw mGao84dR7Uymn+KLW0W3Mptv3J115xzpfym+ZzDqaC/F9aCOM4qJ+RWex67aIM6c JqXEadM2Ul/COjIy5oFwf7925tas9p+3TJ8dCvmBvUdqSe3rb2Y8KDU2PkJssY1t FaKZ3FCB/fpRTAtleA5UJk3uA2u5z3G9ESyojhNF6HRwsawgcFwAH8pQflN//ouO eOwP5b7UiuMEi0Zpw4TwpjzUVzWsEkovr+oSPB2IPinWFIBJQQS2vguknuR/ssil VLnLYIlMlLMB7eNkEDUFL0r+5zRTV46Rz+RU7yGRstk5ckdZ5ieuabA4zYIzdPbs Q4EX1moA/Wc= -----END CERTIFICATE----- nsx-t-host: nsxmanager.pks.vmware.local nsx-t-insecure: true nsx-t-superuser-certificate: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) nsx-t-superuser-key: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) t0-router-id: 7c38157e-6107-4e70-94fe-26f2ac256995 provides: {} release: pks-nsx-t
- consumes: {} name: pks-wavefront-ops-files properties: wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: wavefront-alert-creation properties: wavefront-alert-targets: ignored wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: wavefront-alert-deletion properties: wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: pks-vrli-ops-files properties: fluentd_vrli_ca_cert: ignored fluentd_vrli_host: ignored fluentd_vrli_rate_limit_msec: ignored fluentd_vrli_skip_cert_verify: ignored fluentd_vrli_use_ssl: ignored provides: {} release: pks-vrli
- consumes: pks_api: from: pks_api_http name: telemetry properties: pks_db_host: localhost pks_db_password: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/cf_mysql_telemetry_db_password.value)) pks_instance_id: pivotal-container-service-8f7873397b6d8f2b58f6 telemetry_enabled: true telemetry_url: http://35.188.203.192 tls_ca: certificate: ((telemetry_ca.certificate)) tls_server: certificate: ((telemetry_server_tls.certificate)) private_key: ((telemetry_server_tls.private_key)) vcenter_server: 192.168.111.137 provides: pks_telemetry: as: pks_telemetry shared: true release: pks-telemetry
- consumes: {} name: telemetry-pod-ops-files properties: interval: 600 telemetry_ca: | ((telemetry_ca.certificate)) provides: {} release: pks-telemetry
- consumes: {} name: event-emitter properties: aggregator_ca_cert: ((telemetry_ca.certificate)) aggregator_endpoint: https://telemetry.pks.internal:8011/api/send?objectType=Event&sourceID=PKSServer pks_cloud_provider: vSphere pks_tile_version: 1.2.0-build.47 server_port: 8012 provides: {} release: event-emitter lifecycle: service name: pivotal-container-service networks:
- default:
- dns
- gateway
name: deployment-network
persistent_disk_type: "10240"
properties:
bosh:
authentication:
uaa:
client_id: pivotal-container-service-8f7873397b6d8f2b58f6
client_secret: ((/opsmgr/director/pivotal-container-service-8f7873397b6d8f2b58f6/uaa_client_secret.value))
url: https://30.0.0.11:8443
root_ca_cert: |
-----BEGIN CERTIFICATE-----
MIIDUDCCAjigAwIBAgIUQw3pICIuRKxgSrFmy8WrU0cMIgkwDQYJKoZIhvcNAQEL
BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTgxMDE0MjMy
MjU1WhcNMjIxMDE1MjMyMjU1WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2
b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMzUbJTQ70Kq38zJ
PKdMPbfkdV7KPc3L2+MyolQwFNWZQAIJ+LPKld/8zRFZfosqSzphHy9DKXY441r1
VvhhZdmwPcAHWyOq6we2pt60KjLvrA52oGcNnXImviOnEYM7mWYYoOA3DhBaUaB/
0FjXYevKqtBDLZPirkGlZiMDfuqjHpzT5fmPavNoq3XyU32BflHQdEFPiavUzo7H
QGTVMaUebQwRT7Z8HLlSUT621ARWDwVfbj4GYYKyXNdniqCELsTBemPMQdBKVzUe
VpTWfNGJH9uR467H5SPyp+DtihFbpoZIyw3OhsYLRYEwxsOz4576SIPzJ1pubueP
VhwzkpcCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUJ2J9H598MNr2iKzqaL6N4pWhBCIw
HwYDVR0jBBgwFoAUJ2J9H598MNr2iKzqaL6N4pWhBCIwHQYDVR0lBBYwFAYIKwYB
BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG
MA0GCSqGSIb3DQEBCwUAA4IBAQAx5QdKSJ9N/7T7g80cdpmJsBNjJmpCERTpp5XA
h7TNI75+kjTj2rgoXpGUpQ8yPO/g4xZS0eNh1ZmQXksVoGOWEQUTp4Cey8QraOab
d28kaWrY8UHQj6DQIvHRuzLGWf/pMsOcH1KU2GK9+YKUHRtpvVI9gUGkHamFdXS3
YsdabMzHQgpvVWl4tnXb+fmTfygMHyS5bVxcYgZ4+5MaCxg7QbnHMNfNKGA/MMHr
3dlzV08GOyX4cpbJGlv/AtZcHLbWNu2X+idjHP1wlHcF5g7v9SHleeijXXCaHtb4
3OFNePJv/g7KmMllI9xHzQSbAp0dlq+xjHzqBOGmajTDWPWB
-----END CERTIFICATE-----
url: https://30.0.0.11:25555
disable_cf_startup_checks: true
expose_operational_errors: false
password: ((odb_broker_basicauth.password))
port: 8080
service_adapter:
path: /var/vcap/jobs/service-adapter/bin/service-adapter
service_catalog:
bindable: true
global_properties:
authorization_mode: rbac
deployments_network: service-network
iaas: vsphere
oidc: false
oidc_ca: ((/opsmgr/pivotal-container-service-8f7873397b6d8f2b58f6/pivotal-container-service/pks_tls.cert_pem))
oidc_client_id: pks_cluster_client
oidc_groups_claim: roles
oidc_groups_prefix: ""
oidc_issuer_url: https://api.pks.local:8443/oauth/token
oidc_username_claim: user_name
oidc_username_prefix: '-'
ops_files_paths:
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_pks_nsx_t.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/prepare_master_vm.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_fip_to_tls_certs.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/remove_flannel.yml
- /var/vcap/jobs/pks-wavefront-ops-files/manifests/add-wavefront-job.yml
- null
- /var/vcap/jobs/telemetry-pod-ops-files/manifests/add-agent-job.yml proxy: null routing_mode: external vcenter_dc: kubo-dc vcenter_ds: iscsi-ds-0 vcenter_ip: 192.168.111.137 vcenter_vms: pcf_vms worker_max_in_flight: 1 id: DF8EECC4-7225-42D0-8459-4A6C584314CA metadata: display_name: Kubernetes documentation_url: TBA image_url: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAAKp2lDQ1BJQ0MgUHJvZmlsZQAASImVlwdUFEkax6t7ciINDEHCkJPkDJLjkCWDqAwzhCGMw8CQzMqigmtARQQURVeSAosKiAkQBZVFMIB5QRYV5VwMiIo618Ax3N69u3v39atXv/d19b++rq56798AkHuYPF4KLAFAKjeDH+TpQo+IjKLjRgAauQhAAlCYrHSec2CgL0Biof9rfBoE0Gx/12BW69/v/9eQZMelswCAAhGOZaezUhE+i7RGFo+fAQCKjeTVszJ4s7wdYWk+UiDC5bOcMM+Nsxw7z11zY0KCXBF+AACezGTyEwAg/YHk6ZmsBESHjEbYmMvmcBE2R9iBlchE5iEj98DS1NQ1s3wUYZ3Yf9JJ+ItmrEiTyUwQ8fy7zAXejZPOS2Hm/J/L8b8jNUWwMIca0siJfK+g2fmQNatOXuMjYm6sf8ACc9jzNc1yosArdIFZ6a5RC8xmuvkssCA51HmBmfzFZzkZjJAF5q8JEulzU/x9RfpxDBHHpbsHL3A8x4OxwLmJIeELnMkJ81/g9ORgn8UxrqI8XxAkqjme7yF6x9T0xdpYzMW5MhJDvBZriBDVw45zcxfluaGi8bwMF5EmLyVwsf4UT1E+PTNY9GwGssEWOInpHbioEyhaH8ABfoAJWBlx2bP7Criu4eXwOQmJGXRn5JTE0RlcluFSuqmxiRUAs2du/pN+oM2dJYh2czG39QgA9meFQuGFxZxPGwBnCgEgDi3mtNcDINYBQE8ZS8DPnM/NbnWAAUQgDqSBPFAG6kAHGABTYAnsgBNwB94gAISASLAKsEAiSAV8kAXWgc0gHxSCPeAAKAUV4DioBqdBE2gBF0EHuA5ugX5wHzwGw2AMvAGT4BOYgSAIB1EgKiQPqUCakD5kCllDDpA75AsFQZFQDJQAcSEBtA7aChVCRVApdAyqgX6FzkMd0A1oAHoIjUDj0HvoK4yCybA0rARrwUawNewM+8Ah8Eo4AU6Dc+E8eBdcAlfCp+BmuAO+Bd+Hh+E38BQKoEgoGkoVZYCyRrmiAlBRqHgUH7UBVYAqRlWi6lFtqG7UXdQwagL1BY1FU9F0tAHaDu2FDkWz0GnoDeid6FJ0NboZ3YW+ix5BT6J/YCgYRYw+xhbDwERgEjBZmHxMMeYk5hzmGuY+ZgzzCYvF0rDaWCusFzYSm4Rdi92JPYxtwLZjB7Cj2CkcDieP08fZ4wJwTFwGLh93CHcKdwV3BzeG+4wn4VXwpngPfBSei9+CL8bX4i/j7+Bf4mcIEgRNgi0hgMAm5BB2E04Q2gi3CWOEGaIkUZtoTwwhJhE3E0uI9cRrxCfEDyQSSY1kQ1pO4pA2kUpIjaQe0gjpC1mKrEd2JUeTBeRd5CpyO/kh+QOFQtGiOFGiKBmUXZQaylXKM8pnMaqYoRhDjC22UaxMrFnsjthbcYK4priz+CrxXPFi8TPit8UnJAgSWhKuEkyJDRJlEuclhiSmJKmSJpIBkqmSOyVrJW9IvpLCSWlJuUuxpfKkjktdlRqloqjqVFcqi7qVeoJ6jTomjZXWlmZIJ0kXSp+W7pOelJGSMZcJk8mWKZO5JDNMQ9G0aAxaCm03rYk2SPsqqyTrLBsnu0O2XvaO7LTcEjknuTi5ArkGuftyX+Xp8u7yyfJ75VvknyqgFfQUlitkKRxRuKYwsUR6id0S1pKCJU1LHinCinqKQYprFY8r9ipOKSkreSrxlA4pXVWaUKYpOyknKe9Xvqw8rkJVcVDhqOxXuaLymi5Dd6an0EvoXfRJVUVVL1WB6jHVPtUZNW21ULUtag1qT9WJ6tbq8er71TvVJzVUNPw01mnUaTzSJGhaayZqHtTs1pzW0tYK19qm1aL1SltOm6Gdq12n/USHouOok6ZTqXNPF6trrZuse1i3Xw/Ws9BL1CvTu60P61vqc/QP6w8sxSy1WcpdWrl0yIBs4GyQaVBnMGJIM/Q13GLYYvjWSMMoymivUbfRD2ML4xTjE8aPTaRMvE22mLSZvDfVM2WZlpneM6OYeZhtNGs1e2eubx5nfsT8gQXVws9im0WnxXdLK0u+Zb3luJWGVYxVudWQtbR1oPVO6x4bjI2LzUabizZfbC1tM2ybbP+0M7BLtqu1e7VMe1ncshPLRu3V7Jn2x+yHHegOMQ5HHYYdVR2ZjpWOz53UndhOJ51eOus6Jzmfcn7rYuzCdznnMu1q67retd0N5ebpVuDW5y7lHupe6v7MQ80jwaPOY9LTwnOtZ7sXxsvHa6/XEEOJwWLUMCa9rbzXe3f5kH2CfUp9nvvq+fJ92/xgP2+/fX5P/DX9uf4tASCAEbAv4GmgdmBa4IXl2OWBy8uWvwgyCVoX1B1MDV4dXBv8KcQlZHfI41CdUEFoZ5h4WHRYTdh0uFt4UfhwhFHE+ohbkQqRnMjWKFxUWNTJqKkV7isOrBiLtojOjx5cqb0ye+WNVQqrUlZdWi2+mrn6TAwmJjymNuYbM4BZyZyKZcSWx06yXFkHWW/YTuz97PE4+7iiuJfx9vFF8a8S7BP2JYwnOiYWJ05wXDmlnHdJXkkVSdPJAclVycKU8JSGVHxqTOp5rhQ3mdu1RnlN9poBnj4vnzecZpt2IG2S78M/mQ6lr0xvzZBGzE2vQEfwk2Ak0yGzLPNzVljWmWzJbG52b45ezo6cl7keub+sRa9lre1cp7pu87qR9c7rj22ANsRu6NyovjFv49gmz03Vm4mbkzf/tsV4S9GWj1vDt7blKeVtyhv9yfOnunyxfH7+0Da7bRXb0ds52/t2mO04tONHAbvgZqFxYXHht52snTd/Nvm55Gfhrvhdfbstdx/Zg93D3TO413FvdZFkUW7R6D6/fc376fsL9n88sPrAjWLz4oqDxIOCg8MlviWthzQO7Tn0rTSx9H6ZS1lDuWL5jvLpw+zDd444HamvUKoorPh6lHP0wTHPY82VWpXFx7HHM4+/OBF2ovsX619qTiqcLDz5vYpbNVwdVN1VY1VTU6tYu7sOrhPUjZ+KPtV/2u10a71B/bEGWkNhI2gUNL7+NebXwSafps4z1mfqz2qeLT9HPVfQDDXnNE+2JLYMt0a2Dpz3Pt/ZZtd27oLhhaqLqhfLLslc2n2ZeDnvsvBK7pWpdl77REdCx2jn6s7HVyOu3uta3tV3zedaz3WP61e7nbuv9Nj3XLxhe+P8TeubLbcsbzX3WvSe+83it3N9ln3Nt61ut/bb9LcNLBu4fMfxTsddt7vX7zHu3brvf39gMHTwwVD00PAD9oNXD1MevnuU+Wjm8aYnmCcFTyWeFj9TfFb5u+7vDcOWw5dG3EZ6nwc/fzzKGn3zR/of38byXlBeFL9UeVnzyvTVxXGP8f7XK16PveG9mZnI/5vk38rf6rw9+6fTn72TEZNj7/jvhO93fpD/UPXR/GPnVODUs0+pn2amCz7Lf67+Yv2l+2v415czWd9w30q+635v++Hz44kwVSjkMfnMOSuAQhocHw/A+yoAKJEAUPsR/yA274nnApr38XME/hPP++a5sASgHulmrZBrOwCNSNNyQrSRftYShTgB2MxM1P4R6fFmpvNaZMRZYj4LhR+UAMAhfuY7XyicOSwUfj+BFPsQgPa0eS8+G1jkD6Ue1zS6T3ygzWgT+Jf4O0btBkBVU4mLAAAB1WlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyI+CiAgICAgICAgIDx0aWZmOkNvbXByZXNzaW9uPjE8L3RpZmY6Q29tcHJlc3Npb24+CiAgICAgICAgIDx0aWZmOk9yaWVudGF0aW9uPjE8L3RpZmY6T3JpZW50YXRpb24+CiAgICAgICAgIDx0aWZmOlBob3RvbWV0cmljSW50ZXJwcmV0YXRpb24+MjwvdGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KAtiABQAAO3BJREFUeAHtfQmcXUWZb93e0kuW7iSQEBLosAYQCFtocICOOqLyU4KOT984jonPeU9nVBpmngOIksjuqHQct9E3Ehxn0fccgqKyDJAQCE3Ymn0JkA4JCQlJujtJL+n1ff+699w+9/apc+qcU3WWe6vy69x7z6nlq6/q++r7vvrqqwwzqaQw0HJVWzOrYs3o1Ngoa2QZtjhUB8dZZ0Ul6+F1jLCujlvau0LVZwonCgOZREFjgPHEwDnXtC2urGCNY2OsdXyczRhnWQLPZFirZ2GFGajtdaiOJlAntd1bUcHWjY6xniduau9U2IypSjMGDAPQjOCg1bfednPz4N5dzSD0McZOp4FqprrCreZBgfFfrpMYU1cFY8+CMTAjOfjHYEQlDAOICNFezbRc24YVffHoOLuIBgWE3uxVJmXvu4gpdFZm2HqSGDo7bmhflzL4SxJcwwBiGlaI8oT8ZWMg+IjF95i6PKlZqBEVxBCIMaw1qsMk9ETywDCASNDMWOvKtsa+IbaMmruIFOdlbJwMdCZNYCBDhsZxtpYerG+oYWvXrWzvmXhpvunCgGEAujBL9cIiP17JlpFx7NJyXeWDohfSARk778qMsrVm5yEoFr3LGQbgjSNfOWwr/eVUcLGvwiazCAOd9GK1kQxE6An+3DCA4LgrKLnk6rZlYxn2OULosoIX5odSDMBeUDHO7th0c/tapRWXaWWGAYQYeIj4oxl2Oatgy41OHwKRQYrCZjDG1lSOs9VGRQiCwGwZwwAC4O68669eNjwwYFb7ALjTUcRIBcGxahiAD9zR1t1yyn4d/TXTn0nJw0AXgbSKthTXJA+0ZEJkGIDHuMCo1z/E2sYh6putOw9sJeQ1qQcZUg3qa1i72U50HxPDAAT4MYQvQEyaHhtG4DlahgEUocgQfhFCSuGnYQTCUTQMwIaaJde0rTSivg0hpfY1xwg23dS+stS6FrQ/hgEQ5oxxL+j0SW25LoLcGAsJCWXNAHACb2SUXWfcdFNLyKEAh7txVSVbVc4nE8uSAeTcdW+j2bM81AwyhUsFA2vIzfiKctwxKDsGwMX9DLvNbOmVCu0q6gfZB2hOXFFuPgRlwwDgtjuSYbeXk7hfUVnJqmprCyikZurUgt/FP4YOHix4NDI4SLEFRwuelfIPrhaMsxXl4l5cFgygVK37IO7KmhpWXVfHGMXeqq6nT0rVDQ1aaHS4r4/XO9w/QBFHxxi5Q7PRoSEGJlFSqYx2C0qaAZTKqm+t5Fi9K+uyRF+8ssdNgGACYAajA4MMUkQpSA7lIA2ULAM49+ttbRRM87o06vpY1WtoFcdKXkWretKIXZbZgAmMkLQAyWGI/sAgUpdIGqDgpqsev7G9PXWwSwBccgwAFv6DQ6Trp+xc/pTp0znBT5kxnYv1EmOXuixgAId693OGcGj//lTBjxOHU2vYilLbKSgpBoB9fYqqe2caVn2I9SD6GiJ4fJZjAhMYIoaAz1QYGkkaoKjGl5WS30DJMABu6Mse1U0sLRmiFw9NmpgBEc2qUnEnTj0DSIPIjxUef7Uzm8QUYN7kMTC4r5tLBUlWE0pFJUg1A0BsfZo1t9MfPhOVYMira2piU5oaS1an141wbjPo7mED3d1JNSB2Eg5WpPlOg9QyAAThHK8g4k9YkA5Y7OtnzzarvWLuAKmgf8+e5PkcwGdgjK1Ia5DSVDIAvsU3Tu68CUrYtmuYO0ebE06CuhorKNhS7HtnF99WjBWQosbphqMr0rhVmDoGQGI/RP7lRfiP7SfE/Po5hxsxP+IRgHrQv2s3Vw8ibtqtuTWkDqxwy5C0d6lhAEkz9mHFn7ZgviH8mGc0GMGBbdsTIxGkzTiYCgaQO777EM212I19RtSPmeIFzSdMNeik48VL0+A0lHgGkBTih1V/6rwjytZpR0B3iXuMrcODO3YmYdcgFUwg0Qwgt813J82y5rhmGpx3YNWHnm9SejAA+wB2DWL2MEz8NmFiGQAn/gx7KM5tPjjvYNXH6m9S+jAA+wCkgVgdimibkObw0qT6CiSSAcRN/Fj1YeBLm49+hkI8khGKp+KBtZ7zl9YPZKLvFCSloBwdg81Gi7Tn4wUnB5G0ZxnP15LLnJAPMAAYCmOTBhLMBIrnSexDFjfxQ9xvIHE/Q0wgmQlkniPYHKHy3xYlxgx0npnYYeMwxQvgOEU16supBbGgKKFMIFEMIE7iT+Kqz1f0HN0UEFYsMzhkozTT7Iwq25/omUKs0kACmUBiGEDO2v8MTbPmkFPNd3GI+tNJ5I9z1c+L70QTnDiipw3feFNRIN9Xi0FEoEZAGthPKkFMtoFE7Q4kggHEudU39YgjWP1hs1XM5QB1EPpB6NYolAnRCxFlxwP/rhch/e/uYQd37hSCo/FFYpiAhXKNfXWvOi7ih2V/RvPRkYbbymrvWXxwQ5s7asxbwgAkBCvpMDIibFlv19Y4/AYSwQQo3Fm8CeG7CIJIPfwg8s88/rhIiB9Eb/0DpkH4hvjl55wdXxYe8akq4fQm5kIMOz6Lc3NfVVcC1ROrqZuMfojd9+lAkAcsNHXOHDZt/pEsQ2G09SZM0uxUNQSvFtNZqSCLXxU1Yy7UNjby0ULw0qgS9WDRkRe0NO/Y0HFXVG0WtxMbA8CRXloMryoGSNdvbuU/8khWp1Hft1YortTrVV91oSl99RI3sP6FBb56agOrrK7hQUvHo+Paixdc2NL79oaOjrDwBykPNhp54sE8MhS8M6IE4m889hhtIj8mIJ8vwKYh/IhGtaiZHO75rkLIQYBdoOeNNyN1HMqMs8viCCoSOQOIeq8fOh6MfardeS2it4xU0S0YRRPf/CzAgH08wjADuBHDOAhmEEmKyUdAtyJcgDtY/OlBZGG8QPxNtPJrIf5cz0D4hvgLhjnWH/bxgDAGRh0kYc5g7mAORZKyoe1uz9FIJE2ikUgZQJQWf5zbxwCqcu7hE2mcJhP9cYI3on5kkzRwQxZzzo2bX2aAuYM5hLkUUYp8ZyAyIyDi9hMSvxgFIhGmC2K/Ckt/nvCjANy0EQEGaEQtPUGiNb5DQOHcx4aGI1EHaIlZNP+ClgwZBddJgBc6SzD5yGez/MaeMTraG0EC8eMkn5qUXTOMiK8Gm4mphTMAiHD+xDicKESI8ihSZQVbGsUNRNoZAPf0G2ZbCNfQ/7UmdcQPUV8rqKbypGCAU4D8YEfGBMgo2FDNFuoOK6bdBsD1/pQQf17cl58PSZnGBo6gGMBYk40gK+t5VwLpEguN9kQ0E4WnoFYbAJx9CFH405pgpIHOHyphEmiXh0JBaAprxEBWK6AJIDEHcIPz8EG67nx4WCNEHJRFup2EJLobrI8tV7U1j1ayZ3SL/tZWX1BrPzi/0fGDjXGplgIz8Dp4hCPF3eQspN1PgFSBylF2Rsct7V068K1NBRjJ6N/vN8SvY0qYOrEgeKkE1hahdj8BUgU4LWkaFi0qAN/yy7DlmmDm1VruvRXV1b6b4YOLvWGTDAZcMQC1UDxPsEVYM20aO0QXmOo8O0AgNOvaGhT3zhUx4pdRiP4W8QfjviB/iP3G0iceRfPGwgAYQFYdEM+XSM4OaFIFlKsAUYj+iOITiPi5Rxjpd4b4rfltPj0wwOcKaN9FYsRcxJzUmjSpAkpVADros5wYplarP87z+z3SmxVzlAs7WsfbVJ5ADNDkFs2iqro6LlnqjCdAzTdT/ICtFD+gUxV2RP3xXX8UDj+I2uJ3u88ifrPo+x5SU8AJA5wJjAn9xHCCUGuwUcUOQspUgL4hdpvOLT+czkLkXn8J5G+2+fzhzOR2xQBfSbLzyikf5qjq06cF7ZAqwGmt4GHwH0oYAHz9CYTlwcHwLskP9/i5rCOns5mV3xu3Joc/DOTnlINdANuDfqVUf63z3MtzNBegaGERJQxgZJRdV1it2l/+jX45zUZsuFULoKmt/DBgzS0HJhCFUVAVzYU2Auo2/EHvn3bkPPkJRgNCappJBgORYIDPNTCBojlX3VDPRgYG2eihQ1rgoHaVGARVSADaVn/s9/vR+y0bbV5E04J6U6nBwAQGrLlmzb2JN4zPXcxhjSk07YWCLhfkY5muDk4/+iiG7RW5RCzYEsvkCphcBgNqMVAkesJTsLJ2CjvU06u2nYnaGsN6CBYJLhM1e33Tve2HW3qnzpN0rnDQw7zgN+8NBrRhgEL82tPBHTtZ/5499kfqvofcFgysAvQPkcMPbUmo68lETRCbcEW3VDI6vxSaTKZoMJC3Cdiaw1zWpgoQDXJatLXn52sgCUD36o9tFJmrmqB3cV5byHD99N/kNRhQjwGiKhCW/UgxnIPgJKQlhZACAkkAOld/EL4M8XMHH2DTEL+WOWUqDYEBmpPZaTmxvsrP6wDthpACJiCUbFfn6g8xqYkuavT2pCKwDeFLjpjJFisGOIVlJysuG+ne/LqeG4cCSgG+JQCdqz8Mf17ED7HfN9eKdQaYxssZA5ir1ozF3MYc15ICSgG+GQDZ3C7X0QGOHAnDH/Zdrb1XHXCYOg0GVGKgeL7W0xz3WuSCth+ENn0xAHj9keitxfIvt+Vn1v6gk8OUixkDtq1qubkeAF6iTU6jPor6YgBU73U+6pbOiqi+3oY/Q/zSCDUZk4cBPn2zcxhzXeN1Y75oVJoB4EpvwmqzDsw2zJ3jWm1W7yfkZW0prnnNS4OBRGKA5m5+HhOAXnM+RB+az7v+amnvXGkGMJZhnwsBlLAoOGG1xOWLJoyXEIXmRUowYJ/DmPO6pIDhgQFpWs3KJB4I5IE+K+h6Lw1p1qITPYwiZuXXgHZTZZwY4FQ3zrAtuPeVV7VAUjnGFsrcJSAlAYxqsvzjiiUvi6gUh9KCQlOpwYAeDFhzGnNf1zVjsjQrxQBYhZ5oP9gScU1kOTVbfq4YMi9TiAE+p3O7Ap40ELR/kjTryQC48U/D1h/0H7fVnxtMLFYZFAmmnMFAQjGAQ0OY46ABLbYAotmc4d4VA54MQJfxz8sKWuxA4doL89JgIGUYsM9vL1oI2jUZ2nVlAPD7J0YlvaUgCyhiprlb/s3SL4tLky/tGMhwWgh00Y1H10G7oGG3bK4MgMIPKyd+AKPNH9qtp+adwUCCMaCLJrxo2JUBEL6U+/1D56md2SQciqyzhPC1eWEwUGIYyM540ISbTSxEp11pWMgAsPdPjS4O0bBjUZltD2P5d0SdeViKGLBNdhnaCICCxTladiwqZADjlXrE/ylNYpUEvNAkg4FyxADmvhtthMGJGy0LGcDoGLs0TKNOZXEIwk3MsVtGncqbZwYDpYgBa96DNrwPxfnHgBstOzIAbv3PsFb/TbmX0NE59xbNW4OBdGFAB42Qz0GraDfAkQF4WQ6DoBThvryMf0HqNWUMBkoFA1ADQCM6IgiLaNqRARBCL1KNVC/OZrOFqG7a1GcwkAoMWDTgRSsBO+NI084MIKPeAFgzY7oQbmP8E6LGvCgjDFjuwW60EhgdApqexAAopNBi1WG/INK4cbXxwL0yBQ0GSgcD3BhI3QGtKFcDsuHCJm3rT2IAOlx/3YifD5/hAKUzi01PwmEgRwueNBOgFSfansQAxsbV6/+eIo3Z/g8wnKZISWIgRwueNBOg8060PYkBYMsgQN2uRdy4Ge+vkQBc8WdelhEGiBZAE240ExQbTrRdwABarm1rDVq5qJxXR8Zt4ZJFdZjnBgPlhAGLJrxoJwhOimm8gAGQEWKSkSBII/Yy7sd+KacR/+3oMt8NBvI04Uk7AXBVTOMFDGBUg/4/xWX7j8NvxP8Aw2iKlDQGLEOgF+0EQEIxjVfZ66DFWKkEAN9mN99/vv9PjVoOEHZYZL6fdtRCdtGRRzlm/dWrL7Dd+/Y6vlPxMEMK1Z+ddhabW9/gWt3bBw+wO194OnAfXSuP4OUHFr2HndQ0S3lLozTo+4cOsZ5Dg/R3iHUPDrDNe3axocFB5W2lrUL4A0AMsOgH0YNVpWIazzMAHvqbsWZVDaEemVhnQYkf9R83o4ldfPQx+Dop3f/WFrab6WEAIP6vnHcR+yAxILf01oH97J+ffTK1xI++nXnYHHahgMm69T3IOyx8L+/bwzbufJs9tPUN1ru/N0g1qS8DmsgygSwNDShkAIScZpwLWLeyvQeImlABqtQSPyr30mHCED/qjyPJEv+W/T3sygf/wPr7++IAM5Vt0urETp45m33hlNPZLz58GfuLM89l1SRFlmOyaMOLhoLgZnBkQtLPM4CxMdYapDK3MlX1dW6v88YO90zJeStL/Jt7utnfPvhHNjgwkBzgUwZJJS2Bnz7hZHb7JZ9gx8ydlzLoFYALbkjJk4ay2Xz9b6f1CQbA2Om+avHIDFdGHYEOPZrV9lqW+F8mu8PXHvqj0WUVjUTjlFr27Qvez+aTKlKOCTSk2i14zEbreQZADKdZJYKliD8lOwCyxP/83nfZ1Q/dw4bJqGWSOgzUVlax71z4ATa7aaa6SpNek402pGjJR3/stJ43AlJ5pTsANVOneoIEQ4el63hmjimDLPE/vXsXW/XwfWx0ZCQmSKNv9qpHH2K7Du730XCG1VVXs4bqGv43Y8oUdsqsw9jZh89lM2vd1cWpVOarZ7Swb5JdpRwSp41cR0FLQ31KbUl5WucMgJ8AVIzVyrpazxpLhfg3vbOD3fjIA2VF/Bjct3t7WDcZO8OkB3KF55Dx78tnLGFnuIj6Zx4+hx1FzOKt3e+EaTIVZTlt0AKJJENL2Zzy/4Pmn7ipvZOrAJUVrFG+qFxOt/1/uRrizSW78j+6Yzu7YcN/lR3xqx6dXbT9940H/sB+8vwzrlUvPyW/eLnmK6WXOmjJonnOAOxWQVWIU623qIJLph5Z4l+3/S12C638Y6OjMtWaPBIYuPvFTvbbNzcLcy6ZcwSr93C+EhZO6QsdtGTRPGcAJG7MUIkbWYAtZweVbYetS5b47yNHo+9ufJCNEyZNUouBXzz/FIOnoCjNmaZ0uoqaifV5MW3I0pQs0BbNZxmAYgOgDpFFtmNh8skS/x+63mD/+Nh6In7xJA0DR7mXhf8EVCtRmj9NHF5OVCbtz1XTFM1crktxBqAaOdV17hZdqz0XJm9liexTlvjXvvEa+/HjD9PuhSF+nYPzRm+3sPoFZcAAiqeXLE0JkSZ4wXcBSNxoFbwP9rhCC18JBotEKVni//Xml9kvnu6gvUuJSk2WUBjY7eJCPYX8AlQmjP8C2n04ng49YTsSf7PIAWeY1Lt9dEhpHx1Q2tF3gD33zttsoL9fZdPydSmmKYvm1WIy151qLxdg+W5rzylL/L985QX2H51PKIPn1AXNbCn9FacRmnQ/6ni4+LHv34uPOoZdOH/ySckdfQfZ/6MDSklPbjx26wE1h4ROOGI++wAdJnvvvPlsRs0UKZRAMnn47W3sd6+9GKm3py6aqsqdApTqvOpMxIVidQSSJf6fv/gs+8/nn1ba/WPoJKPTacIh2lFQwQCObXSu/9XuvcQAlHZFS2WH1dUL630rpO/BTML9V+ig0Tm0o+A3HUtl8YdzCv/x2kvst688z4bVntbLggTi0CxqgvarGE4BKjZky55gKtZz/A5GmPyyxP8TIvy7iQGYFC0GFs4Qu6bsDHpMmGjqk6edzT5LMQ4qOIEF71NdVRVbcfJp7KMLj2PXkEfkjj27g1fmVBLEAR6QS7I0ZeWX+iTaj1dZt3VQCmBFmWSJfzWJ/Ib4FSHdRzU4AnwBieVOqZeCiPT1H3R65foMY/6lJRewz510amjitzc0mySVH77vw+z8Y0+0Pw7/PSLaqBgbVesFqPrkUnhMFtYgQ/zQP7/z9OPsftL7TYoeA5eedBqrrqh0bPhnL3T6lowzZEC78r1L2SW0WnulMVp59wz0s1e795Hh7yAZAr2dvKqp/mvOOZ8hepLOpJq2QPtVJGYo9a305bDgZunRgEkZ4scEuOXJx9jGN17VAIGp0gsDp8w/mq/STvneoLMH62knxm/6bxS6bSnVK0oITXbv1jfZfeTf8Q65JBds8dJKXEu7AkvmN7OPHXM8W+QSHu2rp5/NtlAsiDdotyB0Am0USQGgLaWHgoj2tewCyHY+SiOgLPFfv+kR9sSW12W7YPIpwkAFOad/8tSz2GdOPKV43udbaKct2ALizL8Rf0EcAdQpSht3bmffeXyD2KJPhAjHpIeJ8eDvJAqPdl3LBQynE4sT7Ao3nN/K/ud9v2UHKBZkmMRpI0wFkmVjZQCAMQomIEP8gGU3iX5Pb+vCV5M0YwBi+RQ6DnzCbIo5SKsztuKmORCVBcZPX3iGbdm10/op9VlJhrpvnHuBUOf/1Wsvs395xp9fx8tvv0UE/jt264V/ypwckqaR/eLzJHGs3rhOCkanTKCJqFKsDGAc2xyaLwaRJX4gHBF+v3DWeeyfaEUwyRsDN/zJ+9iQz7MQ9USUiAPgtII6tQgd/B+eejyQSvb+405iR06d5lQtQ/yGX3Y+7tuegMr2U7DXb2x8iP3sTz9KtorJdvT3k3/H7eStiHxBEt8dy0AH0J+UMwCZQCAF3QK309RXP8RvwYRtncdILHyODvuY5I6Bo6frPZTTQbr095/qCERIWEU/efwixw7AznPr4+HOcuwhIyGkkr+h1b44QRX4NBkyf0rqZKAkkAA0BAaxRQUOBKmCQpqIH0ElZUJ3O/XgarLoltuRUyc8xPlscHSEdVCglT5Sy4KkRfOOYkc0OEeleoQOGvUpiLBzD+0SHRx2jtn/IfIwhF0jUNJEE06wBITQqapgzzKaRJ2vLj7H0dNOBkroolefd1E+NrtMGZNHLQYQB7CNxvCXH/sU+wStstWkNvhJy1z25de+/oqfqoR5cRQcbsFOqYaC4s6fdbjTK89numjCqeEKYjZHO72I6hlndgKRJwwMIt0PdWLb5/rHH3Hd40VoqktOPj0MCKasAgzAqAaPu599aBlrlnTdhfi/WBBabCft7W9+Z7sCyLJVPLhti7Cu0w+bK3wnfEGwc5oQZlD7oqq6gU499cV8HVOEPe6mk11t6+5le3v2sR/SqoJVRpT+6j1nsKdIDN1J0X5NmowBEJNfIyBqmUKGs+mE+/qq6smVCp7A4+77rReT3t1J3pnkDOSS4OvfQMFHnRKOc3Mjm9PLAM9epVuM+oaHHds7g+IX/u5Fn5WCFjQsiCIoqkb6BreKXkb2HB2OgAm8S/pkG4Xttq6c+i+6P/CCeQvYWTRQTgl2hJWkCvz1vXexURpkkwox8LX194cKCpqpyNDNP1PYEdMb2Zlz5rEldAHIKTNnC7ftYFz74qln8HsEH3Vx1FrkInpvIgOvygQ1YDvt+Z/oELL86CBxCyIkfuAhdhsAgECfIbbpTFitvkpBJy3i520R07mVjt6KDDnIA1Xif9HWoEnqMYCISrgMdCtF+b2TDl1dff/dbPk9d5Hxz92T7n+f1cIOnym+sPRkCjUuSr0BzhGI6rKedx9yvgFquuQRY6se0IBmMrCayn8mggHAH0ClWJbvXe4LLum8nIjfyTsLd/f9A201uaWPNB/LziCrrkn6MbCPztvfsO4+7o4taq2KVIgvnHqm6DVrotuEnNIhOmo9pOHSFgQMcUo4MQiHJ9kEGuC+MbIFFOSTh06ysaGD/k9qoWpdEgACOFxBxO92SedT5AOO24Td0t+ffR5rEGwruZUz74Jh4BGy1P/AJXDJ+RTMQyQFTBdcKLqLmL2OtJeiBokSvB1lkxcNBKUtt/aVMwC3xtzegfN5IcCtvNO7Vyj4xd/RJZ2HXAbIKvejJzcy2AhECZ5r17RgazBqIU0EkfvztMDp1ot7Xn6e4ZZlUTqP9vqd0jSB6K2LARxwCQhSI2noxLSKevUH7hLDAACMSjXgOQrQcBURv+w9fYjqcgNtDbrZIk8/7HD2Ubq6Og2pThg3Lx0MzMLxv7/6kvV10ucSMhw6paglALg2i9IhgaNQcX6Vc7+4brffiWIA4IKqFthf0Oox4tNyj2Ocv6YDIm7pC3QzzbzZwRw83OpV/Q7756WQOl2s9ic4WN7d+gznHB3JLXzZkIt0YMGict5bdcp+VtCS576pKltTLt+IwCAiU00cIlAxXP/67BOsyyXkFLaisDVYJdhnLq4vrt+lwgAGBvoYAqU6JRjZnIJkwNHLKeGwl450uCB+IYyOshfHyMz9MLTl2G+i/QoKvCJWshxLuT9Uck2WKjHAHVTHt4D/Wx3rhZMOheaRMfCLZ5/vWD4pD6dXi8XSpMAoAwdEY5GVHeWdjGz7Bavu3HrnswEycLjlEUkAIkZUUJePua6EtmyNg/YTpQIAtiwndNPEbT3Q9HX3vr3c48ytehz2OIu2B5Oa3PTSpMLsBBfoo6nWmZlhljiJ2AcEEsBsurDGz7acEzxOz0QM4E2KYOSdMOPjm+8VbIR1eQPpL8ewgpNW/lpUn/uPLz/HYEh0S18jB6GpgvPmbuV0v4N64hSsQne7OurHqUxRfEBY9UdHRiY1u1Mw/6C+NQjE9UmVSD5ooujFItvCiwpdyLXQFNF+Rcct7V2SfY0wW3wc0eokQk/dTF6C/SNiF2D4m3+d7AFwaVWVKuE4ErK648i1GW7MpZDmTmsUdmOLYIV9Ya+YcS9onCmsL8iLpXQBiyi9RPEFvVN8cx20r0UFGO4XO0Z4IySbA/M37jkMz8HvPb3JFeRTye300lPOcM3j9LJPwFhAuNUuobGc6ip+dtbhzttjxfnS8PvjgqAegF3kI/Dqu7uEXfv4cc5BQoQFPF5cLPAQBVlvcZEA/M5vFTTl1BXOAGixW+f0MvAzgdXWT31cM4qPOeZB7XjzNbb+7bfyv52+fJ6OqyL4pJ/U7eKc1EBRaAMnYpznC2LqB64zpoKzaZvvQgrCKUqvUVQepwSvz92CO/xajjiSNdHhIxVp7qzZwpBjT1H8QjcHNBg3fen+CmjK3meL5rVIAMpcFilYSNxSAJC2+olH+SWRdgTav/OtwZYLaWtQfu/djQGc4HKazd6u0/dzm49ngU6hOVUW4zNs711x5nlCbegdOtz19LY3hRDeLzinD8Xo4yeeLCzn58X7jxIbgX/75mZhVXxO+wyEo4ymiqDiDICQ0ln0PNRPldsVCRAC+Im1Gzc96oqTubQ1+NcUSkw29VCoaVF6HwWVDJIQBfdLp4kPyQSpM44yIP5rL/gAg+elKP2YTg+OjTr7B6DM7+jePsT+c0ofpt2bGhfvPacyxc9qaUfhkoXODACuwZ3bu4qL5H87Q5V/7fhFJU2hAYvmswwgw9Rct5oDXa3DQjKkgFd3bGN3upxBR9dx2eeShcfnsOD+0UvXTeMiUKd0HompEC/9JMSfu4LOKiBwRpoTbk3+8Ycv47EBRP3ABadPbX1D9Jo/P0gSwgYaM6eEcGN/e+6Foc51XH7Oe5nouO//pfsDRASblWj9swC1NMXta5zmOQMgw/M6J0SFeaYeYAhv8aY1z2xi2zxCPf/dWeeyaRJbg9i+ekTg5gpD4DeJmOvq5YiZx7+/8IOs1eE68HgxJm4dhIA4fzOmzWC4pvszdFvvDz/ycXYzXeHlFs4NNf4A17RL0NAdFDloVCAF4B6Cz5Mzl98dHByy+tTic3ggGafeIe7E2pecL5MNekBLNS0BbovmeVhwkqRkPBac+it8NkpikK9rwoQ12Ywl4AESA+9SVahXINpv0dbgP33gEmHUGoS5uva8VnbVA3eTG6g7sPfSMeT3Ca6sOoqiyfzs4ktZOzEdXFbitKJMIWPhh48/if3ZcSeyxqIz8JiIoqi4oZBgK3z7hy61/fL3FXYT/PlN/0hHhGUvCIFDF+4SRBQhp3TZsSewkymwyC0UvvtdgUHRXg6M/Upa+d2uFf8u3V7kNFaQubNGP/c5YW/P+g5aUp0smucM4Imb2jvPuaZNaRujAxQkYfp0hXUCcf4njEIAeFWID/jPdF34X71nsbDqU0h8//h7zmS/ee4pYR68eHnnNgpvNUiebs4BLEDUK8m42EcOR9hT3kM7B8OkNsDecAQ5yIDAERyjOOFyy+seW89+SoxKZ3JqW2d77c88wRDGzU/6Pa3GWO2xXeuUTqS7/m4nRgv8PkiMdjMxgp7BftZPgUPq6FhxE6lU86ZNI/XuGB5o1G0G/oEY+iuCKMHZtv0TP8pxWnICPsQz0DyKcwaQqwcPxLM6l0n2A1bL+jliI45sPYX5ksEEfvtSJ99qQ/w6UcI11E/S6UKEuxIlGLFWkkTR3vpBV9YGhyO3VcdeP+Lp//2GB1jfIecoNfa8afmOPq0m4t8QIJw3HLq+9ciD7NtLL2YLXbb/TqaxxF/Q9ND2rezHmza4FA9G/KhQww4AJ37UnV8+CLwuPFCVdOgtgA1SYwDJUVW3eD0Q7W967GGGiSlKEG+/Sas37rp3SziCfDtJFCoSptiqjg1sF61mQfVNFXCoqgNW/LvpotbP3v2bQMRvwTFAEtGVFBXqRSnPPKuU/CcuGvnexnWOKp+K+aqaluy0nmcA9EXNLMzhDXqQasBRNXcQyrUR50fvgV62GsYolzSHxPQvk87olf7zhafZ90jXF+0KeJXHe0Q/+vKD97DnSYxFqnBQDfiLFPyHyEzYR/8fdAnnT+ieRhBw2ITAMNdQgBgwlOBrcSEUcBPHHLj1kQeEx37Rli+Hn8ImOA052hSK8vn5aaf1vAoAqyBJpNf5qcgr7wi5BKsyBBa2RWhVwVoLK/X9a8PmV8gafBQ7n7btRAkXRT52zAkMHoXCRN158NUX2TMkDXzlzBZ2NvnyyxrI3iGvtx+TrYFvi2G25ZJseSt/1J+49LOHiLKbVBV89tDny7RCP0Vx9hEYVEeCERcM5U7apvvqGee6+hm4tQ/J5J6tb7KfU+wIXB3umPLGAtugOGZ0fwgaUp2sHQDUmwez9babm/ve3bVFZWN1TU1s2oL5KqssqCtD4HP0hsNxQZ1J+DGFjILnzG9mLXOPZIfRVuAsMgbOJIv/yPgY27p/P9tK0geClmyleHkv0l636hUiCTiIAoY5pPO3zFtA7sYL2AlkDMwTg0PjByi012PEnB4lfD9H27euoeagplIdYVZ+C4QD27azgW61DLGhhjWtW9negzYK+kw7AWAAzXihIlWS/jtr0YkqqhLXQdeLQxgQbPeKy5k3BgM2DFSSoXVqbT2bTsy3MXcWo4d2XQ6QZAKD6jBiDEgsNPm56NPV1wZKwde9r7zKFG8DdtEOwEKrkbwKgAfUv06ipWbrZdhPAI4/MAJtiRA9TkzAJIOBMBjAzU+9w70Mth1n/0G52vlCpIj4LfqRa1kuF2jcnjNvBMTDygxbb3+p4vuh3v0qqnGvAwg3PMAdR+atfgxgDioifgCrg3aKabyAAZD4UsAdVGBMSyQTR8CScWbAETTzsOQxANFfSkfwgQkdtFNM4wUMoOOG9nU+4JPKeoiMVlEly+iSHYyoWjXtlDMGrLlmzT2VuNBBO8U0XsAAADzpMOtUdgJ16eiIEEZuExC+NS8MBpRiQKXObwdMB8040fYkBkDh7ZTbAYaisAPYsWfpYcYuYMeK+a4SA9bcsuaayrqpLh0040TbkxgAmdPWKu5LtBKABXxuYCwRzXpsPg0GwmIgP6c0ET/g0yIBOND2JAbATwll1B4PhqOKjg55DySxM2xu5kfMu4TJYTDgigE+l3LzyjVj8JegFeXOXUTT1glAO2STGAB/Oa5eCtAh0tg7IvqOocowCh1liWyijOa5wYAXBmgOYS5hTulMWmhFQNPODICptwPEIwFkhyk7YLqHTeeUMHUnAwM6bP2Te6aJVtZPbsl2HNj+knyFldsBINIM7lPr02yHWeo7dDZwcaMSSKHLZMrNFUiPGvV9O55BI8rFf2pARNOOEgAOCjhtGdgBDfJdE2fzCQokgSwj8FnQZC83DIDwucAfnfSog0ZAy9bhn+IhdGQAyERBZu8qzhz2Nzqn+GBDIJCygpzxHAyEvDIplBUSMVOiI37Qhg4G4EbLQgaQGVWvBmDuHOruScwUwuAabSAxw5EYQDAnoiR8q+O6aMONloUMIHdpqPKzAf17ZC5MtFCi/5MPtPEZ0I/ohLeQXwhoLsRB/ECPJtrodLsAWMgAcuO1WvW4JcIY6NSpiIw8Tk2bZwnBQIxzQJfxjzDrSsOuDEBkOQw7XJo4XViwspwfk4Abf0JXZypIAwZyFv64Vn0LRbpowouGXRkA3w1wcB+0gA76iWChOo46BoVncjkwgSwjyIuGkzOZJynFAB/THOFnrfzxdgS0oCeALlsrsv5bPXZlAMhUMc7usDKr/Ox7R3yHu8p2wtWVswAbiSAcGpNUOj+WubFNAGy6aEGGdj0ZwKab29eSSKzcdD+kieupHs+saGh2C1TjNY76stJcfEY+pz5j5QctKE9Es5x2PSr2ZAC8/Bhb41FPoNcD7yZrR8CtE5wRcLUgxwzyK4lbKfMuVgzQGGXF/axKF7ee74QLbTQgSbNSDKBy3N2S6NQxmWcId6xD95FpO0weTCSL/jHBsitLmBpNWVUYsI8HxiiJRG/1FXNfdchvq25ZmpViANhHJB6q/HwAgD349g4L5lR9WhJBwQSzuEKqelIiwNpw7zg2CeymrrkPWnXb+7ejQooBoEB1Xd0d9oKqvkP/SfaOgHdPrQmHOZgVOb3LmByKMEBI56s+qiMVrYAhK2pCRzWY81p0fwK2qkJeYpdmAI9942ZIAF06kKHLCqoDVrc6s9MP1mXonPSRm5xuZcw7/xjIM1ngmJN8egjf6q3GOd9VHPjTatPpU5oB5Aqvcqok7DNwQh2HIMLCFa58jhFQJZZeyiduuErLtvRkHE7gN21IwVzXtfoTLnzRKOehfhB4ztfbuonpNvopI5M3kmvEZADRmAd3GSLxSLIa2ym1qi3GmRbx3gv/Gq77yjaJsF83tjd5tW9/71cCgJrl6ltsr9zPdxyF7N+120+R1OXNqwi57cSJrQTqim9WnLruewMMHFh44N+hSllYw4qf/oQ5rutIfBDa9M0A6mtYOw2ScscgDC38oXUhJ5lTJzvBocfyeZ+b/NaKl0yY1UKV7yv1Pdv9CZyobSn+2vgip+s0LNEkp02f3az0mZ91resYnH9BSx2Va/Vb1iv/OMnGY0PDrLZRuYbh1XRy3ueYAKjBHrqME4r1LjnQykPC+zORPduf3Kqe5n5NdMnzG6761uX3Qii89dHr2+/xBKIog28JAOV1SgEwkJSeQbAI65I/uc6bUxey+i9WRzAGqsAiGutTss5Islkw0ecErFlRPnvIKvc9EmCS0YjWeR1w9QdmAjEAnDAKom/IDgU45TgFETXJCQMW8dhEZRoMEJpFbNZ3/puqsH5zpmERJ3+e/WGVyzMX2zsOAbLl6rfqsn/a33EC59ugWN1tsPKKyvM/zGXMaV0JtOh16k/UdiAGgMp0SgEIGrJfI8JEyEjz8yypTRBd/jfNDusfCJL/0bOscQ33JWTfWu/wy/4uT9D5WqzaJj4tQudl04xETbBjLuuI9MvBDbH6o3xgBqBbCtAqMmkaaFOtwUAxBnTP4zCrP2C1CYTFoMv9Pueati2Us1kut79cFZWVbNaiE1mGPk0yGEgbBiD6Y89f2+pPnrl03dfCMHgJLAHYGl1l+670q1EFlKLTVBYxBrSK/tm+hKa90Evrjg0dnfP+pKWVjELNOvA7eugQy1RUsuqGeh3VmzoNBrRgoJ9iXQzs3aulblSKyz6evLn9irANqJAAWFWlP/9jv0Af3LlT2/6pX1hMfoMBLwxgrx9zVmdSRXNKGEDu9NEanR3u7dpqtgZ1ItjUrQQD0PsxVzWnNX5O/LnBooQBoAEKP3wFmRS1uAijfrhRRoBYNGWSwUBgDGCOanVnJxrjtBYYwsKCoW0AVnVwET7yghaE+l1mPVP9OTo8TFLAGKuZNk111aY+g4HQGDi4Yycb7O0NXY9HBV/aeH17h0ce6dcZ6ZySGc++uu0hMgi2SmYPlG36/PmsdqavU4+B2jGFDAZkMYCbffZv1+ftBzhyhr+lsjDJ5KuSyeQnT9U4WzFawZ4hxzFtJ3qA6Kr6OlZVW+sHtLLJaw+xhq3U0YHBQH2vqK5mlVNq8mWrGxry382XCQzA6Keb+KFeV42xFROtqvmmXAIAWEuuaVtJDqXXqQHRuRY4CTUee0zZMQHol2OkCg33DzA2NsaGBwa4cRTqkVbd0zYMCN5SScwBDloUK5L8SStYNTFkzjDoXTklEH/PG2/qdPbh6CRCXbXppvaVqnGrhQEAyChUAUgATcQEStVTECs5CH2UJhmIW2MYKaXzqoYkBc4kaHzAGEpVcoDFv5uIX9cRX2tQdIj+Vt3KVYB8xRGoAkA8BqAUmAD6MkLEzomeVnXdk8oaJx2fnFER87InMGtIC2AGpaC+RUX8ukR/a2y0SQBo4Nyvt7WNjbPbrMZ0faZREsCKfqh3f5bgiVg0+ovrQnuoeqHCgRngb8qM6VxiCFVhhIUjI37qU0WGXfH4je3turqnlQEA6LOvabuTGtG2NWghpq6piU1bMN/6mchPnAwbIqLHChmVvp5IRDgABZUBqkMNMYMp06c75EjOI5zt13Wjj72XZEdb++RN7ZfZn6n+rk0FsACdWsNW9A3TtqDGXQG0ZQ1IkpgAVgqs8rqPhFq4TvMnGOIA/ui6OCQwAf5HDCFJNp6oiB+i/9Rq9Vb/4jmiXQJAgy3XtrWS/85DxY3r+B23JGCIXv2oJoUZREb8hMLKCrZUlbuv24go8wR0a2T7wx1dFEgUzKbVLZ+KdzCeIbBozdQGOkWozNPZEzQY7xDy+QDddQhvMJxiNEkNBoBLSFGD+/ZxvFbQSRioDFElMHXc42dJJ7rbJUJZRXr/Gt3toP5IJACrI1HZA9BeFIZBTIwB8gDDsU+j01ujHM0nGEDdrFmsjjxCdaoIGOMotvosrEWh91tt4VO7DcDeGLcHDLFmerbY/lzHd51bhFjt4foZ1YqgAz9prxMMF0du8Qe1D67hqv0NoiZ+GpNO0EiUYxOpBICOUQixxSR3PKTbKGghEZLA9KMWKPEYBNEPkpEqLQ45Fg7K5RO7CLU5ZhC2z1hA9r+1LTp/DJykHWdLKcRXZ1jY/ZSPnAEAuCVXty0bz7A7/QAaJm9Yt2EQft9ufVc6hembKTsZA1APGg4/PPCBMRB/FO69dsgpuOdlm25uX2t/FsX3SIyAxR15+5GOVxZc2NJL+s6Hit/p+I0bhwb27iP/9RpWBd91icT1ewrrhEMeWPXx26R0YABjxY2G3T0USGKMxrxW2iAMZt+zpYtO3tHsjCjB2Yf8/NdE1FxBM7FIABYEpA7cTt+XW7+j+KyfPZtNnXeEa1OI59ZPK37avPOmHnEkO5r+mptmTupf58632V76G+zeN+ldqT+ABFhPEkH9YbNdu4rz/LifMuK0hsT+SPV+e/9iZQAAJMqdAavj0BVnNB89yXqcRlEfRP+X51/Ilp18Kmus9ZZuuogBrNvyOvu3p5+gle51CyX5z8aFx7H7vvA3+d9hviz5euiYlWGan1RWpBpAYkAkn6htO1Fb/CchhB5EugvgBECUOwNW+xjofZtf50wARkJY9RHCOU1bebW0yl/5kWWc8K1+yXxCOljetIQtP3MJZwTf/M2/l41UgPGFSgd7znRyG8euAfR97WG8nAcmcou/ExixMwDcMNS6sm1p3xD3FNS+PWghAZNh32ub+e4AJkGaElbpX//F56VWfLd+LSbpoRxVAow99vbB/GMa+06K67c06H1+bmPq9110rnIukAERQAhliXQLBCDFNAFcsOH+6hRauSGiy4j77jXRSZOXnvfKUtLvYxr7xBA/Bjd2CcCaYWACZBRcEaWPgNV2Wj6h7992yTIhuD2DA5yon925g71IBj8rvfeY49jpc+exVvq0M45fbHzYymI+o8BAdq9/RRJWfqu7iWEAAAhOEMQElhomYA1P4ef3PvHfCwjY/rZ943r26wfuJYmGQoUVpd+Tse/39OzbZCQ8kYyF173vYp7joI1JFBWZ9POD/+eHjkbDSRnNA2cMxOTo4wzMxNNEMQCAZZjAxODYv0H0h87ulFaQIe/Fpzc5vSp4BuaAfH9Ooj+kCZMiwkBCiR+9T4QNoHgYuDskuUWSJECeHCYBA9aqXYyNK36/Vor47eXACJy2AO15zHdFGEgw8aOHiWQAACzPBGIwDKL9JCWs1k7OPdjPf5REf5MSi4HOOPz7/WAjsQwAnQATiGt3wA8Sdee9iPR2p3QHOfOYlFgM8LnLF7LEgphgCcDCWZxbhBYMcX9etPBYRxBk9H7HguahbgwkaqvPrbOJMwI6AQsmAGehg0PsdvJdFu+DORUugWfNjZN9+yH+R5lOITXkRYkGB3v2laVzkYUauPfCuzVJW30WbE6fqWAAADyH0MviOEDkhLgonznp/z10d0CUyc3/wA7HqgfvZb9/4B77o3L6voai+K5IU4cTbQNwQiTpVCtwfNLpXTk9e/adHeXU3cT3FXMSczPxgBYBmDoGAPhxUQICKJTzNuEM8mM3KQEYoG0+zEWdl3fo7GUqGQAQwqOnwFegTLcJRU5BOieLqXsSBvg2XxyRfCZBEvBBamwATv3DFks5GAcRzKOY4J0Mg044UvUMMOCsgVdCvIFySGkz9onGJNUMAJ2yjINRXEkuQqLu5yCqSQyAzvXDQciPP38YOL9GHofGezCLQQossqpj5bdXhsFnUsqmVgUoRiDuTsdtKqVoF1i/5Y3i7vLfHzvzHMfn5qEmDJC+z2/sKRHiB5ZKhgGgM7hKqaGaLYR4ht+lkp562fncPqL6IDKQSfoxgDmFuRXFdV36ezPRQkkxAHQLKgFuVOVbhSVymAhRe5wcf3C2//uf+TxFtvGOBTgx5OabLwzQHMJcwpxKi3OPn/6VHAOwOo9tmcpRdgZFd15nPUvz50101t8pwTbwI4oQ5IcJ8KhC37jJVxmntkv9GeYO5lBat/hkxieWewFkAFORZ/sjHT07Hum4g19MmuHXkaV28xwutocRsS86bM4k1MydNp194tzz2b7qaraN8jkFBYGqcC6pDN/91F/ygKC1VdXsV5tfYajXnpDvs5SvOP0LHTwqzlucp2R+Y2+fsVufuLl9BeZQyfTLoSPUz/JILVe1NY9k6CxBhrWmtcdY5bHaF+8IFPcHuwZdNsLGlqGTO7GT264oLHi5RATCql81zlZ03NLeVYzXUvyd+m1A2UHJDehSOkuwnHYKbovqbkJZ+GTyYWW/kqL//Jz0fieCturAO7f3Vj7ECUSoMJMIA7AXjZOuf3P7mnLCR8naAESDSM5Da2DNpfepHGjs+//5D77jaBQU9Vn03EuSEJUrwed8TmBulGDfXLtU0jYAUc+71nUM7tjQcddRF7WsHxtjzaQWNIvyJvH52MgIu5908o093VwdaJS879DeF+wqwLmnWK8vJxsAF/cr2YpNN7avxpyw46dcvpeNDcBtQLlaQGH3KE+zW76kvoPe/pGT38OZQSt9d0qwC8CdF05F8CsQXQgCBvB+ByPgAxRMVFTGqb2EP+si+FaV44pfPC6GAdgwwt2JM+zyNNoHbN3gX+EmDKOhcd+1YQbW/XG2Gl6jtqdl/dUwgKLhp8NFjf1DrG28RBhBUffK82eO8OtrWHspOvOEGVTDAATYM4xAgJg0PTaE7zlahgF4oMgwAg8EJfG1IXzpUTEMQBpVjKXdWOijq2nN2kWAG+Oej9EzDMAHsqysLde2tY6MscsJeWUXodjCQZI+cVKvqoKtLrWTelHg2DCAEFiGe/EojIUVbHkp7ByEQEX0RUnMZ2NsTSVZ9cvFbVcHkg0DUITVJVe3LRvLsM8ZqUARQgXVYLWvGGd3pDkOn6BrsTw2DEAx2mE07BviqsHlVPVixdWXa3Wd1PHVdE3cWrONp3YKGAagFp8FtUFFGK9ky0bH2KVpPoVY0KmIfvCz+BXsrswoW2tEfH1INwxAH24LarZJBhfRybNlxmZQgB7rNN5aerrerPRFuNH40zAAjch1q5q2FBcT8peNjbOLylU6wCpP4bbWQ69HiHc3fJl3ejBgGIAevPquFVuLRBCLR8EQsraDZt+VJLtAFxF6ZyURPDG8TrNll4zBMgwgGeMwCQqoDIMjbDEdV24dY+x0GqhmypQWo2InEXsXBZt4tqKCrautYp3GeDdpiBPxwDCARAyDPBBQHSg2fSMYA0kMM4jQOFOIWo2A+A6oaQJ1Utu9IHQydvYYUV5+LJOQ0zCAJIyCQhiw88CquLTAxkZZI1FoOKlhnHVWVJLTDdII6zIWeYWDlYCq/j+A3luv/BDsyQAAAABJRU5ErkJggg== provider_display_name: Pivotal support_url: TBA plan_updatable: true plans:
- description: 'Example: This plan will configure a lightweight kubernetes cluster.
Not recommended for production workloads.'
instance_groups:
- azs:
- az-1
- az-2
- az-3 instances: 1 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
- azs:
- az-1
- az-2
- az-3 instances: 3 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
- azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
name: wavefront-proxy-errand
-
name: telemetry-pod pre_delete:
-
name: drain-cluster metadata: master_instances: 1 worker_instances: 3 name: Plan 1 plan_id: 8A0E21A8-8072-4D80-B365-D1F502085560 properties: addons-spec: |+
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: namespace-sink-modifier rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group
resources: ["configmaps"]
verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks
- apiGroups: ["apps.pivotal.io"] resources: ["sinks"] verbs: ["get", "list", "watch"]
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: service-account-namespace-sink-modifier subjects:
- kind: ServiceAccount name: default namespace: pks-system roleRef: kind: ClusterRole name: namespace-sink-modifier apiGroup: rbac.authorization.k8s.io
https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Addr example.com:12345
parsers.conf: | [PARSER] Name apache Format regex Regex ^(?
[^ ]) [^ ] (? [^ ]) [(?)] "(? \S+)(?: +(? [^"]?)(?: +\S)?)?" (? [^ ]) (?[^ ] )(?: "(?[^"])" "(? [^"] )")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z[PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.2 imagePullPolicy: Always name: fluent-bit resources: limits: memory: 100Mi requests: cpu: 100m memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true serviceAccountName: default terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.4 imagePullPolicy: Always name: sink-controller
- apiGroups: [""] # "" indicates the core API group
resources: ["configmaps"]
verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
-
- azs:
- description: 'Example: This plan will configure a medium sized kubernetes
cluster, suitable for more pods.'
instance_groups:
-
azs:
- az-1
- az-2
- az-3 instances: 3 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
-
azs:
- az-1
- az-2
- az-3 instances: 5 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
-
azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
name: wavefront-proxy-errand
-
name: telemetry-pod pre_delete:
-
name: drain-cluster metadata: master_instances: 3 worker_instances: 5 name: multi-master plan_id: 58375a45-17f7-4291-acf1-455bfdc8e371 properties: addons-spec: |
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: namespace-sink-modifier rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group
resources: ["configmaps"]
verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks
- apiGroups: ["apps.pivotal.io"] resources: ["sinks"] verbs: ["get", "list", "watch"]
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: service-account-namespace-sink-modifier subjects:
- kind: ServiceAccount name: default namespace: pks-system roleRef: kind: ClusterRole name: namespace-sink-modifier apiGroup: rbac.authorization.k8s.io
https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/output/elasticsearch/fluent-bit-configmap.yaml
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Addr example.com:12345
parsers.conf: | [PARSER] Name apache Format regex Regex ^(?
[^ ]) [^ ] (? [^ ]) [(?)] "(? \S+)(?: +(? [^"]?)(?: +\S)?)?" (? [^ ]) (?[^ ] )(?: "(?[^"])" "(? [^"] )")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z[PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.2 imagePullPolicy: Always name: fluent-bit resources: limits: memory: 100Mi requests: cpu: 100m memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true serviceAccountName: default terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.4 imagePullPolicy: Always name: sink-controller
- apiGroups: [""] # "" indicates the core API group
resources: ["configmaps"]
verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
allow-privileged-containers: true disable_deny_escalating_exec: true
-
-
- null service_description: Default on-demand Kubernetes service. service_name: p.pks tags:
- pivotal
- kubernetes
- k8s service_deployment: releases:
- jobs:
- kube-apiserver
- kube-controller-manager
- kube-scheduler
- kubelet
- kube-proxy
- kubernetes-roles
- flanneld
- nginx
- kubernetes-api-route-registrar
- apply-specs
- secure-var-vcap name: kubo version: 0.21.0
- jobs:
- etcd name: cfcr-etcd version: 1.4.1
- jobs:
- docker name: docker version: 32.0.3
- jobs:
- pks-nsx-t-prepare-master-vm
- pks-nsx-t-ncp name: pks-nsx-t version: 1.11.0
- jobs:
- ncp
- nsx-node-agent
- openvswitch
- nsx-cni
- nsx-kube-proxy name: nsx-cf-cni version: 2.3.0.10066840
- jobs:
- fluentd name: pks-vrli version: 0.6.0
- jobs:
- telemetry-pod name: pks-telemetry version: 0.9.2
- jobs:
- syslog_forwarder name: syslog-migration version: 11.1.1
- jobs:
- bpm name: bpm version: 0.6.0
- jobs:
- wavefront-proxy name: wavefront-proxy version: 0.8.0
- jobs:
- drain-cluster name: pks-helpers version: 50.0.0 stemcell: os: ubuntu-xenial version: "97.19" startup_banner: true username: ((odb_broker_basicauth.username)) stemcell: bosh-vsphere-esxi-ubuntu-xenial-go_agent update: max_in_flight: 1 vm_type: large name: pivotal-container-service-8f7873397b6d8f2b58f6 releases:
- name: cf-mysql version: 36.14.0
- name: docker version: 32.0.3
- name: kubo version: 0.21.0
- name: cfcr-etcd version: 1.4.1
- name: kubo-service-adapter version: 1.2.0-build.166
- name: on-demand-service-broker version: 0.22.0
- name: pks-api version: 1.2.0-build.166
- name: pks-helpers version: 50.0.0
- name: pks-nsx-t version: 1.11.0
- name: nsx-cf-cni version: 2.3.0.10066840
- name: pks-vrli version: 0.6.0
- name: syslog-migration version: 11.1.1
- name: pks-telemetry version: 0.9.2
- name: event-emitter version: 0.13.0
- name: uaa version: "60.2"
- name: bpm version: 0.6.0
- name: backup-and-restore-sdk version: 1.8.0
- name: wavefront-proxy version: 0.8.0 stemcells:
- alias: bosh-vsphere-esxi-ubuntu-xenial-go_agent os: ubuntu-xenial version: "97.19" update: canaries: 1 canary_watch_time: 30000-300000 max_errors: 2 max_in_flight: 1 serial: false update_watch_time: 30000-300000 variables:
- name: kubo_odb_ca options: common_name: ca is_ca: true type: certificate
- name: pks_api_internal
options:
alternative_names:
- localhost
- 127.0.0.1 ca: kubo_odb_ca common_name: localhost type: certificate
- name: uaa_jwt_signing_key_1 type: rsa
- name: uaa_active_pks_saml_key options: common_name: ca is_ca: true type: certificate
- name: pks_api_basicauth type: user
- name: odb_broker_basicauth type: user
- name: telemetry_ca options: common_name: ca is_ca: true type: certificate
- name: telemetry_server_tls options: ca: telemetry_ca common_name: telemetry.pks.internal type: certificate
Succeeded #+END_EXAMPLE ** pks tile: with telemetry enabled :noexport: #+BEGIN_EXAMPLE kubo@jumper:~$ bosh -d pivotal-container-service-69c3e199ae8c7bf63cd0 manifest Using environment '30.0.0.11' as client 'ops_manager'
Using deployment 'pivotal-container-service-69c3e199ae8c7bf63cd0'
instance_groups:
- azs:
- az-1 env: bosh: password: $6$d0454c2683427356$xhIwOfS0hOB1NO8hY71mo0vVrCmS750snO/4m8cCtyf/1TJZUf5yyBUMCruByEdMfDlljufyO4hraIf1mQSJu1 instances: 1 jobs:
- consumes: {} name: service-adapter properties: deployment: broker_deployment_name: pivotal-container-service-69c3e199ae8c7bf63cd0 director_url: https://30.0.0.11:25555 kubo_odb_ca: ((kubo_odb_ca.certificate)) kubo_odb_ca_2018: ((kubo_odb_ca_2018.certificate)) nsxt: upgrade_defaults: nsxt_fip_pool_ids: [] nsxt_lb_service_id: "" nsxt_lb_service_size: "" nsxt_pod_enable_snat: true nsxt_pod_ip_block_ids: [] nsxt_pod_subnet_prefix: 24 nsxt_t0_router_id: "" syslog: migration: disabled: true provides: {} release: kubo-service-adapter
- consumes: broker: from: proxy-broker name: pks-api properties: pks: db_password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_pks_db_password.value)) fqdn: api.pks.local internal_tls: certificate: ((pks_api_internal_2018.certificate)) private_key: ((pks_api_internal_2018.private_key)) password: ((pks_api_basicauth.password)) pks_client_secret: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pks_api_uaa_client.value)) telemetry: authenticationMode: service_account enabled: true eventEmitterBaseUrl: http://localhost:8888 tls: certificate: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pivotal-container-service/pks_tls.cert_pem)) private_key: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pivotal-container-service/pks_tls.private_key_pem)) uaa_service_admin_client_id: service_admin_client uaa_service_admin_client_secret: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pks_services_admin_uaa_client.value)) username: ((pks_api_basicauth.username)) provides: pks_api: as: pks_api_http pks_api_shared: as: pks_api_shared_http shared: true pks_uaa_service_admin_client: as: pks_uaa_service_admin_client shared: true release: pks-api
- consumes: {} name: bosh-update-config properties: bosh: authentication: uaa: client_id: pivotal-container-service-69c3e199ae8c7bf63cd0 client_secret: ((/opsmgr/director/pivotal-container-service-69c3e199ae8c7bf63cd0/uaa_client_secret.value)) url: https://30.0.0.11:8443 root_ca_cert: | -----BEGIN CERTIFICATE----- MIIDUDCCAjigAwIBAgIUZY4hxnalS8wu99hVTPj4WMzHCLUwDQYJKoZIhvcNAQEL BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTkwMTI0MDUz NzU3WhcNMjMwMTI1MDUzNzU3WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2 b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMy3ohmryxgPDHgk VfazuGAkvcWoAYulBG+uBEQYVqV9PTr2A1xr1gOY1jdwPQi1VlbwCnaxteVzIWK/ J/GQwoeJ5TU+pW/0RI70NtuyQ3e6g+sSwLyWTvmlX3dhd5cHpIvU1e3sgU5RkBrJ EXYhGWDI+dtwJFgajVYEhOPvFF2sV+wQ29BYfvuE9LjIETG3N/bH4bhgy7OoZd63 0KtHiVidyTYSDKfXes+nfIa/SPygn5Ap1BKbeQFl1zYAj2A06F5x4h1GqkzjD/ef nh3A+7kC5wwwn70lXMqauS5ik0MIZcVERtX+7q0dc4Rmv6qD7GotqWegVNwcQteK iOQPihsCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUhcHmbshaw71Qve/mcoKXfsCI3aUw HwYDVR0jBBgwFoAUhcHmbshaw71Qve/mcoKXfsCI3aUwHQYDVR0lBBYwFAYIKwYB BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG MA0GCSqGSIb3DQEBCwUAA4IBAQCC64i4seT5kV/VVxjZD5wx+lSPe355y4Fpu99v wJkks2ZPZhH0zVxN0HtBl5D9DTVZCsCFycURt7NcNUrpVkCW4F+2xyD17/LBFn2s eFLl5wrzMkd0S8xIgq8+phnEo7yrNYuTsl+nbxMmDwz/RRIlAEjQZMvXWHVkgktj o+p1POoItxiWcsQNVOXU4JaEk0JJMVW8tdJtfP1dP0iBAbq3OSDNUIg6dlYUZ8gp qEfCoAJSVPA/BzKQY3/cmrPueXoxA1L2T+Mi6gLBEpSjywnW6OXt52sLLbW3IM+j R7t+Thwq0H8M8XT9lG4+pQRYOvRi/0ioD1pvtWa++CIaFKB5 -----END CERTIFICATE----- url: https://30.0.0.11:25555 cloud_config: vm_extensions: - cloud_properties: vmx_options: disk.enableUUID: "1" name: disk_enable_uuid cloud_config_name: pivotal-container-service-69c3e199ae8c7bf63cd0 provides: {} release: pks-api
- consumes: {} name: broker provides: broker: as: odb-broker release: on-demand-service-broker
- consumes:
broker:
from: odb-broker
name: pks-nsx-t-osb-proxy
properties:
bosh:
authentication:
uaa:
client_id: pivotal-container-service-69c3e199ae8c7bf63cd0
client_secret: ((/opsmgr/director/pivotal-container-service-69c3e199ae8c7bf63cd0/uaa_client_secret.value))
cloud_config_dns: 192.168.115.1
cloud_config_prefix: pks
log_level: INFO
root_ca_cert: |
-----BEGIN CERTIFICATE-----
MIIDUDCCAjigAwIBAgIUZY4hxnalS8wu99hVTPj4WMzHCLUwDQYJKoZIhvcNAQEL
BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTkwMTI0MDUz
NzU3WhcNMjMwMTI1MDUzNzU3WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2
b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMy3ohmryxgPDHgk
VfazuGAkvcWoAYulBG+uBEQYVqV9PTr2A1xr1gOY1jdwPQi1VlbwCnaxteVzIWK/
J/GQwoeJ5TU+pW/0RI70NtuyQ3e6g+sSwLyWTvmlX3dhd5cHpIvU1e3sgU5RkBrJ
EXYhGWDI+dtwJFgajVYEhOPvFF2sV+wQ29BYfvuE9LjIETG3N/bH4bhgy7OoZd63
0KtHiVidyTYSDKfXes+nfIa/SPygn5Ap1BKbeQFl1zYAj2A06F5x4h1GqkzjD/ef
nh3A+7kC5wwwn70lXMqauS5ik0MIZcVERtX+7q0dc4Rmv6qD7GotqWegVNwcQteK
iOQPihsCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUhcHmbshaw71Qve/mcoKXfsCI3aUw
HwYDVR0jBBgwFoAUhcHmbshaw71Qve/mcoKXfsCI3aUwHQYDVR0lBBYwFAYIKwYB
BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG
MA0GCSqGSIb3DQEBCwUAA4IBAQCC64i4seT5kV/VVxjZD5wx+lSPe355y4Fpu99v
wJkks2ZPZhH0zVxN0HtBl5D9DTVZCsCFycURt7NcNUrpVkCW4F+2xyD17/LBFn2s
eFLl5wrzMkd0S8xIgq8+phnEo7yrNYuTsl+nbxMmDwz/RRIlAEjQZMvXWHVkgktj
o+p1POoItxiWcsQNVOXU4JaEk0JJMVW8tdJtfP1dP0iBAbq3OSDNUIg6dlYUZ8gp
qEfCoAJSVPA/BzKQY3/cmrPueXoxA1L2T+Mi6gLBEpSjywnW6OXt52sLLbW3IM+j
R7t+Thwq0H8M8XT9lG4+pQRYOvRi/0ioD1pvtWa++CIaFKB5
-----END CERTIFICATE-----
url: https://30.0.0.11:25555
create_network_with_lb: false
enabled: true
fip_address_parameter: nsxt_fip_address
generate_lb_name: true
kubernetes_master_host_parameter: kubernetes_master_host
lb_service_id_parameter: nsxt_lb_service_id
log_level: INFO
network_name_parameter: nsxt_network_name
nsxt:
ca_cert: |
-----BEGIN CERTIFICATE-----
MIIDZDCCAkygAwIBAgIGAWiDfMRGMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM
G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww
CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ
UGFsbyBBbHRvMB4XDTE5MDEyNTA1MzAxOFoXDTI0MDEyNDA1MzAxOFowczEkMCIG
A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh
cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD
VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDr
kLoDtLBQbpRbid5ZMpU9ljWnSTB1VMYHRJ8VIxGfGqkx3Qxswzt8Sxxa3o2a1heD
1tsXVokXXX+XQJeHto5nzEkLqe5vp5ieqkyazTDmpby879aRno6NNICHnsNWtBFD
4IlfT0Z6qJhCz3M41Z3mrnfeboaqHD3OueBCCBN3i3r2VF3awaKIaUtB5SWG9rud
853Q/lf3LG65VICHYZxczhgjqFolKO51lgpqT1QhGGKdZoUfSHwsZEXpXAkwivJn
ivn01jVdNIk+EDPJEqpLt7L8YC1Q0+POyho7Yvp0b0DaWUVqkGckjxyIQKp/wVpb
kPd+g6kRrJj+YGMBxZwrAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAK2H5AzaC+qY
HnTI+om/u1pFYC0ZbL62CIRwdirsKhgnq5ATViYXS1NWWihX8wBXZa47gQB9uQeF
BbeImPJQcXHX5b7f5e+iNoAJYNUCtcL7u/qW1RZShCXR2KKWhIkDNmGJ1ngYX7YY
1u3SV47rimn6Vrp84KcW/yTLXGt8Gt66N1LEJSOekt8zqGLnKDChOPxbgWwijwJ9
l5+Db7EZXiXwjgnMqPYt/sre+oX0iIz5iCqkOgNHdRrSnmoYO9NjSoVgC1zrInC7
Z3tduIy2sZSmLXHZnKDLALcPsgF8W2S7kiyTJKS66iq+PIH4oGnhONyY5rQ09Mvb
HRio4yUphiA=
-----END CERTIFICATE-----
floating_ip_pool_ids:
- 03f956ef-de75-4da1-ab38-a2b8a7128b22
host: nsxmanager.pks.vmware.local
insecure: true
ip_block_id: 7361596a-92ce-48ea-8996-eb7d24da0789
lb_size_large_supported: true
lb_size_medium_supported: true
log_level: INFO
nat_mode: true
network_prefix: pks
pod_ip_block_id: a2aa1b3e-1904-4240-9918-9f737d6ddbf8
superuser_cert: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/network_selector/nsx/nsx-t-superuser-certificate.cert_pem))
superuser_key: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem))
t0_router_id: ddf4515a-dd2f-4bff-8c9f-97dd9cda9c5a
password: ((odb_broker_basicauth.password))
plans:
- description: 'Example: This plan will configure a lightweight kubernetes cluster.
Not recommended for production workloads.'
instance_groups:
- azs:
- az-1
- az-2
- az-3 instances: 1 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
- azs:
- az-1
- az-2
- az-3 instances: 3 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
- azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
disabled: true name: wavefront-proxy-errand
-
name: telemetry-agent pre_delete:
-
name: drain-cluster metadata: allow-privileged-containers: true master_instances: 1 max_worker_instances: 50 worker_instances: 3 name: Plan 1 plan_id: 8A0E21A8-8072-4D80-B365-D1F502085560 properties: addons-spec: |+
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Sinks [{"addr":"example.com:12345"}]
parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log
apiVersion: v1 kind: Service metadata: name: fluent-bit namespace: pks-system spec: selector: k8s-app: logging-agent ports: - protocol: TCP port: 24224 targetPort: forward-plugin type: ClusterIP
apiVersion: batch/v1 kind: Job metadata: labels: job: cert-generator name: cert-generator-v0.11 namespace: pks-system spec: backoffLimit: 0 template: metadata: labels: job: cert-generator spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/cert-generator:v0.12 imagePullPolicy: IfNotPresent name: cert-generator restartPolicy: Never serviceAccountName: cert-generator
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: fluent-bit resources: limits: memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true - command: - ghostunnel - server - --listen - :24224 - --target - localhost:24225 - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --allow-dns-san - event-controller - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel ports: - containerPort: 24224 name: forward-plugin volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /fluent-bit-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /fluent-bit-certs name: fluent-bit-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config - emptyDir: {} name: keystore - name: fluent-bit-certs secret: secretName: fluent-bit - name: pks-ca secret: secretName: pks-ca updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: event-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: event-controller spec: containers: - env: - name: FORWARDER_HOST value: localhost image: oratos/event-controller:v0.12 imagePullPolicy: IfNotPresent name: event-controller - command: - ghostunnel - client - --listen - localhost:24224 - --target - fluent-bit.pks-system.svc.cluster.local:24224 - --override-server-name - fluent-bit - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /event-controller-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /event-controller-certs name: event-controller-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: event-controller volumes: - emptyDir: {} name: keystore - name: event-controller-certs secret: secretName: event-controller - name: pks-ca secret: secretName: pks-ca
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.12 imagePullPolicy: IfNotPresent name: sink-controller serviceAccountName: sink-controller
-
- azs:
- description: 'Example: This plan will configure a medium sized kubernetes
cluster, suitable for more pods.'
instance_groups:
-
azs:
- az-1
- az-2
- az-3 instances: 3 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
-
azs:
- az-1
- az-2
- az-3 instances: 5 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
-
azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
disabled: true name: wavefront-proxy-errand
-
name: telemetry-agent pre_delete:
-
name: drain-cluster metadata: allow-privileged-containers: true master_instances: 3 max_worker_instances: 50 worker_instances: 5 name: multi-master plan_id: 58375a45-17f7-4291-acf1-455bfdc8e371 properties: addons-spec: |
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Sinks [{"addr":"example.com:12345"}]
parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log
apiVersion: v1 kind: Service metadata: name: fluent-bit namespace: pks-system spec: selector: k8s-app: logging-agent ports: - protocol: TCP port: 24224 targetPort: forward-plugin type: ClusterIP
apiVersion: batch/v1 kind: Job metadata: labels: job: cert-generator name: cert-generator-v0.11 namespace: pks-system spec: backoffLimit: 0 template: metadata: labels: job: cert-generator spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/cert-generator:v0.12 imagePullPolicy: IfNotPresent name: cert-generator restartPolicy: Never serviceAccountName: cert-generator
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: fluent-bit resources: limits: memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true - command: - ghostunnel - server - --listen - :24224 - --target - localhost:24225 - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --allow-dns-san - event-controller - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel ports: - containerPort: 24224 name: forward-plugin volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /fluent-bit-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /fluent-bit-certs name: fluent-bit-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config - emptyDir: {} name: keystore - name: fluent-bit-certs secret: secretName: fluent-bit - name: pks-ca secret: secretName: pks-ca updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: event-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: event-controller spec: containers: - env: - name: FORWARDER_HOST value: localhost image: oratos/event-controller:v0.12 imagePullPolicy: IfNotPresent name: event-controller - command: - ghostunnel - client - --listen - localhost:24224 - --target - fluent-bit.pks-system.svc.cluster.local:24224 - --override-server-name - fluent-bit - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /event-controller-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /event-controller-certs name: event-controller-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: event-controller volumes: - emptyDir: {} name: keystore - name: event-controller-certs secret: secretName: event-controller - name: pks-ca secret: secretName: pks-ca
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.12 imagePullPolicy: IfNotPresent name: sink-controller serviceAccountName: sink-controller
allow-privileged-containers: true disable_deny_escalating_exec: true max_worker_instances: 50
-
-
- null port: 3000 proxy: null username: ((odb_broker_basicauth.username)) provides: {} release: pks-nsx-t
- description: 'Example: This plan will configure a lightweight kubernetes cluster.
Not recommended for production workloads.'
instance_groups:
- consumes: {} name: mysql properties: cf_mysql: mysql: admin_password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_mysql_admin_password.value)) cluster_health: password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_mysql_cluster_health_password.value)) galera_healthcheck: db_password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_mysql_galera_healthcheck_db_password.value)) endpoint_password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_mysql_galera_healthcheck_endpoint_password.value)) seeded_databases: - name: pks password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_pks_db_password.value)) username: pks - name: uaa password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_uaa_db_password.value)) username: uaa - name: telemetry password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_telemetry_db_password.value)) username: telemetry - name: billing password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_billing_db_password.value)) username: billing provides: mysql: as: mysql release: cf-mysql
- consumes: {} name: uaa properties: encryption: active_key_label: key-1 encryption_keys: - label: key-1 passphrase: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/uaa_encryption_passphrase.value)) login: saml: activeKeyId: active-pks-saml-key keys: active-pks-saml-key: certificate: ((uaa_active_pks_saml_key_2018.certificate)) key: ((uaa_active_pks_saml_key_2018.private_key)) passphrase: "" signatureAlgorithm: SHA256 release_level_backup: true uaa: clients: admin: authorities: uaa.admin,clients.read,clients.write,clients.secret,scim.read,scim.write,clients.admin,pks.clusters.admin,pks.clusters.manage authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pks_uaa_management_admin_client.value)) pks_cli: access-token-validity: 7200 authorities: uaa.resource authorized-grant-types: password,refresh_token refresh-token-validity: 21600 scope: pks.clusters.admin,pks.clusters.manage secret: "" pks_client: access-token-validity: 86400 authorities: pks.clusters.admin,pks.clusters.manage,uaa.resource authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pks_api_uaa_client.value)) pks_cluster_client: authorities: uaa.resource authorized-grant-types: password,refresh_token scope: openid,roles secret: "" service_admin_client: authorities: clients.admin authorized-grant-types: client_credentials scope: uaa.none secret: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pks_services_admin_uaa_client.value)) jwt: policy: active_key_id: key-1 keys: key-1: signingKey: ((uaa_jwt_signing_key_1.private_key)) ldap: emailDomain: [] enabled: false externalGroupsWhitelist: - '*' groups: groupSearchFilter: member={0} profile_type: no-groups searchBase: null mailAttributeName: mail referral: follow searchBase: null searchFilter: cn={0} sslCertificate: null sslCertificateAlias: null url: null userDN: null userPassword: null port: 35684 scim: groups: pks.clusters.admin: Allows a user to admin PKS pks.clusters.manage: Allows a user to manage PKS clusters user: override: true users: - groups: - uaa.admin - pks.clusters.admin name: admin password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/uaa_admin_password.value)) sslCertificate: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pivotal-container-service/pks_tls.cert_pem)) sslPrivateKey: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pivotal-container-service/pks_tls.private_key_pem)) url: https://api.pks.local:8443 uaadb: address: 127.0.0.1 databases: - name: uaa tag: uaa db_scheme: mysql port: 3306 roles: - name: uaa password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_uaa_db_password.value)) tag: admin provides: {} release: uaa
- consumes: {} name: bbr-uaadb provides: {} release: uaa
- consumes: {} name: database-backup-restorer provides: {} release: backup-and-restore-sdk
- consumes: broker: from: proxy-broker name: upgrade-all-service-instances provides: {} release: on-demand-service-broker
- consumes: {} name: syslog_forwarder properties: syslog: migration: disabled: true provides: {} release: syslog
- consumes: {} name: sink-resources-images provides: {} release: sink-resources-release
- consumes: {} name: sink-resources-images-ops-files provides: {} release: sink-resources-release
- consumes: pks_api: from: pks_api_http name: delete-all-clusters provides: {} release: pks-api
- consumes: {}
name: pks-nsx-t-precheck
properties:
floating-ip-pool-ids:
- 03f956ef-de75-4da1-ab38-a2b8a7128b22 ip-block-id: a2aa1b3e-1904-4240-9918-9f737d6ddbf8 network-automation: true nodes-ip-block-id: 7361596a-92ce-48ea-8996-eb7d24da0789 nsx-t-ca-cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWiDfMRGMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE5MDEyNTA1MzAxOFoXDTI0MDEyNDA1MzAxOFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDr kLoDtLBQbpRbid5ZMpU9ljWnSTB1VMYHRJ8VIxGfGqkx3Qxswzt8Sxxa3o2a1heD 1tsXVokXXX+XQJeHto5nzEkLqe5vp5ieqkyazTDmpby879aRno6NNICHnsNWtBFD 4IlfT0Z6qJhCz3M41Z3mrnfeboaqHD3OueBCCBN3i3r2VF3awaKIaUtB5SWG9rud 853Q/lf3LG65VICHYZxczhgjqFolKO51lgpqT1QhGGKdZoUfSHwsZEXpXAkwivJn ivn01jVdNIk+EDPJEqpLt7L8YC1Q0+POyho7Yvp0b0DaWUVqkGckjxyIQKp/wVpb kPd+g6kRrJj+YGMBxZwrAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAK2H5AzaC+qY HnTI+om/u1pFYC0ZbL62CIRwdirsKhgnq5ATViYXS1NWWihX8wBXZa47gQB9uQeF BbeImPJQcXHX5b7f5e+iNoAJYNUCtcL7u/qW1RZShCXR2KKWhIkDNmGJ1ngYX7YY 1u3SV47rimn6Vrp84KcW/yTLXGt8Gt66N1LEJSOekt8zqGLnKDChOPxbgWwijwJ9 l5+Db7EZXiXwjgnMqPYt/sre+oX0iIz5iCqkOgNHdRrSnmoYO9NjSoVgC1zrInC7 Z3tduIy2sZSmLXHZnKDLALcPsgF8W2S7kiyTJKS66iq+PIH4oGnhONyY5rQ09Mvb HRio4yUphiA= -----END CERTIFICATE----- nsx-t-host: nsxmanager.pks.vmware.local nsx-t-insecure: true nsx-t-superuser-certificate: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) nsx-t-superuser-key: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) proxy: null t0-router-id: ddf4515a-dd2f-4bff-8c9f-97dd9cda9c5a vcenter-cluster: kubo-az-1 vcenter-datacenter: kubo-dc vcenter-host: 192.168.111.8 vcenter-insecure: true vcenter-password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cloud_provider/vsphere/vcenter_master_creds.password)) vcenter-user: [email protected] provides: {} release: pks-nsx-t
- consumes: {}
name: pks-nsx-t-ops-files
properties:
fip_address_parameter: nsxt_fip_address
floating-ip-pool-ids:
- 03f956ef-de75-4da1-ab38-a2b8a7128b22 kubo-odb-ca-2018-added: true kubo-odb-ca-2018-used: true lb-created-by-proxy: false nsx-t-ca-cert: | -----BEGIN CERTIFICATE----- MIIDZDCCAkygAwIBAgIGAWiDfMRGMA0GCSqGSIb3DQEBCwUAMHMxJDAiBgNVBAMM G25zeG1hbmFnZXIucGtzLnZtd2FyZS5sb2NhbDEPMA0GA1UECgwGVk13YXJlMQww CgYDVQQLDANDTkExCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJ UGFsbyBBbHRvMB4XDTE5MDEyNTA1MzAxOFoXDTI0MDEyNDA1MzAxOFowczEkMCIG A1UEAwwbbnN4bWFuYWdlci5wa3Mudm13YXJlLmxvY2FsMQ8wDQYDVQQKDAZWTXdh cmUxDDAKBgNVBAsMA0NOQTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRIwEAYD VQQHDAlQYWxvIEFsdG8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDr kLoDtLBQbpRbid5ZMpU9ljWnSTB1VMYHRJ8VIxGfGqkx3Qxswzt8Sxxa3o2a1heD 1tsXVokXXX+XQJeHto5nzEkLqe5vp5ieqkyazTDmpby879aRno6NNICHnsNWtBFD 4IlfT0Z6qJhCz3M41Z3mrnfeboaqHD3OueBCCBN3i3r2VF3awaKIaUtB5SWG9rud 853Q/lf3LG65VICHYZxczhgjqFolKO51lgpqT1QhGGKdZoUfSHwsZEXpXAkwivJn ivn01jVdNIk+EDPJEqpLt7L8YC1Q0+POyho7Yvp0b0DaWUVqkGckjxyIQKp/wVpb kPd+g6kRrJj+YGMBxZwrAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAK2H5AzaC+qY HnTI+om/u1pFYC0ZbL62CIRwdirsKhgnq5ATViYXS1NWWihX8wBXZa47gQB9uQeF BbeImPJQcXHX5b7f5e+iNoAJYNUCtcL7u/qW1RZShCXR2KKWhIkDNmGJ1ngYX7YY 1u3SV47rimn6Vrp84KcW/yTLXGt8Gt66N1LEJSOekt8zqGLnKDChOPxbgWwijwJ9 l5+Db7EZXiXwjgnMqPYt/sre+oX0iIz5iCqkOgNHdRrSnmoYO9NjSoVgC1zrInC7 Z3tduIy2sZSmLXHZnKDLALcPsgF8W2S7kiyTJKS66iq+PIH4oGnhONyY5rQ09Mvb HRio4yUphiA= -----END CERTIFICATE----- nsx-t-host: nsxmanager.pks.vmware.local nsx-t-insecure: true nsx-t-superuser-certificate: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/network_selector/nsx/nsx-t-superuser-certificate.cert_pem)) nsx-t-superuser-key: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/network_selector/nsx/nsx-t-superuser-certificate.private_key_pem)) pod-ip-block-id: a2aa1b3e-1904-4240-9918-9f737d6ddbf8 t0-router-id: ddf4515a-dd2f-4bff-8c9f-97dd9cda9c5a provides: {} release: pks-nsx-t
- consumes: {} name: pks-wavefront-ops-files properties: wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: wavefront-alert-creation properties: wavefront-alert-targets: ignored wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: wavefront-alert-deletion properties: wavefront-api-url: ignored wavefront-token: ignored provides: {} release: wavefront-proxy
- consumes: {} name: pks-vrli-ops-files properties: fluentd_vrli_ca_cert: ignored fluentd_vrli_host: ignored fluentd_vrli_rate_limit_msec: ignored fluentd_vrli_skip_cert_verify: ignored fluentd_vrli_use_ssl: ignored provides: {} release: pks-vrli
- consumes: mysql: from: mysql name: telemetry-server properties: billing: db-name: billing db-password: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/cf_mysql_billing_db_password.value)) db-username: billing cloud-provider: vSphere forward: tls: certificate: ((telemetry_server_tls_2018.certificate)) private_key: ((telemetry_server_tls_2018.private_key)) hostname: telemetry.pks.internal networking: nsx pks-instance-id: pivotal-container-service-69c3e199ae8c7bf63cd0 product-version: 1.3.0-build.10 telemetry-enabled: true vac-server-url: https://vcsa.vmware.com/ph-stg vcenter-server-url: https://192.168.111.8 provides: telemetry-server: as: telemetry-server shared: true release: pks-telemetry
- consumes: {} name: telemetry-ops-files properties: telemetry-agent: billing: polling-interval-seconds: 60 telemetry: polling-interval-seconds: 600 telemetry-server: ca: certificate: | ((telemetry_ca_2018.certificate)) provides: {} release: pks-telemetry
- consumes: {} name: bpm provides: {} release: bpm
- consumes: pks_api: from: pks_api_http name: smoke-tests properties: smoke_tests: client: admin secret: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pks_uaa_management_admin_client.value)) provides: {} release: pks-api lifecycle: service name: pivotal-container-service networks:
- default:
- dns
- gateway
name: deployment-network
persistent_disk_type: "10240"
properties:
bosh:
authentication:
uaa:
client_id: pivotal-container-service-69c3e199ae8c7bf63cd0
client_secret: ((/opsmgr/director/pivotal-container-service-69c3e199ae8c7bf63cd0/uaa_client_secret.value))
url: https://30.0.0.11:8443
root_ca_cert: |
-----BEGIN CERTIFICATE-----
MIIDUDCCAjigAwIBAgIUZY4hxnalS8wu99hVTPj4WMzHCLUwDQYJKoZIhvcNAQEL
BQAwHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bpdm90YWwwHhcNMTkwMTI0MDUz
NzU3WhcNMjMwMTI1MDUzNzU3WjAfMQswCQYDVQQGEwJVUzEQMA4GA1UECgwHUGl2
b3RhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMy3ohmryxgPDHgk
VfazuGAkvcWoAYulBG+uBEQYVqV9PTr2A1xr1gOY1jdwPQi1VlbwCnaxteVzIWK/
J/GQwoeJ5TU+pW/0RI70NtuyQ3e6g+sSwLyWTvmlX3dhd5cHpIvU1e3sgU5RkBrJ
EXYhGWDI+dtwJFgajVYEhOPvFF2sV+wQ29BYfvuE9LjIETG3N/bH4bhgy7OoZd63
0KtHiVidyTYSDKfXes+nfIa/SPygn5Ap1BKbeQFl1zYAj2A06F5x4h1GqkzjD/ef
nh3A+7kC5wwwn70lXMqauS5ik0MIZcVERtX+7q0dc4Rmv6qD7GotqWegVNwcQteK
iOQPihsCAwEAAaOBgzCBgDAdBgNVHQ4EFgQUhcHmbshaw71Qve/mcoKXfsCI3aUw
HwYDVR0jBBgwFoAUhcHmbshaw71Qve/mcoKXfsCI3aUwHQYDVR0lBBYwFAYIKwYB
BQUHAwIGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEG
MA0GCSqGSIb3DQEBCwUAA4IBAQCC64i4seT5kV/VVxjZD5wx+lSPe355y4Fpu99v
wJkks2ZPZhH0zVxN0HtBl5D9DTVZCsCFycURt7NcNUrpVkCW4F+2xyD17/LBFn2s
eFLl5wrzMkd0S8xIgq8+phnEo7yrNYuTsl+nbxMmDwz/RRIlAEjQZMvXWHVkgktj
o+p1POoItxiWcsQNVOXU4JaEk0JJMVW8tdJtfP1dP0iBAbq3OSDNUIg6dlYUZ8gp
qEfCoAJSVPA/BzKQY3/cmrPueXoxA1L2T+Mi6gLBEpSjywnW6OXt52sLLbW3IM+j
R7t+Thwq0H8M8XT9lG4+pQRYOvRi/0ioD1pvtWa++CIaFKB5
-----END CERTIFICATE-----
url: https://30.0.0.11:25555
disable_cf_startup_checks: true
expose_operational_errors: true
password: ((odb_broker_basicauth.password))
port: 8080
service_adapter:
path: /var/vcap/jobs/service-adapter/bin/service-adapter
service_catalog:
bindable: true
global_properties:
authorization_mode: rbac
deployments_network: service-network
iaas: vsphere
oidc: false
oidc_ca: ((/opsmgr/pivotal-container-service-69c3e199ae8c7bf63cd0/pivotal-container-service/pks_tls.cert_pem))
oidc_client_id: pks_cluster_client
oidc_groups_claim: roles
oidc_groups_prefix: ""
oidc_issuer_url: https://api.pks.local:8443/oauth/token
oidc_username_claim: user_name
oidc_username_prefix: '-'
ops_files_paths:
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_pks_nsx_t.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/prepare_master_vm.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_fip_to_tls_certs.yml
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/remove_flannel.yml
- ""
- /var/vcap/jobs/pks-nsx-t-ops-files/manifests/add_master_vms_to_nsgroup.yml
- /var/vcap/jobs/pks-wavefront-ops-files/manifests/add-wavefront-job.yml
- null
- /var/vcap/jobs/telemetry-ops-files/manifests/add-telemetry-dns.yml
- /var/vcap/jobs/telemetry-ops-files/manifests/add-telemetry-agent-image.yml
- /var/vcap/jobs/telemetry-ops-files/manifests/add-telemetry-agent-deploy-errand.yml
- /var/vcap/jobs/sink-resources-images-ops-files/manifests/add-sink-resources-images.yml pod_network_cidr: null proxy: null routing_mode: external service_cluster_cidr: null vcenter_dc: kubo-dc vcenter_ds: iscsi-ds-0 vcenter_ip: 192.168.111.8 vcenter_vms: pcf_vms worker_max_in_flight: 1 id: DF8EECC4-7225-42D0-8459-4A6C584314CA metadata: display_name: Kubernetes documentation_url: TBA image_url: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQAAAAEACAYAAABccqhmAAAKp2lDQ1BJQ0MgUHJvZmlsZQAASImVlwdUFEkax6t7ciINDEHCkJPkDJLjkCWDqAwzhCGMw8CQzMqigmtARQQURVeSAosKiAkQBZVFMIB5QRYV5VwMiIo618Ax3N69u3v39atXv/d19b++rq56798AkHuYPF4KLAFAKjeDH+TpQo+IjKLjRgAauQhAAlCYrHSec2CgL0Biof9rfBoE0Gx/12BW69/v/9eQZMelswCAAhGOZaezUhE+i7RGFo+fAQCKjeTVszJ4s7wdYWk+UiDC5bOcMM+Nsxw7z11zY0KCXBF+AACezGTyEwAg/YHk6ZmsBESHjEbYmMvmcBE2R9iBlchE5iEj98DS1NQ1s3wUYZ3Yf9JJ+ItmrEiTyUwQ8fy7zAXejZPOS2Hm/J/L8b8jNUWwMIca0siJfK+g2fmQNatOXuMjYm6sf8ACc9jzNc1yosArdIFZ6a5RC8xmuvkssCA51HmBmfzFZzkZjJAF5q8JEulzU/x9RfpxDBHHpbsHL3A8x4OxwLmJIeELnMkJ81/g9ORgn8UxrqI8XxAkqjme7yF6x9T0xdpYzMW5MhJDvBZriBDVw45zcxfluaGi8bwMF5EmLyVwsf4UT1E+PTNY9GwGssEWOInpHbioEyhaH8ABfoAJWBlx2bP7Criu4eXwOQmJGXRn5JTE0RlcluFSuqmxiRUAs2du/pN+oM2dJYh2czG39QgA9meFQuGFxZxPGwBnCgEgDi3mtNcDINYBQE8ZS8DPnM/NbnWAAUQgDqSBPFAG6kAHGABTYAnsgBNwB94gAISASLAKsEAiSAV8kAXWgc0gHxSCPeAAKAUV4DioBqdBE2gBF0EHuA5ugX5wHzwGw2AMvAGT4BOYgSAIB1EgKiQPqUCakD5kCllDDpA75AsFQZFQDJQAcSEBtA7aChVCRVApdAyqgX6FzkMd0A1oAHoIjUDj0HvoK4yCybA0rARrwUawNewM+8Ah8Eo4AU6Dc+E8eBdcAlfCp+BmuAO+Bd+Hh+E38BQKoEgoGkoVZYCyRrmiAlBRqHgUH7UBVYAqRlWi6lFtqG7UXdQwagL1BY1FU9F0tAHaDu2FDkWz0GnoDeid6FJ0NboZ3YW+ix5BT6J/YCgYRYw+xhbDwERgEjBZmHxMMeYk5hzmGuY+ZgzzCYvF0rDaWCusFzYSm4Rdi92JPYxtwLZjB7Cj2CkcDieP08fZ4wJwTFwGLh93CHcKdwV3BzeG+4wn4VXwpngPfBSei9+CL8bX4i/j7+Bf4mcIEgRNgi0hgMAm5BB2E04Q2gi3CWOEGaIkUZtoTwwhJhE3E0uI9cRrxCfEDyQSSY1kQ1pO4pA2kUpIjaQe0gjpC1mKrEd2JUeTBeRd5CpyO/kh+QOFQtGiOFGiKBmUXZQaylXKM8pnMaqYoRhDjC22UaxMrFnsjthbcYK4priz+CrxXPFi8TPit8UnJAgSWhKuEkyJDRJlEuclhiSmJKmSJpIBkqmSOyVrJW9IvpLCSWlJuUuxpfKkjktdlRqloqjqVFcqi7qVeoJ6jTomjZXWlmZIJ0kXSp+W7pOelJGSMZcJk8mWKZO5JDNMQ9G0aAxaCm03rYk2SPsqqyTrLBsnu0O2XvaO7LTcEjknuTi5ArkGuftyX+Xp8u7yyfJ75VvknyqgFfQUlitkKRxRuKYwsUR6id0S1pKCJU1LHinCinqKQYprFY8r9ipOKSkreSrxlA4pXVWaUKYpOyknKe9Xvqw8rkJVcVDhqOxXuaLymi5Dd6an0EvoXfRJVUVVL1WB6jHVPtUZNW21ULUtag1qT9WJ6tbq8er71TvVJzVUNPw01mnUaTzSJGhaayZqHtTs1pzW0tYK19qm1aL1SltOm6Gdq12n/USHouOok6ZTqXNPF6trrZuse1i3Xw/Ws9BL1CvTu60P61vqc/QP6w8sxSy1WcpdWrl0yIBs4GyQaVBnMGJIM/Q13GLYYvjWSMMoymivUbfRD2ML4xTjE8aPTaRMvE22mLSZvDfVM2WZlpneM6OYeZhtNGs1e2eubx5nfsT8gQXVws9im0WnxXdLK0u+Zb3luJWGVYxVudWQtbR1oPVO6x4bjI2LzUabizZfbC1tM2ybbP+0M7BLtqu1e7VMe1ncshPLRu3V7Jn2x+yHHegOMQ5HHYYdVR2ZjpWOz53UndhOJ51eOus6Jzmfcn7rYuzCdznnMu1q67retd0N5ebpVuDW5y7lHupe6v7MQ80jwaPOY9LTwnOtZ7sXxsvHa6/XEEOJwWLUMCa9rbzXe3f5kH2CfUp9nvvq+fJ92/xgP2+/fX5P/DX9uf4tASCAEbAv4GmgdmBa4IXl2OWBy8uWvwgyCVoX1B1MDV4dXBv8KcQlZHfI41CdUEFoZ5h4WHRYTdh0uFt4UfhwhFHE+ohbkQqRnMjWKFxUWNTJqKkV7isOrBiLtojOjx5cqb0ye+WNVQqrUlZdWi2+mrn6TAwmJjymNuYbM4BZyZyKZcSWx06yXFkHWW/YTuz97PE4+7iiuJfx9vFF8a8S7BP2JYwnOiYWJ05wXDmlnHdJXkkVSdPJAclVycKU8JSGVHxqTOp5rhQ3mdu1RnlN9poBnj4vnzecZpt2IG2S78M/mQ6lr0xvzZBGzE2vQEfwk2Ak0yGzLPNzVljWmWzJbG52b45ezo6cl7keub+sRa9lre1cp7pu87qR9c7rj22ANsRu6NyovjFv49gmz03Vm4mbkzf/tsV4S9GWj1vDt7blKeVtyhv9yfOnunyxfH7+0Da7bRXb0ds52/t2mO04tONHAbvgZqFxYXHht52snTd/Nvm55Gfhrvhdfbstdx/Zg93D3TO413FvdZFkUW7R6D6/fc376fsL9n88sPrAjWLz4oqDxIOCg8MlviWthzQO7Tn0rTSx9H6ZS1lDuWL5jvLpw+zDd444HamvUKoorPh6lHP0wTHPY82VWpXFx7HHM4+/OBF2ovsX619qTiqcLDz5vYpbNVwdVN1VY1VTU6tYu7sOrhPUjZ+KPtV/2u10a71B/bEGWkNhI2gUNL7+NebXwSafps4z1mfqz2qeLT9HPVfQDDXnNE+2JLYMt0a2Dpz3Pt/ZZtd27oLhhaqLqhfLLslc2n2ZeDnvsvBK7pWpdl77REdCx2jn6s7HVyOu3uta3tV3zedaz3WP61e7nbuv9Nj3XLxhe+P8TeubLbcsbzX3WvSe+83it3N9ln3Nt61ut/bb9LcNLBu4fMfxTsddt7vX7zHu3brvf39gMHTwwVD00PAD9oNXD1MevnuU+Wjm8aYnmCcFTyWeFj9TfFb5u+7vDcOWw5dG3EZ6nwc/fzzKGn3zR/of38byXlBeFL9UeVnzyvTVxXGP8f7XK16PveG9mZnI/5vk38rf6rw9+6fTn72TEZNj7/jvhO93fpD/UPXR/GPnVODUs0+pn2amCz7Lf67+Yv2l+2v415czWd9w30q+635v++Hz44kwVSjkMfnMOSuAQhocHw/A+yoAKJEAUPsR/yA274nnApr38XME/hPP++a5sASgHulmrZBrOwCNSNNyQrSRftYShTgB2MxM1P4R6fFmpvNaZMRZYj4LhR+UAMAhfuY7XyicOSwUfj+BFPsQgPa0eS8+G1jkD6Ue1zS6T3ygzWgT+Jf4O0btBkBVU4mLAAAB1WlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyI+CiAgICAgICAgIDx0aWZmOkNvbXByZXNzaW9uPjE8L3RpZmY6Q29tcHJlc3Npb24+CiAgICAgICAgIDx0aWZmOk9yaWVudGF0aW9uPjE8L3RpZmY6T3JpZW50YXRpb24+CiAgICAgICAgIDx0aWZmOlBob3RvbWV0cmljSW50ZXJwcmV0YXRpb24+MjwvdGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KAtiABQAAO3BJREFUeAHtfQmcXUWZb93e0kuW7iSQEBLosAYQCFtocICOOqLyU4KOT984jonPeU9nVBpmngOIksjuqHQct9E3Ehxn0fccgqKyDJAQCE3Ymn0JkA4JCQlJujtJL+n1ff+699w+9/apc+qcU3WWe6vy69x7z6nlq6/q++r7vvrqqwwzqaQw0HJVWzOrYs3o1Ngoa2QZtjhUB8dZZ0Ul6+F1jLCujlvau0LVZwonCgOZREFjgPHEwDnXtC2urGCNY2OsdXyczRhnWQLPZFirZ2GFGajtdaiOJlAntd1bUcHWjY6xniduau9U2IypSjMGDAPQjOCg1bfednPz4N5dzSD0McZOp4FqprrCreZBgfFfrpMYU1cFY8+CMTAjOfjHYEQlDAOICNFezbRc24YVffHoOLuIBgWE3uxVJmXvu4gpdFZm2HqSGDo7bmhflzL4SxJcwwBiGlaI8oT8ZWMg+IjF95i6PKlZqBEVxBCIMaw1qsMk9ETywDCASNDMWOvKtsa+IbaMmruIFOdlbJwMdCZNYCBDhsZxtpYerG+oYWvXrWzvmXhpvunCgGEAujBL9cIiP17JlpFx7NJyXeWDohfSARk778qMsrVm5yEoFr3LGQbgjSNfOWwr/eVUcLGvwiazCAOd9GK1kQxE6An+3DCA4LgrKLnk6rZlYxn2OULosoIX5odSDMBeUDHO7th0c/tapRWXaWWGAYQYeIj4oxl2Oatgy41OHwKRQYrCZjDG1lSOs9VGRQiCwGwZwwAC4O68669eNjwwYFb7ALjTUcRIBcGxahiAD9zR1t1yyn4d/TXTn0nJw0AXgbSKthTXJA+0ZEJkGIDHuMCo1z/E2sYh6putOw9sJeQ1qQcZUg3qa1i72U50HxPDAAT4MYQvQEyaHhtG4DlahgEUocgQfhFCSuGnYQTCUTQMwIaaJde0rTSivg0hpfY1xwg23dS+stS6FrQ/hgEQ5oxxL+j0SW25LoLcGAsJCWXNAHACb2SUXWfcdFNLyKEAh7txVSVbVc4nE8uSAeTcdW+j2bM81AwyhUsFA2vIzfiKctwxKDsGwMX9DLvNbOmVCu0q6gfZB2hOXFFuPgRlwwDgtjuSYbeXk7hfUVnJqmprCyikZurUgt/FP4YOHix4NDI4SLEFRwuelfIPrhaMsxXl4l5cFgygVK37IO7KmhpWXVfHGMXeqq6nT0rVDQ1aaHS4r4/XO9w/QBFHxxi5Q7PRoSEGJlFSqYx2C0qaAZTKqm+t5Fi9K+uyRF+8ssdNgGACYAajA4MMUkQpSA7lIA2ULAM49+ttbRRM87o06vpY1WtoFcdKXkWretKIXZbZgAmMkLQAyWGI/sAgUpdIGqDgpqsev7G9PXWwSwBccgwAFv6DQ6Trp+xc/pTp0znBT5kxnYv1EmOXuixgAId693OGcGj//lTBjxOHU2vYilLbKSgpBoB9fYqqe2caVn2I9SD6GiJ4fJZjAhMYIoaAz1QYGkkaoKjGl5WS30DJMABu6Mse1U0sLRmiFw9NmpgBEc2qUnEnTj0DSIPIjxUef7Uzm8QUYN7kMTC4r5tLBUlWE0pFJUg1A0BsfZo1t9MfPhOVYMira2piU5oaS1an141wbjPo7mED3d1JNSB2Eg5WpPlOg9QyAAThHK8g4k9YkA5Y7OtnzzarvWLuAKmgf8+e5PkcwGdgjK1Ia5DSVDIAvsU3Tu68CUrYtmuYO0ebE06CuhorKNhS7HtnF99WjBWQosbphqMr0rhVmDoGQGI/RP7lRfiP7SfE/Po5hxsxP+IRgHrQv2s3Vw8ibtqtuTWkDqxwy5C0d6lhAEkz9mHFn7ZgviH8mGc0GMGBbdsTIxGkzTiYCgaQO777EM212I19RtSPmeIFzSdMNeik48VL0+A0lHgGkBTih1V/6rwjytZpR0B3iXuMrcODO3YmYdcgFUwg0Qwgt813J82y5rhmGpx3YNWHnm9SejAA+wB2DWL2MEz8NmFiGQAn/gx7KM5tPjjvYNXH6m9S+jAA+wCkgVgdimibkObw0qT6CiSSAcRN/Fj1YeBLm49+hkI8khGKp+KBtZ7zl9YPZKLvFCSloBwdg81Gi7Tn4wUnB5G0ZxnP15LLnJAPMAAYCmOTBhLMBIrnSexDFjfxQ9xvIHE/Q0wgmQlkniPYHKHy3xYlxgx0npnYYeMwxQvgOEU16supBbGgKKFMIFEMIE7iT+Kqz1f0HN0UEFYsMzhkozTT7Iwq25/omUKs0kACmUBiGEDO2v8MTbPmkFPNd3GI+tNJ5I9z1c+L70QTnDiipw3feFNRIN9Xi0FEoEZAGthPKkFMtoFE7Q4kggHEudU39YgjWP1hs1XM5QB1EPpB6NYolAnRCxFlxwP/rhch/e/uYQd37hSCo/FFYpiAhXKNfXWvOi7ih2V/RvPRkYbbymrvWXxwQ5s7asxbwgAkBCvpMDIibFlv19Y4/AYSwQQo3Fm8CeG7CIJIPfwg8s88/rhIiB9Eb/0DpkH4hvjl55wdXxYe8akq4fQm5kIMOz6Lc3NfVVcC1ROrqZuMfojd9+lAkAcsNHXOHDZt/pEsQ2G09SZM0uxUNQSvFtNZqSCLXxU1Yy7UNjby0ULw0qgS9WDRkRe0NO/Y0HFXVG0WtxMbA8CRXloMryoGSNdvbuU/8khWp1Hft1YortTrVV91oSl99RI3sP6FBb56agOrrK7hQUvHo+Paixdc2NL79oaOjrDwBykPNhp54sE8MhS8M6IE4m889hhtIj8mIJ8vwKYh/IhGtaiZHO75rkLIQYBdoOeNNyN1HMqMs8viCCoSOQOIeq8fOh6MfardeS2it4xU0S0YRRPf/CzAgH08wjADuBHDOAhmEEmKyUdAtyJcgDtY/OlBZGG8QPxNtPJrIf5cz0D4hvgLhjnWH/bxgDAGRh0kYc5g7mAORZKyoe1uz9FIJE2ikUgZQJQWf5zbxwCqcu7hE2mcJhP9cYI3on5kkzRwQxZzzo2bX2aAuYM5hLkUUYp8ZyAyIyDi9hMSvxgFIhGmC2K/Ckt/nvCjANy0EQEGaEQtPUGiNb5DQOHcx4aGI1EHaIlZNP+ClgwZBddJgBc6SzD5yGez/MaeMTraG0EC8eMkn5qUXTOMiK8Gm4mphTMAiHD+xDicKESI8ihSZQVbGsUNRNoZAPf0G2ZbCNfQ/7UmdcQPUV8rqKbypGCAU4D8YEfGBMgo2FDNFuoOK6bdBsD1/pQQf17cl58PSZnGBo6gGMBYk40gK+t5VwLpEguN9kQ0E4WnoFYbAJx9CFH405pgpIHOHyphEmiXh0JBaAprxEBWK6AJIDEHcIPz8EG67nx4WCNEHJRFup2EJLobrI8tV7U1j1ayZ3SL/tZWX1BrPzi/0fGDjXGplgIz8Dp4hCPF3eQspN1PgFSBylF2Rsct7V068K1NBRjJ6N/vN8SvY0qYOrEgeKkE1hahdj8BUgU4LWkaFi0qAN/yy7DlmmDm1VruvRXV1b6b4YOLvWGTDAZcMQC1UDxPsEVYM20aO0QXmOo8O0AgNOvaGhT3zhUx4pdRiP4W8QfjviB/iP3G0iceRfPGwgAYQFYdEM+XSM4OaFIFlKsAUYj+iOITiPi5Rxjpd4b4rfltPj0wwOcKaN9FYsRcxJzUmjSpAkpVADros5wYplarP87z+z3SmxVzlAs7WsfbVJ5ADNDkFs2iqro6LlnqjCdAzTdT/ICtFD+gUxV2RP3xXX8UDj+I2uJ3u88ifrPo+x5SU8AJA5wJjAn9xHCCUGuwUcUOQspUgL4hdpvOLT+czkLkXn8J5G+2+fzhzOR2xQBfSbLzyikf5qjq06cF7ZAqwGmt4GHwH0oYAHz9CYTlwcHwLskP9/i5rCOns5mV3xu3Joc/DOTnlINdANuDfqVUf63z3MtzNBegaGERJQxgZJRdV1it2l/+jX45zUZsuFULoKmt/DBgzS0HJhCFUVAVzYU2Auo2/EHvn3bkPPkJRgNCappJBgORYIDPNTCBojlX3VDPRgYG2eihQ1rgoHaVGARVSADaVn/s9/vR+y0bbV5E04J6U6nBwAQGrLlmzb2JN4zPXcxhjSk07YWCLhfkY5muDk4/+iiG7RW5RCzYEsvkCphcBgNqMVAkesJTsLJ2CjvU06u2nYnaGsN6CBYJLhM1e33Tve2HW3qnzpN0rnDQw7zgN+8NBrRhgEL82tPBHTtZ/5499kfqvofcFgysAvQPkcMPbUmo68lETRCbcEW3VDI6vxSaTKZoMJC3Cdiaw1zWpgoQDXJatLXn52sgCUD36o9tFJmrmqB3cV5byHD99N/kNRhQjwGiKhCW/UgxnIPgJKQlhZACAkkAOld/EL4M8XMHH2DTEL+WOWUqDYEBmpPZaTmxvsrP6wDthpACJiCUbFfn6g8xqYkuavT2pCKwDeFLjpjJFisGOIVlJysuG+ne/LqeG4cCSgG+JQCdqz8Mf17ED7HfN9eKdQaYxssZA5ir1ozF3MYc15ICSgG+GQDZ3C7X0QGOHAnDH/Zdrb1XHXCYOg0GVGKgeL7W0xz3WuSCth+ENn0xAHj9keitxfIvt+Vn1v6gk8OUixkDtq1qubkeAF6iTU6jPor6YgBU73U+6pbOiqi+3oY/Q/zSCDUZk4cBPn2zcxhzXeN1Y75oVJoB4EpvwmqzDsw2zJ3jWm1W7yfkZW0prnnNS4OBRGKA5m5+HhOAXnM+RB+az7v+amnvXGkGMJZhnwsBlLAoOGG1xOWLJoyXEIXmRUowYJ/DmPO6pIDhgQFpWs3KJB4I5IE+K+h6Lw1p1qITPYwiZuXXgHZTZZwY4FQ3zrAtuPeVV7VAUjnGFsrcJSAlAYxqsvzjiiUvi6gUh9KCQlOpwYAeDFhzGnNf1zVjsjQrxQBYhZ5oP9gScU1kOTVbfq4YMi9TiAE+p3O7Ap40ELR/kjTryQC48U/D1h/0H7fVnxtMLFYZFAmmnMFAQjGAQ0OY46ABLbYAotmc4d4VA54MQJfxz8sKWuxA4doL89JgIGUYsM9vL1oI2jUZ2nVlAPD7J0YlvaUgCyhiprlb/s3SL4tLky/tGMhwWgh00Y1H10G7oGG3bK4MgMIPKyd+AKPNH9qtp+adwUCCMaCLJrxo2JUBEL6U+/1D56md2SQciqyzhPC1eWEwUGIYyM540ISbTSxEp11pWMgAsPdPjS4O0bBjUZltD2P5d0SdeViKGLBNdhnaCICCxTladiwqZADjlXrE/ylNYpUEvNAkg4FyxADmvhtthMGJGy0LGcDoGLs0TKNOZXEIwk3MsVtGncqbZwYDpYgBa96DNrwPxfnHgBstOzIAbv3PsFb/TbmX0NE59xbNW4OBdGFAB42Qz0GraDfAkQF4WQ6DoBThvryMf0HqNWUMBkoFA1ADQCM6IgiLaNqRARBCL1KNVC/OZrOFqG7a1GcwkAoMWDTgRSsBO+NI084MIKPeAFgzY7oQbmP8E6LGvCgjDFjuwW60EhgdApqexAAopNBi1WG/INK4cbXxwL0yBQ0GSgcD3BhI3QGtKFcDsuHCJm3rT2IAOlx/3YifD5/hAKUzi01PwmEgRwueNBOgFSfansQAxsbV6/+eIo3Z/g8wnKZISWIgRwueNBOg8060PYkBYMsgQN2uRdy4Ge+vkQBc8WdelhEGiBZAE240ExQbTrRdwABarm1rDVq5qJxXR8Zt4ZJFdZjnBgPlhAGLJrxoJwhOimm8gAGQEWKSkSBII/Yy7sd+KacR/+3oMt8NBvI04Uk7AXBVTOMFDGBUg/4/xWX7j8NvxP8Aw2iKlDQGLEOgF+0EQEIxjVfZ66DFWKkEAN9mN99/vv9PjVoOEHZYZL6fdtRCdtGRRzlm/dWrL7Dd+/Y6vlPxMEMK1Z+ddhabW9/gWt3bBw+wO194OnAfXSuP4OUHFr2HndQ0S3lLozTo+4cOsZ5Dg/R3iHUPDrDNe3axocFB5W2lrUL4A0AMsOgH0YNVpWIazzMAHvqbsWZVDaEemVhnQYkf9R83o4ldfPQx+Dop3f/WFrab6WEAIP6vnHcR+yAxILf01oH97J+ffTK1xI++nXnYHHahgMm69T3IOyx8L+/bwzbufJs9tPUN1ru/N0g1qS8DmsgygSwNDShkAIScZpwLWLeyvQeImlABqtQSPyr30mHCED/qjyPJEv+W/T3sygf/wPr7++IAM5Vt0urETp45m33hlNPZLz58GfuLM89l1SRFlmOyaMOLhoLgZnBkQtLPM4CxMdYapDK3MlX1dW6v88YO90zJeStL/Jt7utnfPvhHNjgwkBzgUwZJJS2Bnz7hZHb7JZ9gx8ydlzLoFYALbkjJk4ay2Xz9b6f1CQbA2Om+avHIDFdGHYEOPZrV9lqW+F8mu8PXHvqj0WUVjUTjlFr27Qvez+aTKlKOCTSk2i14zEbreQZADKdZJYKliD8lOwCyxP/83nfZ1Q/dw4bJqGWSOgzUVlax71z4ATa7aaa6SpNek402pGjJR3/stJ43AlJ5pTsANVOneoIEQ4el63hmjimDLPE/vXsXW/XwfWx0ZCQmSKNv9qpHH2K7Du730XCG1VVXs4bqGv43Y8oUdsqsw9jZh89lM2vd1cWpVOarZ7Swb5JdpRwSp41cR0FLQ31KbUl5WucMgJ8AVIzVyrpazxpLhfg3vbOD3fjIA2VF/Bjct3t7WDcZO8OkB3KF55Dx78tnLGFnuIj6Zx4+hx1FzOKt3e+EaTIVZTlt0AKJJENL2Zzy/4Pmn7ipvZOrAJUVrFG+qFxOt/1/uRrizSW78j+6Yzu7YcN/lR3xqx6dXbT9940H/sB+8vwzrlUvPyW/eLnmK6WXOmjJonnOAOxWQVWIU623qIJLph5Z4l+3/S12C638Y6OjMtWaPBIYuPvFTvbbNzcLcy6ZcwSr93C+EhZO6QsdtGTRPGcAJG7MUIkbWYAtZweVbYetS5b47yNHo+9ufJCNEyZNUouBXzz/FIOnoCjNmaZ0uoqaifV5MW3I0pQs0BbNZxmAYgOgDpFFtmNh8skS/x+63mD/+Nh6In7xJA0DR7mXhf8EVCtRmj9NHF5OVCbtz1XTFM1crktxBqAaOdV17hZdqz0XJm9liexTlvjXvvEa+/HjD9PuhSF+nYPzRm+3sPoFZcAAiqeXLE0JkSZ4wXcBSNxoFbwP9rhCC18JBotEKVni//Xml9kvnu6gvUuJSk2WUBjY7eJCPYX8AlQmjP8C2n04ng49YTsSf7PIAWeY1Lt9dEhpHx1Q2tF3gD33zttsoL9fZdPydSmmKYvm1WIy151qLxdg+W5rzylL/L985QX2H51PKIPn1AXNbCn9FacRmnQ/6ni4+LHv34uPOoZdOH/ySckdfQfZ/6MDSklPbjx26wE1h4ROOGI++wAdJnvvvPlsRs0UKZRAMnn47W3sd6+9GKm3py6aqsqdApTqvOpMxIVidQSSJf6fv/gs+8/nn1ba/WPoJKPTacIh2lFQwQCObXSu/9XuvcQAlHZFS2WH1dUL630rpO/BTML9V+ig0Tm0o+A3HUtl8YdzCv/x2kvst688z4bVntbLggTi0CxqgvarGE4BKjZky55gKtZz/A5GmPyyxP8TIvy7iQGYFC0GFs4Qu6bsDHpMmGjqk6edzT5LMQ4qOIEF71NdVRVbcfJp7KMLj2PXkEfkjj27g1fmVBLEAR6QS7I0ZeWX+iTaj1dZt3VQCmBFmWSJfzWJ/Ib4FSHdRzU4AnwBieVOqZeCiPT1H3R65foMY/6lJRewz510amjitzc0mySVH77vw+z8Y0+0Pw7/PSLaqBgbVesFqPrkUnhMFtYgQ/zQP7/z9OPsftL7TYoeA5eedBqrrqh0bPhnL3T6lowzZEC78r1L2SW0WnulMVp59wz0s1e795Hh7yAZAr2dvKqp/mvOOZ8hepLOpJq2QPtVJGYo9a305bDgZunRgEkZ4scEuOXJx9jGN17VAIGp0gsDp8w/mq/STvneoLMH62knxm/6bxS6bSnVK0oITXbv1jfZfeTf8Q65JBds8dJKXEu7AkvmN7OPHXM8W+QSHu2rp5/NtlAsiDdotyB0Am0USQGgLaWHgoj2tewCyHY+SiOgLPFfv+kR9sSW12W7YPIpwkAFOad/8tSz2GdOPKV43udbaKct2ALizL8Rf0EcAdQpSht3bmffeXyD2KJPhAjHpIeJ8eDvJAqPdl3LBQynE4sT7Ao3nN/K/ud9v2UHKBZkmMRpI0wFkmVjZQCAMQomIEP8gGU3iX5Pb+vCV5M0YwBi+RQ6DnzCbIo5SKsztuKmORCVBcZPX3iGbdm10/op9VlJhrpvnHuBUOf/1Wsvs395xp9fx8tvv0UE/jt264V/ypwckqaR/eLzJHGs3rhOCkanTKCJqFKsDGAc2xyaLwaRJX4gHBF+v3DWeeyfaEUwyRsDN/zJ+9iQz7MQ9USUiAPgtII6tQgd/B+eejyQSvb+405iR06d5lQtQ/yGX3Y+7tuegMr2U7DXb2x8iP3sTz9KtorJdvT3k3/H7eStiHxBEt8dy0AH0J+UMwCZQCAF3QK309RXP8RvwYRtncdILHyODvuY5I6Bo6frPZTTQbr095/qCERIWEU/efwixw7AznPr4+HOcuwhIyGkkr+h1b44QRX4NBkyf0rqZKAkkAA0BAaxRQUOBKmCQpqIH0ElZUJ3O/XgarLoltuRUyc8xPlscHSEdVCglT5Sy4KkRfOOYkc0OEeleoQOGvUpiLBzD+0SHRx2jtn/IfIwhF0jUNJEE06wBITQqapgzzKaRJ2vLj7H0dNOBkroolefd1E+NrtMGZNHLQYQB7CNxvCXH/sU+wStstWkNvhJy1z25de+/oqfqoR5cRQcbsFOqYaC4s6fdbjTK89numjCqeEKYjZHO72I6hlndgKRJwwMIt0PdWLb5/rHH3Hd40VoqktOPj0MCKasAgzAqAaPu599aBlrlnTdhfi/WBBabCft7W9+Z7sCyLJVPLhti7Cu0w+bK3wnfEGwc5oQZlD7oqq6gU499cV8HVOEPe6mk11t6+5le3v2sR/SqoJVRpT+6j1nsKdIDN1J0X5NmowBEJNfIyBqmUKGs+mE+/qq6smVCp7A4+77rReT3t1J3pnkDOSS4OvfQMFHnRKOc3Mjm9PLAM9epVuM+oaHHds7g+IX/u5Fn5WCFjQsiCIoqkb6BreKXkb2HB2OgAm8S/pkG4Xttq6c+i+6P/CCeQvYWTRQTgl2hJWkCvz1vXexURpkkwox8LX194cKCpqpyNDNP1PYEdMb2Zlz5rEldAHIKTNnC7ftYFz74qln8HsEH3Vx1FrkInpvIgOvygQ1YDvt+Z/oELL86CBxCyIkfuAhdhsAgECfIbbpTFitvkpBJy3i520R07mVjt6KDDnIA1Xif9HWoEnqMYCISrgMdCtF+b2TDl1dff/dbPk9d5Hxz92T7n+f1cIOnym+sPRkCjUuSr0BzhGI6rKedx9yvgFquuQRY6se0IBmMrCayn8mggHAH0ClWJbvXe4LLum8nIjfyTsLd/f9A201uaWPNB/LziCrrkn6MbCPztvfsO4+7o4taq2KVIgvnHqm6DVrotuEnNIhOmo9pOHSFgQMcUo4MQiHJ9kEGuC+MbIFFOSTh06ysaGD/k9qoWpdEgACOFxBxO92SedT5AOO24Td0t+ffR5rEGwruZUz74Jh4BGy1P/AJXDJ+RTMQyQFTBdcKLqLmL2OtJeiBokSvB1lkxcNBKUtt/aVMwC3xtzegfN5IcCtvNO7Vyj4xd/RJZ2HXAbIKvejJzcy2AhECZ5r17RgazBqIU0EkfvztMDp1ot7Xn6e4ZZlUTqP9vqd0jSB6K2LARxwCQhSI2noxLSKevUH7hLDAACMSjXgOQrQcBURv+w9fYjqcgNtDbrZIk8/7HD2Ubq6Og2pThg3Lx0MzMLxv7/6kvV10ucSMhw6paglALg2i9IhgaNQcX6Vc7+4brffiWIA4IKqFthf0Oox4tNyj2Ocv6YDIm7pC3QzzbzZwRw83OpV/Q7756WQOl2s9ic4WN7d+gznHB3JLXzZkIt0YMGict5bdcp+VtCS576pKltTLt+IwCAiU00cIlAxXP/67BOsyyXkFLaisDVYJdhnLq4vrt+lwgAGBvoYAqU6JRjZnIJkwNHLKeGwl450uCB+IYyOshfHyMz9MLTl2G+i/QoKvCJWshxLuT9Uck2WKjHAHVTHt4D/Wx3rhZMOheaRMfCLZ5/vWD4pD6dXi8XSpMAoAwdEY5GVHeWdjGz7Bavu3HrnswEycLjlEUkAIkZUUJePua6EtmyNg/YTpQIAtiwndNPEbT3Q9HX3vr3c48ytehz2OIu2B5Oa3PTSpMLsBBfoo6nWmZlhljiJ2AcEEsBsurDGz7acEzxOz0QM4E2KYOSdMOPjm+8VbIR1eQPpL8ewgpNW/lpUn/uPLz/HYEh0S18jB6GpgvPmbuV0v4N64hSsQne7OurHqUxRfEBY9UdHRiY1u1Mw/6C+NQjE9UmVSD5ooujFItvCiwpdyLXQFNF+Rcct7V2SfY0wW3wc0eokQk/dTF6C/SNiF2D4m3+d7AFwaVWVKuE4ErK648i1GW7MpZDmTmsUdmOLYIV9Ya+YcS9onCmsL8iLpXQBiyi9RPEFvVN8cx20r0UFGO4XO0Z4IySbA/M37jkMz8HvPb3JFeRTye300lPOcM3j9LJPwFhAuNUuobGc6ip+dtbhzttjxfnS8PvjgqAegF3kI/Dqu7uEXfv4cc5BQoQFPF5cLPAQBVlvcZEA/M5vFTTl1BXOAGixW+f0MvAzgdXWT31cM4qPOeZB7XjzNbb+7bfyv52+fJ6OqyL4pJ/U7eKc1EBRaAMnYpznC2LqB64zpoKzaZvvQgrCKUqvUVQepwSvz92CO/xajjiSNdHhIxVp7qzZwpBjT1H8QjcHNBg3fen+CmjK3meL5rVIAMpcFilYSNxSAJC2+olH+SWRdgTav/OtwZYLaWtQfu/djQGc4HKazd6u0/dzm49ngU6hOVUW4zNs711x5nlCbegdOtz19LY3hRDeLzinD8Xo4yeeLCzn58X7jxIbgX/75mZhVXxO+wyEo4ymiqDiDICQ0ln0PNRPldsVCRAC+Im1Gzc96oqTubQ1+NcUSkw29VCoaVF6HwWVDJIQBfdLp4kPyQSpM44yIP5rL/gAg+elKP2YTg+OjTr7B6DM7+jePsT+c0ofpt2bGhfvPacyxc9qaUfhkoXODACuwZ3bu4qL5H87Q5V/7fhFJU2hAYvmswwgw9Rct5oDXa3DQjKkgFd3bGN3upxBR9dx2eeShcfnsOD+0UvXTeMiUKd0HompEC/9JMSfu4LOKiBwRpoTbk3+8Ycv47EBRP3ABadPbX1D9Jo/P0gSwgYaM6eEcGN/e+6Foc51XH7Oe5nouO//pfsDRASblWj9swC1NMXta5zmOQMgw/M6J0SFeaYeYAhv8aY1z2xi2zxCPf/dWeeyaRJbg9i+ekTg5gpD4DeJmOvq5YiZx7+/8IOs1eE68HgxJm4dhIA4fzOmzWC4pvszdFvvDz/ycXYzXeHlFs4NNf4A17RL0NAdFDloVCAF4B6Cz5Mzl98dHByy+tTic3ggGafeIe7E2pecL5MNekBLNS0BbovmeVhwkqRkPBac+it8NkpikK9rwoQ12Ywl4AESA+9SVahXINpv0dbgP33gEmHUGoS5uva8VnbVA3eTG6g7sPfSMeT3Ca6sOoqiyfzs4ktZOzEdXFbitKJMIWPhh48/if3ZcSeyxqIz8JiIoqi4oZBgK3z7hy61/fL3FXYT/PlN/0hHhGUvCIFDF+4SRBQhp3TZsSewkymwyC0UvvtdgUHRXg6M/Upa+d2uFf8u3V7kNFaQubNGP/c5YW/P+g5aUp0smucM4Imb2jvPuaZNaRujAxQkYfp0hXUCcf4njEIAeFWID/jPdF34X71nsbDqU0h8//h7zmS/ee4pYR68eHnnNgpvNUiebs4BLEDUK8m42EcOR9hT3kM7B8OkNsDecAQ5yIDAERyjOOFyy+seW89+SoxKZ3JqW2d77c88wRDGzU/6Pa3GWO2xXeuUTqS7/m4nRgv8PkiMdjMxgp7BftZPgUPq6FhxE6lU86ZNI/XuGB5o1G0G/oEY+iuCKMHZtv0TP8pxWnICPsQz0DyKcwaQqwcPxLM6l0n2A1bL+jliI45sPYX5ksEEfvtSJ99qQ/w6UcI11E/S6UKEuxIlGLFWkkTR3vpBV9YGhyO3VcdeP+Lp//2GB1jfIecoNfa8afmOPq0m4t8QIJw3HLq+9ciD7NtLL2YLXbb/TqaxxF/Q9ND2rezHmza4FA9G/KhQww4AJ37UnV8+CLwuPFCVdOgtgA1SYwDJUVW3eD0Q7W967GGGiSlKEG+/Sas37rp3SziCfDtJFCoSptiqjg1sF61mQfVNFXCoqgNW/LvpotbP3v2bQMRvwTFAEtGVFBXqRSnPPKuU/CcuGvnexnWOKp+K+aqaluy0nmcA9EXNLMzhDXqQasBRNXcQyrUR50fvgV62GsYolzSHxPQvk87olf7zhafZ90jXF+0KeJXHe0Q/+vKD97DnSYxFqnBQDfiLFPyHyEzYR/8fdAnnT+ieRhBw2ITAMNdQgBgwlOBrcSEUcBPHHLj1kQeEx37Rli+Hn8ImOA052hSK8vn5aaf1vAoAqyBJpNf5qcgr7wi5BKsyBBa2RWhVwVoLK/X9a8PmV8gafBQ7n7btRAkXRT52zAkMHoXCRN158NUX2TMkDXzlzBZ2NvnyyxrI3iGvtx+TrYFvi2G25ZJseSt/1J+49LOHiLKbVBV89tDny7RCP0Vx9hEYVEeCERcM5U7apvvqGee6+hm4tQ/J5J6tb7KfU+wIXB3umPLGAtugOGZ0fwgaUp2sHQDUmwez9babm/ve3bVFZWN1TU1s2oL5KqssqCtD4HP0hsNxQZ1J+DGFjILnzG9mLXOPZIfRVuAsMgbOJIv/yPgY27p/P9tK0geClmyleHkv0l636hUiCTiIAoY5pPO3zFtA7sYL2AlkDMwTg0PjByi012PEnB4lfD9H27euoeagplIdYVZ+C4QD27azgW61DLGhhjWtW9negzYK+kw7AWAAzXihIlWS/jtr0YkqqhLXQdeLQxgQbPeKy5k3BgM2DFSSoXVqbT2bTsy3MXcWo4d2XQ6QZAKD6jBiDEgsNPm56NPV1wZKwde9r7zKFG8DdtEOwEKrkbwKgAfUv06ipWbrZdhPAI4/MAJtiRA9TkzAJIOBMBjAzU+9w70Mth1n/0G52vlCpIj4LfqRa1kuF2jcnjNvBMTDygxbb3+p4vuh3v0qqnGvAwg3PMAdR+atfgxgDioifgCrg3aKabyAAZD4UsAdVGBMSyQTR8CScWbAETTzsOQxANFfSkfwgQkdtFNM4wUMoOOG9nU+4JPKeoiMVlEly+iSHYyoWjXtlDMGrLlmzT2VuNBBO8U0XsAAADzpMOtUdgJ16eiIEEZuExC+NS8MBpRiQKXObwdMB8040fYkBkDh7ZTbAYaisAPYsWfpYcYuYMeK+a4SA9bcsuaayrqpLh0040TbkxgAmdPWKu5LtBKABXxuYCwRzXpsPg0GwmIgP6c0ET/g0yIBOND2JAbATwll1B4PhqOKjg55DySxM2xu5kfMu4TJYTDgigE+l3LzyjVj8JegFeXOXUTT1glAO2STGAB/Oa5eCtAh0tg7IvqOocowCh1liWyijOa5wYAXBmgOYS5hTulMWmhFQNPODICptwPEIwFkhyk7YLqHTeeUMHUnAwM6bP2Te6aJVtZPbsl2HNj+knyFldsBINIM7lPr02yHWeo7dDZwcaMSSKHLZMrNFUiPGvV9O55BI8rFf2pARNOOEgAOCjhtGdgBDfJdE2fzCQokgSwj8FnQZC83DIDwucAfnfSog0ZAy9bhn+IhdGQAyERBZu8qzhz2Nzqn+GBDIJCygpzxHAyEvDIplBUSMVOiI37Qhg4G4EbLQgaQGVWvBmDuHOruScwUwuAabSAxw5EYQDAnoiR8q+O6aMONloUMIHdpqPKzAf17ZC5MtFCi/5MPtPEZ0I/ohLeQXwhoLsRB/ECPJtrodLsAWMgAcuO1WvW4JcIY6NSpiIw8Tk2bZwnBQIxzQJfxjzDrSsOuDEBkOQw7XJo4XViwspwfk4Abf0JXZypIAwZyFv64Vn0LRbpowouGXRkA3w1wcB+0gA76iWChOo46BoVncjkwgSwjyIuGkzOZJynFAB/THOFnrfzxdgS0oCeALlsrsv5bPXZlAMhUMc7usDKr/Ox7R3yHu8p2wtWVswAbiSAcGpNUOj+WubFNAGy6aEGGdj0ZwKab29eSSKzcdD+kieupHs+saGh2C1TjNY76stJcfEY+pz5j5QctKE9Es5x2PSr2ZAC8/Bhb41FPoNcD7yZrR8CtE5wRcLUgxwzyK4lbKfMuVgzQGGXF/axKF7ee74QLbTQgSbNSDKBy3N2S6NQxmWcId6xD95FpO0weTCSL/jHBsitLmBpNWVUYsI8HxiiJRG/1FXNfdchvq25ZmpViANhHJB6q/HwAgD349g4L5lR9WhJBwQSzuEKqelIiwNpw7zg2CeymrrkPWnXb+7ejQooBoEB1Xd0d9oKqvkP/SfaOgHdPrQmHOZgVOb3LmByKMEBI56s+qiMVrYAhK2pCRzWY81p0fwK2qkJeYpdmAI9942ZIAF06kKHLCqoDVrc6s9MP1mXonPSRm5xuZcw7/xjIM1ngmJN8egjf6q3GOd9VHPjTatPpU5oB5Aqvcqok7DNwQh2HIMLCFa58jhFQJZZeyiduuErLtvRkHE7gN21IwVzXtfoTLnzRKOehfhB4ztfbuonpNvopI5M3kmvEZADRmAd3GSLxSLIa2ym1qi3GmRbx3gv/Gq77yjaJsF83tjd5tW9/71cCgJrl6ltsr9zPdxyF7N+120+R1OXNqwi57cSJrQTqim9WnLruewMMHFh44N+hSllYw4qf/oQ5rutIfBDa9M0A6mtYOw2ScscgDC38oXUhJ5lTJzvBocfyeZ+b/NaKl0yY1UKV7yv1Pdv9CZyobSn+2vgip+s0LNEkp02f3az0mZ91resYnH9BSx2Va/Vb1iv/OMnGY0PDrLZRuYbh1XRy3ueYAKjBHrqME4r1LjnQykPC+zORPduf3Kqe5n5NdMnzG6761uX3Qii89dHr2+/xBKIog28JAOV1SgEwkJSeQbAI65I/uc6bUxey+i9WRzAGqsAiGutTss5Islkw0ecErFlRPnvIKvc9EmCS0YjWeR1w9QdmAjEAnDAKom/IDgU45TgFETXJCQMW8dhEZRoMEJpFbNZ3/puqsH5zpmERJ3+e/WGVyzMX2zsOAbLl6rfqsn/a33EC59ugWN1tsPKKyvM/zGXMaV0JtOh16k/UdiAGgMp0SgEIGrJfI8JEyEjz8yypTRBd/jfNDusfCJL/0bOscQ33JWTfWu/wy/4uT9D5WqzaJj4tQudl04xETbBjLuuI9MvBDbH6o3xgBqBbCtAqMmkaaFOtwUAxBnTP4zCrP2C1CYTFoMv9Pueati2Us1kut79cFZWVbNaiE1mGPk0yGEgbBiD6Y89f2+pPnrl03dfCMHgJLAHYGl1l+670q1EFlKLTVBYxBrSK/tm+hKa90Evrjg0dnfP+pKWVjELNOvA7eugQy1RUsuqGeh3VmzoNBrRgoJ9iXQzs3aulblSKyz6evLn9irANqJAAWFWlP/9jv0Af3LlT2/6pX1hMfoMBLwxgrx9zVmdSRXNKGEDu9NEanR3u7dpqtgZ1ItjUrQQD0PsxVzWnNX5O/LnBooQBoAEKP3wFmRS1uAijfrhRRoBYNGWSwUBgDGCOanVnJxrjtBYYwsKCoW0AVnVwET7yghaE+l1mPVP9OTo8TFLAGKuZNk111aY+g4HQGDi4Yycb7O0NXY9HBV/aeH17h0ce6dcZ6ZySGc++uu0hMgi2SmYPlG36/PmsdqavU4+B2jGFDAZkMYCbffZv1+ftBzhyhr+lsjDJ5KuSyeQnT9U4WzFawZ4hxzFtJ3qA6Kr6OlZVW+sHtLLJaw+xhq3U0YHBQH2vqK5mlVNq8mWrGxry382XCQzA6Keb+KFeV42xFROtqvmmXAIAWEuuaVtJDqXXqQHRuRY4CTUee0zZMQHol2OkCg33DzA2NsaGBwa4cRTqkVbd0zYMCN5SScwBDloUK5L8SStYNTFkzjDoXTklEH/PG2/qdPbh6CRCXbXppvaVqnGrhQEAyChUAUgATcQEStVTECs5CH2UJhmIW2MYKaXzqoYkBc4kaHzAGEpVcoDFv5uIX9cRX2tQdIj+Vt3KVYB8xRGoAkA8BqAUmAD6MkLEzomeVnXdk8oaJx2fnFER87InMGtIC2AGpaC+RUX8ukR/a2y0SQBo4Nyvt7WNjbPbrMZ0faZREsCKfqh3f5bgiVg0+ovrQnuoeqHCgRngb8qM6VxiCFVhhIUjI37qU0WGXfH4je3turqnlQEA6LOvabuTGtG2NWghpq6piU1bMN/6mchPnAwbIqLHChmVvp5IRDgABZUBqkMNMYMp06c75EjOI5zt13Wjj72XZEdb++RN7ZfZn6n+rk0FsACdWsNW9A3TtqDGXQG0ZQ1IkpgAVgqs8rqPhFq4TvMnGOIA/ui6OCQwAf5HDCFJNp6oiB+i/9Rq9Vb/4jmiXQJAgy3XtrWS/85DxY3r+B23JGCIXv2oJoUZREb8hMLKCrZUlbuv24go8wR0a2T7wx1dFEgUzKbVLZ+KdzCeIbBozdQGOkWozNPZEzQY7xDy+QDddQhvMJxiNEkNBoBLSFGD+/ZxvFbQSRioDFElMHXc42dJJ7rbJUJZRXr/Gt3toP5IJACrI1HZA9BeFIZBTIwB8gDDsU+j01ujHM0nGEDdrFmsjjxCdaoIGOMotvosrEWh91tt4VO7DcDeGLcHDLFmerbY/lzHd51bhFjt4foZ1YqgAz9prxMMF0du8Qe1D67hqv0NoiZ+GpNO0EiUYxOpBICOUQixxSR3PKTbKGghEZLA9KMWKPEYBNEPkpEqLQ45Fg7K5RO7CLU5ZhC2z1hA9r+1LTp/DJykHWdLKcRXZ1jY/ZSPnAEAuCVXty0bz7A7/QAaJm9Yt2EQft9ufVc6hembKTsZA1APGg4/PPCBMRB/FO69dsgpuOdlm25uX2t/FsX3SIyAxR15+5GOVxZc2NJL+s6Hit/p+I0bhwb27iP/9RpWBd91icT1ewrrhEMeWPXx26R0YABjxY2G3T0USGKMxrxW2iAMZt+zpYtO3tHsjCjB2Yf8/NdE1FxBM7FIABYEpA7cTt+XW7+j+KyfPZtNnXeEa1OI59ZPK37avPOmHnEkO5r+mptmTupf58632V76G+zeN+ldqT+ABFhPEkH9YbNdu4rz/LifMuK0hsT+SPV+e/9iZQAAJMqdAavj0BVnNB89yXqcRlEfRP+X51/Ilp18Kmus9ZZuuogBrNvyOvu3p5+gle51CyX5z8aFx7H7vvA3+d9hviz5euiYlWGan1RWpBpAYkAkn6htO1Fb/CchhB5EugvgBECUOwNW+xjofZtf50wARkJY9RHCOU1bebW0yl/5kWWc8K1+yXxCOljetIQtP3MJZwTf/M2/l41UgPGFSgd7znRyG8euAfR97WG8nAcmcou/ExixMwDcMNS6sm1p3xD3FNS+PWghAZNh32ub+e4AJkGaElbpX//F56VWfLd+LSbpoRxVAow99vbB/GMa+06K67c06H1+bmPq9110rnIukAERQAhliXQLBCDFNAFcsOH+6hRauSGiy4j77jXRSZOXnvfKUtLvYxr7xBA/Bjd2CcCaYWACZBRcEaWPgNV2Wj6h7992yTIhuD2DA5yon925g71IBj8rvfeY49jpc+exVvq0M45fbHzYymI+o8BAdq9/RRJWfqu7iWEAAAhOEMQElhomYA1P4ef3PvHfCwjY/rZ943r26wfuJYmGQoUVpd+Tse/39OzbZCQ8kYyF173vYp7joI1JFBWZ9POD/+eHjkbDSRnNA2cMxOTo4wzMxNNEMQCAZZjAxODYv0H0h87ulFaQIe/Fpzc5vSp4BuaAfH9Ooj+kCZMiwkBCiR+9T4QNoHgYuDskuUWSJECeHCYBA9aqXYyNK36/Vor47eXACJy2AO15zHdFGEgw8aOHiWQAACzPBGIwDKL9JCWs1k7OPdjPf5REf5MSi4HOOPz7/WAjsQwAnQATiGt3wA8Sdee9iPR2p3QHOfOYlFgM8LnLF7LEgphgCcDCWZxbhBYMcX9etPBYRxBk9H7HguahbgwkaqvPrbOJMwI6AQsmAGehg0PsdvJdFu+DORUugWfNjZN9+yH+R5lOITXkRYkGB3v2laVzkYUauPfCuzVJW30WbE6fqWAAADyH0MviOEDkhLgonznp/z10d0CUyc3/wA7HqgfvZb9/4B77o3L6voai+K5IU4cTbQNwQiTpVCtwfNLpXTk9e/adHeXU3cT3FXMSczPxgBYBmDoGAPhxUQICKJTzNuEM8mM3KQEYoG0+zEWdl3fo7GUqGQAQwqOnwFegTLcJRU5BOieLqXsSBvg2XxyRfCZBEvBBamwATv3DFks5GAcRzKOY4J0Mg044UvUMMOCsgVdCvIFySGkz9onGJNUMAJ2yjINRXEkuQqLu5yCqSQyAzvXDQciPP38YOL9GHofGezCLQQossqpj5bdXhsFnUsqmVgUoRiDuTsdtKqVoF1i/5Y3i7vLfHzvzHMfn5qEmDJC+z2/sKRHiB5ZKhgGgM7hKqaGaLYR4ht+lkp562fncPqL6IDKQSfoxgDmFuRXFdV36ezPRQkkxAHQLKgFuVOVbhSVymAhRe5wcf3C2//uf+TxFtvGOBTgx5OabLwzQHMJcwpxKi3OPn/6VHAOwOo9tmcpRdgZFd15nPUvz50101t8pwTbwI4oQ5IcJ8KhC37jJVxmntkv9GeYO5lBat/hkxieWewFkAFORZ/sjHT07Hum4g19MmuHXkaV28xwutocRsS86bM4k1MydNp194tzz2b7qaraN8jkFBYGqcC6pDN/91F/ygKC1VdXsV5tfYajXnpDvs5SvOP0LHTwqzlucp2R+Y2+fsVufuLl9BeZQyfTLoSPUz/JILVe1NY9k6CxBhrWmtcdY5bHaF+8IFPcHuwZdNsLGlqGTO7GT264oLHi5RATCql81zlZ03NLeVYzXUvyd+m1A2UHJDehSOkuwnHYKbovqbkJZ+GTyYWW/kqL//Jz0fieCturAO7f3Vj7ECUSoMJMIA7AXjZOuf3P7mnLCR8naAESDSM5Da2DNpfepHGjs+//5D77jaBQU9Vn03EuSEJUrwed8TmBulGDfXLtU0jYAUc+71nUM7tjQcddRF7WsHxtjzaQWNIvyJvH52MgIu5908o093VwdaJS879DeF+wqwLmnWK8vJxsAF/cr2YpNN7avxpyw46dcvpeNDcBtQLlaQGH3KE+zW76kvoPe/pGT38OZQSt9d0qwC8CdF05F8CsQXQgCBvB+ByPgAxRMVFTGqb2EP+si+FaV44pfPC6GAdgwwt2JM+zyNNoHbN3gX+EmDKOhcd+1YQbW/XG2Gl6jtqdl/dUwgKLhp8NFjf1DrG28RBhBUffK82eO8OtrWHspOvOEGVTDAATYM4xAgJg0PTaE7zlahgF4oMgwAg8EJfG1IXzpUTEMQBpVjKXdWOijq2nN2kWAG+Oej9EzDMAHsqysLde2tY6MscsJeWUXodjCQZI+cVKvqoKtLrWTelHg2DCAEFiGe/EojIUVbHkp7ByEQEX0RUnMZ2NsTSVZ9cvFbVcHkg0DUITVJVe3LRvLsM8ZqUARQgXVYLWvGGd3pDkOn6BrsTw2DEAx2mE07BviqsHlVPVixdWXa3Wd1PHVdE3cWrONp3YKGAagFp8FtUFFGK9ky0bH2KVpPoVY0KmIfvCz+BXsrswoW2tEfH1INwxAH24LarZJBhfRybNlxmZQgB7rNN5aerrerPRFuNH40zAAjch1q5q2FBcT8peNjbOLylU6wCpP4bbWQ69HiHc3fJl3ejBgGIAevPquFVuLRBCLR8EQsraDZt+VJLtAFxF6ZyURPDG8TrNll4zBMgwgGeMwCQqoDIMjbDEdV24dY+x0GqhmypQWo2InEXsXBZt4tqKCrautYp3GeDdpiBPxwDCARAyDPBBQHSg2fSMYA0kMM4jQOFOIWo2A+A6oaQJ1Utu9IHQydvYYUV5+LJOQ0zCAJIyCQhiw88CquLTAxkZZI1FoOKlhnHVWVJLTDdII6zIWeYWDlYCq/j+A3luv/BDsyQAAAABJRU5ErkJggg== provider_display_name: Pivotal support_url: TBA plan_updatable: true plans:
- description: 'Example: This plan will configure a lightweight kubernetes cluster.
Not recommended for production workloads.'
instance_groups:
- azs:
- az-1
- az-2
- az-3 instances: 1 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
- azs:
- az-1
- az-2
- az-3 instances: 3 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
- azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
disabled: true name: wavefront-proxy-errand
-
name: telemetry-agent pre_delete:
-
name: drain-cluster metadata: allow-privileged-containers: true master_instances: 1 max_worker_instances: 50 worker_instances: 3 name: Plan 1 plan_id: 8A0E21A8-8072-4D80-B365-D1F502085560 properties: addons-spec: |+
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Sinks [{"addr":"example.com:12345"}]
parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log
apiVersion: v1 kind: Service metadata: name: fluent-bit namespace: pks-system spec: selector: k8s-app: logging-agent ports: - protocol: TCP port: 24224 targetPort: forward-plugin type: ClusterIP
apiVersion: batch/v1 kind: Job metadata: labels: job: cert-generator name: cert-generator-v0.11 namespace: pks-system spec: backoffLimit: 0 template: metadata: labels: job: cert-generator spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/cert-generator:v0.12 imagePullPolicy: IfNotPresent name: cert-generator restartPolicy: Never serviceAccountName: cert-generator
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: fluent-bit resources: limits: memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true - command: - ghostunnel - server - --listen - :24224 - --target - localhost:24225 - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --allow-dns-san - event-controller - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel ports: - containerPort: 24224 name: forward-plugin volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /fluent-bit-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /fluent-bit-certs name: fluent-bit-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config - emptyDir: {} name: keystore - name: fluent-bit-certs secret: secretName: fluent-bit - name: pks-ca secret: secretName: pks-ca updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: event-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: event-controller spec: containers: - env: - name: FORWARDER_HOST value: localhost image: oratos/event-controller:v0.12 imagePullPolicy: IfNotPresent name: event-controller - command: - ghostunnel - client - --listen - localhost:24224 - --target - fluent-bit.pks-system.svc.cluster.local:24224 - --override-server-name - fluent-bit - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /event-controller-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /event-controller-certs name: event-controller-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: event-controller volumes: - emptyDir: {} name: keystore - name: event-controller-certs secret: secretName: event-controller - name: pks-ca secret: secretName: pks-ca
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.12 imagePullPolicy: IfNotPresent name: sink-controller serviceAccountName: sink-controller
-
- azs:
- description: 'Example: This plan will configure a medium sized kubernetes
cluster, suitable for more pods.'
instance_groups:
-
azs:
- az-1
- az-2
- az-3 instances: 3 name: master networks:
- service-network persistent_disk_type: "10240" vm_extensions: [] vm_type: medium
-
azs:
- az-1
- az-2
- az-3 instances: 5 name: worker networks:
- service-network persistent_disk_type: "51200" vm_extensions: [] vm_type: medium
-
azs:
-
az-1
-
az-2
-
az-3 instances: 1 lifecycle: errand name: apply-addons networks:
-
service-network vm_type: medium lifecycle_errands: post_deploy:
-
name: apply-addons
-
disabled: true name: wavefront-proxy-errand
-
name: telemetry-agent pre_delete:
-
name: drain-cluster metadata: allow-privileged-containers: true master_instances: 3 max_worker_instances: 50 worker_instances: 5 name: multi-master plan_id: 58375a45-17f7-4291-acf1-455bfdc8e371 properties: addons-spec: |
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10
filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Merge_Log On K8S-Logging.Parser On
output-file.conf: | [OUTPUT] Name file Match * Path /tmp/output.txt
output-null.conf: | [OUTPUT] Name null
output-syslog.conf: | [OUTPUT] Name syslog Match * Sinks [{"addr":"example.com:12345"}]
parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As escaped log
apiVersion: v1 kind: Service metadata: name: fluent-bit namespace: pks-system spec: selector: k8s-app: logging-agent ports: - protocol: TCP port: 24224 targetPort: forward-plugin type: ClusterIP
apiVersion: batch/v1 kind: Job metadata: labels: job: cert-generator name: cert-generator-v0.11 namespace: pks-system spec: backoffLimit: 0 template: metadata: labels: job: cert-generator spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/cert-generator:v0.12 imagePullPolicy: IfNotPresent name: cert-generator restartPolicy: Never serviceAccountName: cert-generator
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 name: fluent-bit namespace: pks-system spec: template: metadata: labels: k8s-app: logging-agent kubernetes.io/cluster-service: "true" version: v1 spec: containers: - image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: fluent-bit resources: limits: memory: 100Mi volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /var/lib/docker/containers name: varlibdockercontainers readOnly: true - mountPath: /var/vcap/store name: varvcapstore readOnly: true - mountPath: /var/vcap/data name: varvcapdata readOnly: true - command: - ghostunnel - server - --listen - :24224 - --target - localhost:24225 - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --allow-dns-san - event-controller - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel ports: - containerPort: 24224 name: forward-plugin volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /fluent-bit-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /fluent-bit-certs name: fluent-bit-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: fluent-bit terminationGracePeriodSeconds: 10 volumes: - hostPath: path: /var/log name: varlog - hostPath: path: /var/lib/docker/containers name: varlibdockercontainers - hostPath: path: /var/vcap/store/ name: varvcapstore - hostPath: path: /var/vcap/data/ name: varvcapdata - configMap: name: fluent-bit name: fluent-bit-config - emptyDir: {} name: keystore - name: fluent-bit-certs secret: secretName: fluent-bit - name: pks-ca secret: secretName: pks-ca updateStrategy: type: RollingUpdate
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: event-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: event-controller spec: containers: - env: - name: FORWARDER_HOST value: localhost image: oratos/event-controller:v0.12 imagePullPolicy: IfNotPresent name: event-controller - command: - ghostunnel - client - --listen - localhost:24224 - --target - fluent-bit.pks-system.svc.cluster.local:24224 - --override-server-name - fluent-bit - --keystore - /keystore/keystore.pem - --cacert - /pks-ca/tls.crt - --cipher-suites - AES image: oratos/ghostunnel:v0.12 imagePullPolicy: IfNotPresent name: ghostunnel volumeMounts: - mountPath: /keystore name: keystore readOnly: true - mountPath: /pks-ca/tls.crt name: pks-ca readOnly: true subPath: tls.crt initContainers: - command: - /bin/bash - -c - cat /event-controller-certs/* > /keystore/keystore.pem image: oratos/fluent-bit-out-syslog:v0.11 imagePullPolicy: IfNotPresent name: concat-keystore volumeMounts: - mountPath: /event-controller-certs name: event-controller-certs readOnly: true - mountPath: /keystore name: keystore serviceAccountName: event-controller volumes: - emptyDir: {} name: keystore - name: event-controller-certs secret: secretName: event-controller - name: pks-ca secret: secretName: pks-ca
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: sink-controller namespace: pks-system spec: replicas: 1 template: metadata: labels: app: sink-controller spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: oratos/sink-controller:v0.12 imagePullPolicy: IfNotPresent name: sink-controller serviceAccountName: sink-controller
allow-privileged-containers: true disable_deny_escalating_exec: true max_worker_instances: 50
-
-
- null service_description: Default on-demand Kubernetes service. service_name: p.pks tags:
- pivotal
- kubernetes
- k8s service_deployment: releases:
- jobs:
- kube-apiserver
- kube-controller-manager
- kube-scheduler
- kubelet
- kube-proxy
- kubernetes-roles
- flanneld
- nginx
- kubernetes-api-route-registrar
- apply-specs
- secure-var-vcap name: kubo version: 0.25.8
- jobs:
- etcd name: cfcr-etcd version: 1.8.0
- jobs:
- docker name: docker version: 33.0.2
- jobs:
- pks-nsx-t-prepare-master-vm
- pks-nsx-t-ncp name: pks-nsx-t version: 1.19.0
- jobs:
- ncp
- nsx-node-agent
- openvswitch
- nsx-cni
- nsx-kube-proxy name: nsx-cf-cni version: 2.3.1.10693410
- jobs:
- fluentd name: pks-vrli version: 0.7.0
- jobs:
- syslog_forwarder name: syslog version: 11.4.0
- jobs:
- bpm name: bpm version: 0.13.0
- jobs:
- wavefront-proxy name: wavefront-proxy version: 0.9.0
- jobs:
- drain-cluster name: pks-helpers version: 50.0.0
- jobs:
- telemetry-dns-alias name: pks-telemetry version: 2.0.0-build.113
- jobs:
- sink-resources-images name: sink-resources-release version: 0.1.15 stemcell: os: ubuntu-xenial version: "170.19" service_instances_api: authentication: basic: password: ((pks_api_basicauth.password)) username: ((pks_api_basicauth.username)) root_ca_cert: ((pks_api_internal_2018.ca)) url: https://localhost:9021/service_instances startup_banner: true username: ((odb_broker_basicauth.username)) stemcell: bosh-vsphere-esxi-ubuntu-xenial-go_agent update: max_in_flight: 1 vm_type: large name: pivotal-container-service-69c3e199ae8c7bf63cd0 releases:
- name: cf-mysql version: 36.14.0
- name: docker version: 33.0.2
- name: kubo version: 0.25.8
- name: cfcr-etcd version: 1.8.0
- name: kubo-service-adapter version: 1.3.0-build.129
- name: on-demand-service-broker version: 0.24.0
- name: pks-api version: 1.3.0-build.129
- name: pks-helpers version: 50.0.0
- name: pks-nsx-t version: 1.19.0
- name: nsx-cf-cni version: 2.3.1.10693410
- name: pks-vrli version: 0.7.0
- name: syslog version: 11.4.0
- name: sink-resources-release version: 0.1.15
- name: pks-telemetry version: 2.0.0-build.113
- name: uaa version: "64.0"
- name: bpm version: 0.13.0
- name: backup-and-restore-sdk version: 1.8.0
- name: wavefront-proxy version: 0.9.0 stemcells:
- alias: bosh-vsphere-esxi-ubuntu-xenial-go_agent os: ubuntu-xenial version: "170.19" update: canaries: 1 canary_watch_time: 30000-300000 max_errors: 2 max_in_flight: 1 serial: false update_watch_time: 30000-300000 variables:
- name: kubo_odb_ca options: common_name: ca is_ca: true type: certificate
- name: kubo_odb_ca_2018 options: common_name: ca duration: 1461 is_ca: true type: certificate
- name: pks_api_internal_2018
options:
alternative_names:
- localhost
- 127.0.0.1 ca: kubo_odb_ca_2018 common_name: localhost duration: 1461 type: certificate
- name: uaa_jwt_signing_key_1 type: rsa
- name: uaa_active_pks_saml_key_2018 options: common_name: ca duration: 1461 is_ca: true type: certificate
- name: pks_api_basicauth type: user
- name: odb_broker_basicauth type: user
- name: telemetry_ca_2018 options: common_name: ca duration: 1461 is_ca: true type: certificate
- name: telemetry_server_tls_2018 options: ca: telemetry_ca_2018 common_name: telemetry.pks.internal duration: 1461 type: certificate
Succeeded #+END_EXAMPLE ** k8s manifest :noexport: #+BEGIN_EXAMPLE kubo@jumper:~$ bosh -d service-instance_09514531-7d36-405c-8bda-493057b8d805 manifest Using environment '30.0.0.11' as client 'ops_manager'
Using deployment 'service-instance_09514531-7d36-405c-8bda-493057b8d805'
addons:
- name: bosh-dns-aliases
jobs:
- name: kubo-dns-aliases release: kubo name: service-instance_09514531-7d36-405c-8bda-493057b8d805 releases:
- name: kubo version: 0.25.8
- name: cfcr-etcd version: 1.8.0
- name: docker version: 33.0.2
- name: pks-nsx-t version: 1.19.0
- name: nsx-cf-cni version: 2.3.1.10693410
- name: pks-vrli version: 0.7.0
- name: syslog version: 11.4.0
- name: bpm version: 0.13.0
- name: wavefront-proxy version: 0.10.0-dev.141
- name: pks-vrops version: 0.10.0+dev.5
- name: pks-helpers version: 50.0.0
- name: pks-telemetry version: 2.0.0-build.113
- name: sink-resources-release version: 0.1.15 stemcells:
- alias: default os: ubuntu-xenial version: "170.15" instance_groups:
- nam:e apply-addons
lifecycle: errand
instances: 1
jobs:
- name: apply-specs
release: kubo
consumes:
cloud-provider:
from: master-cloud-provider
properties:
addons:
-
kube-dns
-
metrics-server
-
heapster
-
kubernetes-dashboard addons-spec: |+
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: clustersinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Cluster names: plural: clustersinks singular: clustersink kind: ClusterSink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: sinks.apps.pivotal.io spec: group: apps.pivotal.io version: v1beta1 versions: - name: v1beta1 served: true storage: true scope: Namespaced names: plural: sinks singular: sink kind: Sink validation: openAPIV3Schema: properties: spec: required: - type - port - host properties: port: type: integer minimum: 0 maximum: 65535 type: type: string enum: - syslog host: type: string pattern: '^([a-zA-Z0-9-.]+)$|^([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})$|^([a-fA-F0-9:]+)$' enable_tls: type: boolean insecure_skip_verify: type: boolean additionalPrinterColumns: - name: Type JSONPath: .spec.type type: string - name: Host JSONPath: .spec.host type: string - name: Port JSONPath: .spec.port type: integer - name: TLS JSONPath: .spec.enable_tls type: boolean - name: Insecure JSONPath: .spec.insecure_skip_verify type: boolean description: | Accept any certificate presented by the server and any host name in that certificate. - JSONPath: .metadata.creationTimestamp name: Age type: date
apiVersion: v1 kind: Namespace metadata: name: pks-system
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system rules:
- apiGroups: [""] resources: ["secrets"] verbs: ["get", "create"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- cert-generator
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: cert-generator namespace: pks-system subjects:
- kind: ServiceAccount name: cert-generator namespace: pks-system roleRef: kind: Role name: cert-generator apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller rules:
The event-controller needs to be able to watch events
- apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- event-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: event-controller subjects:
- kind: ServiceAccount name: event-controller namespace: pks-system roleRef: kind: ClusterRole name: event-controller apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit rules:
This rule is for kubernetes-metadata-filter
- apiGroups:
- ""
- "apps"
- "batch" resources: ["*"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- fluent-bit
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: fluent-bit subjects:
- kind: ServiceAccount name: fluent-bit namespace: pks-system roleRef: kind: ClusterRole name: fluent-bit apiGroup: rbac.authorization.k8s.io
kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller rules:
The sink-controller needs to patch the configmap for fluent-bit
- apiGroups: [""] # "" indicates the core API group resources: ["configmaps"] verbs: ["get", "list", "watch", "patch"] # TODO: Do we need watch?
The sink-controller needs to be able to delete the fluent-bit pods
- apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["deletecollection"]
The sink-controller needs to be able to watch sinks and clustersinks
- apiGroups: ["pksapi.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
This api group is for backwards compatability
- apiGroups: ["apps.pivotal.io"] resources: ["sinks", "clustersinks"] verbs: ["get", "list", "watch"]
- apiGroups:
- policy resources:
- podsecuritypolicies verbs:
- use resourceNames:
- sink-controller
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sink-controller subjects:
- kind: ServiceAccount name: sink-controller namespace: pks-system roleRef: kind: ClusterRole name: sink-controller apiGroup: rbac.authorization.k8s.io
apiVersion: v1 kind: ServiceAccount metadata: name: cert-generator namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: event-controller namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit namespace: pks-system
apiVersion: v1 kind: ServiceAccount metadata: name: sink-controller namespace: pks-system
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: cert-generator spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: event-controller spec: volumes:
- emptyDir
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: fluent-bit spec: volumes:
- hostPath
- configMap
- emptyDir
- secret allowedHostPaths:
- pathPrefix: /var/log readOnly: false
- pathPrefix: /var/lib/docker/containers readOnly: true
- pathPrefix: /var/vcap/store readOnly: true
- pathPrefix: /var/vcap/data readOnly: true runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: sink-controller spec: volumes:
- secret runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit namespace: pks-system labels: k8s-app: fluent-bit data:
Configuration files: server, input, filters and output
======================================================
fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
@INCLUDE inputs.conf @INCLUDE filters.conf @INCLUDE outputs.confinputs.conf: | @INCLUDE input-kubernetes.conf @INCLUDE input-forward.conf
filters.conf: | @INCLUDE filter-kubernetes.conf
outputs.conf: | @INCLUDE output-null.conf
input-forward.conf: | [INPUT] Name forward Port 24225 Listen localhost
input-kubernetes.conf: | [INPUT] Name tail
-
- name: apply-specs
release: kubo
consumes:
cloud-provider:
from: master-cloud-provider
properties:
addons:
dennyzhang