vectordotdev
In the global configuration, add a configuration item to add fields such as cluster name and region for k8s, which are used to identify k8s cluster information
In the global configuration, add a configuration item to add fields such as cluster name and region for k8s, which are used to identify k8s cluster information
Achieving the goal: It is possible to filter out different k8s cluster logs clearly in the index of es
Hi @junlintianxiazhifulinzhongguo,
- By global configuration do refer to the global options?
- Can you elaborate on your use case with an example?
Thank you!
The current situation is that I want to achieve generating different Kafka topics from different namespaces in a cluster, and each index has a common cluster name identification field. The current solution is as follows:
agent-k8s-namespace-lijun-app-xiaoyou.yaml: |-
sources:
kubernetes_logs_namespace_lijun_app_xiaoyou:
type: kubernetes_logs
auto_partial_merge: true
delay_deletion_ms: 60000
extra_namespace_label_selector: kubesphere.io/namespace=lijun
extra_label_selector: app=xiaoyou
fingerprint_lines: 1
glob_minimum_cooldown_ms: 60000
ignore_older_secs: 600
ingestion_timestamp_field: .ingest_timestamp
max_line_bytes: 32768
max_read_bytes: 2048
oldest_first: true
read_from: beginning
self_node_name: ${VECTOR_SELF_NODE_NAME}
timezone: local
transforms:
transform_kubernetes_logs_namespace_lijun_app_xiaoyou:
type: remap
inputs:
- kubernetes_logs_namespace_lijun_app_xiaoyou
source: |
.cluster = to_string("hangli")
The result of doing so is to add the following paragraph to the log collection for each namespace：
transforms:
transform_kubernetes_logs_namespace_demo_app_xiaoyou:
type: remap
inputs:
- kubernetes_logs_namespace_demo_app_xiaoyou
source: |
.cluster = to_string("hangli")
The effect I want is to add a configuration field in the global configuration, which can be customized to enable clear identification of clusters in different regions, such as:
data_dir: /vector-data-dir
custom_fields:
region: China
cluster: hainan-1
Just like customizing a label for a k8s cluster
In this way, custom log messages related to region and cluster names will be automatically generated in the final log output
@pront Thank you. If there are any other better configuration methods, I would like to test them and see if they are available。
Hi @jszwedko，As for the current vector, is there any way to implement it in the situation I mentioned? If not, can this feature be added to the future development plan? Thank you
Hi @junlintianxiazhifulinzhongguo ,
Just so I understand: the request here is to add global configuration of "custom fields" that would then automatically be added to all events generated by all sources in the configuration?
