Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[aws-for-fluent-bit] does not Filter: Kuberntes for Labels, Annotation #1164

Open
sav-work opened this issue Sep 30, 2024 · 0 comments
Open
Labels
bug Something isn't working

Comments

@sav-work
Copy link

Describe the bug
I want to filter out all stdout.
I pass a parameter to the filter and it works
Exclude $stream stdout

But I want to have the opportunity to include it separately via Labels: fluentbit-logging=stdout or Annotations: in pods or namespace, this is what my config looks like now
Regex $kubernetes['labels']['fluentbit-logging'] stdout
And it doesn't work the whole stdout is filtered.

Maybe there's another way I don't know
Thank you for your help.

Other methods
I also tried different options and it didn't work for me.
option 1

    [FILTER]
        Name                grep
        Match               application.*
        Exclude             $stream stdout

    [FILTER]
        Name                grep
        Match               application.*
        Regex               $stream stdout
        Regex               $kubernetes['pod_annotations']['fluentbit-logging'] stdout
        Regex               $kubernetes['labels']['fluentbit-logging'] stdout`

And it doesn't work the whole stdout is filtered.

Expected outcome
Filter separately through the Label or Annotations for Pods, Namespace

  • Chart name: manifest aws-for-fluent
  • Chart version: aws-for-fluent-bit:2.31.11
  • Kubernetes version: 1.28
  • Using EKS (yes/no), if so version? 1.28

Additional Context:
Configmap
data:
application-log.conf: |

    [INPUT]
        Name                tail
        Tag                 application.*
        Path                /var/log/containers/fluent-bit*
        multiline.parser    docker, cri
        DB                  /var/fluent-bit/state/flb_log.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}

    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     application.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Labels              On
        Annotations         On
        Use_Kubelet         On
        Kubelet_Port        10250
        Buffer_Size         0


    [FILTER]
        Name                grep
        Match               application.*
        Exclude             $stream stdout

    [FILTER]
        Name                grep
        Match               application.*
        Regex               $kubernetes['labels']['fluentbit-logging'] stdout

    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/logs
        log_group_template  /aws/containerinsights/${CLUSTER_NAME}/workload/$kubernetes['namespace_name']
        log_stream_prefix   ${HOST_NAME}-
        log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
        auto_create_group   true
        extra_user_agent    container-insights
        log_retention_days  30
        Log_Level           debug
@sav-work sav-work added the bug Something isn't working label Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant