upgrade to 4.9.0 results in plugin capability used before init: k8s meta

[jemag]

Describe the bug
Upgrading the chart from 4.8.3 to 4.9.0 without any change to our values.yaml, there appear to now be a schema validation error:

Thu Oct  3 16:27:33 2024: Falco initialized with configuration files:
Thu Oct  3 16:27:33 2024:    /etc/falco/falco.yaml | schema validation: failed for <root>[plugins][0][init_config]: Value type not permitted by 'type' constraint.

Here is our plugins section from the falco configmap

    plugins:
    - init_config: null
      library_path: libk8saudit.so
      name: k8saudit
      open_params: http://:9765/k8s-audit
    - library_path: libcloudtrail.so
      name: cloudtrail
    - init_config: ""
      library_path: libjson.so
      name: json
    - init_config:
        collectorHostname: falco-k8s-metacollector.falco.svc
        collectorPort: 45000
        nodeName: ${FALCO_K8S_NODE_NAME}
      library_path: libk8smeta.so
      name: k8smeta

This eventually result (not 100% sure if related) in the pod terminating with the following error:
Thu Oct 3 16:38:28 2024: Runtime error: plugin capability used before init: k8smeta. Exiting.

How to reproduce it

Deploy helm chart 4.9.0 with following values:

podAnnotations:                                                                                                                                                                                                                
  container.apparmor.security.beta.kubernetes.io/falco: unconfined                                                                                                                                                             
serviceMonitor:                                                                                                                                                                                                                
  create: true                                                                                                                                                                                                                 
  labels:                                                                                                                                                                                                                      
    app: kube-prometheus-stack                                                                                                                                                                                                 
    release: kube-prometheus-stack                                                                                                                                                                                      
extra:                                                                                                                                                                                                                         
  args:                                                                                                                                                                                                                        
    - --disable-cri-async                                                                                                                                                                                                      
resources:                                                                                                                                                                                                                     
  requests:                                                                                                                                                                                                                    
    cpu: 100m                                                                                                                                                                                                                  
    memory: 300Mi                                                                                                                                                                                                              
  limits:                                                                                                                                                                                                                      
    cpu: 999                                                                                                                                                                                                                   
    memory: 500Mi                                                                                                                                                                                                              
tty: false                                                                                                                                                                                                                     
controller:                                                                                                                                                                                                                    
  kind: daemonset                                                                                                                                                                                                              
driver:                                                                                                                                                                                                                        
  enabled: true                                                                                                                                                                                                                
  kind: modern_ebpf                                                                                                                                                                                                            
  modernEbpf:                                                                                                                                                                                                                  
    leastPrivileged: true                                                                                                                                                                                                      
collectors:                                                                                                                                                                                                                    
  enabled: true                                                                                                                                                                                                                
  kubernetes:                                                                                                                                                                                                                  
    enabled: true                                                                                                                                                                                                              
metrics:                                                                                                                                                                                                                       
  enabled: true                                                                                                                                                                                                                
falco:                                                                                                                                                                                                                         
  grpc:                                                                                                                                                                                                                        
    enabled: true                                                                                                                                                                                                              
  grpc_output:                                                                                                                                                                                                                 
    enabled: true                                                                                                                                                                                                              
  rules_files:                                                                                                                                                                                                                 
    - /etc/falco/falco_rules.yaml                                                                                                                                                                                              
    - /etc/falco/falco-incubating_rules.yaml                                                                                                                                                                                   
    - /etc/falco/rules.d                                                                                                                                                                                                       
  webserver:                                                                                                                                                                                                                   
    prometheus_metrics_enabled: true                                                                                                                                                                                           
falcoctl:                                                                                                                                                                                   
  artifact:                                                                                                                                                                                                                    
    install:                                                                                                                                                                                                                   
      resources:                                                                                                                                                                                                               
        requests:                                                                                                                                                                                                              
          cpu: 10m                                                                                                                                                                                                             
          memory: 50Mi                                                                                                                                                                                                         
        limits:                                                                                                                                                                                                                
          memory: 100Mi                                                                                                                                                                                                        
      # -- Enable the init container. We do not recommend installing (or following) plugins for security reasons since they are executable objects.                                                                            
      enabled: true                                                                                                                                                                                                            
    follow:                                                                                                                                                                                                                    
      resources:                                                                                                                                                                                                               
        requests:                                                                                                                                                                                                              
          cpu: 10m                                                                                                                                                                                                             
          memory: 50Mi                                                                                                                                                                                                         
        limits:                                                                                                                                                                                                                
          memory: 100Mi                                                                                                                                                                                                        
      # -- Enable the sidecar container. We do not support it yet for plugins. It is used only for rules feed such as k8saudit-rules rules.                                                                                    
      enabled: true                                                                                                                                                                                                            
  config:                                                                                                                                                                                                                      
    artifact:                                                                                                                                                                                                                  
      install:                                                                                                                                                                                                                 
        # -- List of artifacts to be installed by the falcoctl init container.                                                                                                                                                 
        refs: [falco-rules:3, falco-incubating-rules:3]                                                                                                                                                                        
      follow:                                                                                                                                                                                                                  
        # -- List of artifacts to be installed by the falcoctl init container.                                                                                                                                                 
        refs: [falco-rules:3, falco-incubating-rules:3]                                                                                                                                                                        
                                                                                                                                                               
k8s-metacollector:                                                                                                                                                                                
  resources:                                                                                                                                                                                                                   
    requests:                                                                                                                                                                                                                  
      cpu: 10m                                                                                                                                                                                                                 
      memory: 50Mi                                                                                                                                                                                                             
    limits:                                                                                                                                                                                                                    
      memory: 100Mi                                                                                                                                                                                                            
  containerSecurityContext:                                                                                                                                                                                                    
    allowPrivilegeEscalation: false                                                                                                                                                                                            
    readOnlyRootFilesystem: true                                                                                                                                                                                               
    seccompProfile:                                                                                                                                                                                                            
      type: RuntimeDefault                                                                                                                                                                                                     
  serviceMonitor:                                                                                                                                                                                                              
    create: true                                                                                                                                                                                                               
    # -- path at which the metrics are expose by the k8s-metacollector.                                                                                                                                                        
    path: /metrics                                                                                                                                                                                                             
    # -- labels set of labels to be applied to the ServiceMonitor resource.                                                                                                                                                    
    labels:                                                                                                                                                                                                                    
      app: kube-prometheus-stack                                                                                                                                                                                               
      release: kube-prometheus-stack                                                                                                                                                                                           
  grafana:                                                                                                                                                                                                                     
    dashboards:                                                                                                                                                                                                                
      enabled: true                                                                                                                                                                                                            

Expected behaviour

Falco runs normally just as in 4.8.3

Screenshots

Environment

• Falco version:
  cannot run command with crashlooping container, image falco-no-driver:0.39.0
• System info:
  cannot run command with crashlooping container, hope this is sufficient;:
  Linux version 5.15.164.1-1.cm2 (root@CBL-Mariner) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP

• Cloud provider or hardware configuration:
  Azure AKS

• OS:
  NAME="Common Base Linux Mariner"
  VERSION="2.0.20240829"
  ID=mariner
  VERSION_ID="2.0"
  PRETTY_NAME="CBL-Mariner/Linux"
  ANSI_COLOR="1;34"
  HOME_URL="https://aka.ms/cbl-mariner"
  BUG_REPORT_URL="https://aka.ms/cbl-mariner"
  SUPPORT_URL="https://aka.ms/cbl-mariner"

• Kernel:
  Linux aks-main-14571596-vmss000008 5.15.164.1-1.cm2 Pull falco chart from helm/charts #1 SMP Sun Aug 18 19:16:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

• Installation method:
  -helm chart 4.9.0

Additional context

Full logs before crash

Thu Oct  3 16:38:28 2024: The --cri option is deprecated and will be removed in Falco 0.40.0. Use -o container_engines.cri.sockets[]=<socket_path> instead.
Thu Oct  3 16:38:28 2024: The --disable-cri-async option is deprecated and will be removed in Falco 0.40.0. Use -o container_engines.cri.disable_async=true instead.
Thu Oct  3 16:38:28 2024: Falco version: 0.39.0 (x86_64)
Thu Oct  3 16:38:28 2024: Falco initialized with configuration files:
Thu Oct  3 16:38:28 2024:    /etc/falco/falco.yaml | schema validation: failed for <root>[plugins][0][init_config]: Value type not permitted by 'type' constraint.
Thu Oct  3 16:38:28 2024: System info: Linux version 5.15.164.1-1.cm2 (root@CBL-Mariner) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP Sun Aug 18 19:16:21 UTC 2024
Thu Oct  3 16:38:28 2024: Loading plugin 'k8smeta' from file /usr/share/falco/plugins/libk8smeta.so
Thu Oct 3 16:38:28 2024: [warning] [k8smeta] [EXPERIMENTAL] This plugin is in active development and may undergo changes in behavior without prioritizing backward compatibility.
Thu Oct 3 16:38:28 2024: [info] [k8smeta] Start the process scan under: '/host/proc'
Thu Oct 3 16:38:28 2024: [info] [k8smeta] Process scan correctly completed. Found '125' threads inside pods.
Thu Oct  3 16:38:28 2024: Loading rules from:
Thu Oct  3 16:38:28 2024:    /etc/falco/falco_rules.yaml | schema validation: ok
Thu Oct  3 16:38:28 2024:    /etc/falco/falco-incubating_rules.yaml | schema validation: ok
Thu Oct  3 16:38:28 2024:    /etc/falco/rules.d/custom.local.yaml | schema validation: ok
Thu Oct  3 16:38:28 2024: /etc/falco/rules.d/custom.local.yaml: Ok, with warnings
5 Warnings:
In rules content: (/etc/falco/falco-incubating_rules.yaml:0:0)
    rule 'System procs network activity': (/etc/falco/falco-incubating_rules.yaml:663:2)
------
- rule: System procs network activity
  ^
------
LOAD_NO_EVTTYPE (Condition has no event-type restriction): Rule matches too many evt.type values. This has a significant performance penalty.
In rules content: (/etc/falco/falco-incubating_rules.yaml:0:0)
    rule 'Unexpected UDP Traffic': (/etc/falco/falco-incubating_rules.yaml:746:2)
------
- rule: Unexpected UDP Traffic
  ^
------
LOAD_NO_EVTTYPE (Condition has no event-type restriction): Rule matches too many evt.type values. This has a significant performance penalty.
In rules content: (/etc/falco/falco-incubating_rules.yaml:0:0)
    rule 'Network Connection outside Local Subnet': (/etc/falco/falco-incubating_rules.yaml:1108:2)
------
- rule: Network Connection outside Local Subnet
  ^
------
LOAD_NO_EVTTYPE (Condition has no event-type restriction): Rule matches too many evt.type values. This has a significant performance penalty.
In rules content: (/etc/falco/rules.d/custom.local.yaml:0:0)
    rule 'Calico packet': (/etc/falco/rules.d/custom.local.yaml:6:2)
------
- rule: Calico packet
  ^
------
LOAD_NO_EVTTYPE (Condition has no event-type restriction): Rule matches too many evt.type values. This has a significant performance penalty.
In rules content: (/etc/falco/rules.d/custom.local.yaml:0:0)
    list 'user_known_ingress_remote_file_copy_images': (/etc/falco/rules.d/custom.local.yaml:393:2)
------
- list: user_known_ingress_remote_file_copy_images
  ^
------
LOAD_UNUSED_LIST (Unused list): List not referred to by any other rule/macro
Thu Oct  3 16:38:28 2024: Hostname value has been overridden via environment variable to: aks-main-14571596-vmss000008
Thu Oct  3 16:38:28 2024: Runtime error: plugin capability used before init: k8smeta. Exiting.

[CarpathianUA]

We're experiencing the same behavior and rollbacked chart version back to 4.8.3

[alacuku]

@jemag, @CarpathianUA it's a regression introduced in Falco 0.39.0. We are working on the fix and will release a patch as soon as possible.

Thanks for reporting the bug.