Describe the bug 采集的数据无法上报到kafka，出现报错；我这边kafka是正常的logstash可以消费测试的数据，集群是没问题的

iLogtail Running Environment Please provide the following information:

• ilogtail version: 最新镜像版本

• Yaml configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ilogtail-user-cm
  namespace: default
data:
  cloud_stdout.yaml: |
    enable: true
    inputs:
      - Type: service_docker_stdout
        Stderr: false
        Stdout: true                # only collect stdout
        IncludeK8sLabel:
          app: nginx                # choose containers with this label
    flushers:
      - Type: flusher_kafka
        Brokers:
          - kafka-cluster:9092
        Topic: access-log
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ilogtail-ds
  namespace: default
  labels:
    k8s-app: logtail-ds
spec:
  selector:
    matchLabels:
      k8s-app: logtail-ds
  template:
    metadata:
      labels:
        k8s-app: logtail-ds
    spec:
      tolerations:
        - operator: Exists                    # deploy on all nodes
      containers:
        - name: logtail
          env:
            - name: ALIYUN_LOG_ENV_TAGS       # add log tags from env
              value: _node_name_|_node_ip_
            - name: _node_name_
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: _node_ip_
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
            - name: cpu_usage_limit           # iLogtail's self monitor cpu limit
              value: "1"
            - name: mem_usage_limit           # iLogtail's self monitor mem limit
              value: "512"
          image: >-
            sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:latest
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi
            requests:
              cpu: 400m
              memory: 384Mi
          volumeMounts:
            - mountPath: /var/run                       # for container runtime socket
              name: run
            - mountPath: /logtail_host                  # for log access on the node
              mountPropagation: HostToContainer
              name: root
              readOnly: true
            - mountPath: /usr/local/ilogtail/checkpoint # for checkpoint between container restart
              name: checkpoint
            - mountPath: /usr/local/ilogtail/user_yaml_config.d # mount config dir
              name: user-config
              readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      volumes:
        - hostPath:
            path: /var/run
            type: Directory
          name: run
        - hostPath:
            path: /
            type: Directory
          name: root
        - hostPath:
            path: /etc/ilogtail-ilogtail-ds/checkpoint
            type: DirectoryOrCreate
          name: checkpoint
        - configMap:
            defaultMode: 420
            name: ilogtail-user-cm
          name: user-config
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: 'nginx:latest'
          name: nginx
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 100Mi

• ilogtail.LOG:

[2022-12-05 07:55:32.433230] [info] [000011] /src/core/app_config/AppConfigBase.cpp:150 AppConfigBase AppConfigBase:success [2022-12-05 07:55:32.433250] [info] [000011] /src/core/app_config/AppConfigBase.cpp:314 load env tags from env key:node_name|node_ip [2022-12-05 07:55:32.433281] [info] [000011] /src/core/app_config/AppConfigBase.cpp:322 load env, key:node_name value:k8s-master [2022-12-05 07:55:32.433284] [info] [000011] /src/core/app_config/AppConfigBase.cpp:322 load env, key:node_ip value:10.3.0.14 [2022-12-05 07:55:32.433292] [info] [000011] /src/core/app_config/AppConfigBase.cpp:379 purage container mode:true [2022-12-05 07:55:32.433310] [info] [000011] /src/core/logtail.cpp:156 change working dir:/usr/local/ilogtail/1.3.0 fail, reason:No such file or directory [2022-12-05 07:55:32.433326] [info] [000011] /src/core/logtail.cpp:158 change working dir:/usr/local/ilogtail/ result:0 [2022-12-05 07:55:32.433407] [info] [000011] /src/core/app_config/AppConfigBase.cpp:1117 set logtail sys conf dir:/usr/local/ilogtail/./ user local config path:/usr/local/ilogtail/./user_local_config.json user local config dir path:/usr/local/ilogtail/./user_config.d/ user local yaml config dir path:/usr/local/ilogtail/./user_yaml_config.d/ [2022-12-05 07:55:32.433446] [info] [000011] /src/core/app_config/AppConfigBase.cpp:283 load logtail config file, path:ilogtail_config.json [2022-12-05 07:55:32.433449] [info] [000011] /src/core/app_config/AppConfigBase.cpp:284 load logtail config file, detail:{}

[2022-12-05 07:55:32.433622] [info] [000011] /src/core/app_config/AppConfigBase.cpp:592 logtail checkpoint path:/usr/local/ilogtail/checkpoint/logtail_check_point [2022-12-05 07:55:32.433634] [info] [000011] /src/core/common/JsonUtil.cpp:138 load config from env:docker_file_cache_path value:checkpoint/docker_path_config.json [2022-12-05 07:55:32.433637] [info] [000011] /src/core/common/JsonUtil.cpp:171 load parameter from env:docker_file_cache_path value:checkpoint/docker_path_config.json [2022-12-05 07:55:32.433663] [info] [000011] /src/core/app_config/AppConfigBase.cpp:340 set cpu_usage_limit from env, value:1 [2022-12-05 07:55:32.433672] [info] [000011] /src/core/app_config/AppConfigBase.cpp:340 set mem_usage_limit from env, value:512 [2022-12-05 07:55:32.433678] [info] [000011] /src/core/app_config/AppConfigBase.cpp:1043 default cache size, polling total max stat:250000 per dir max stat:25000 per config max stat:25000 cache config max size:250000 modify max:25000 watch dir count max:25000 max open files limit:32765 max reader open files limit:25000 [2022-12-05 07:55:32.433681] [info] [000011] /src/core/app_config/AppConfigBase.cpp:1047 send byte per second limit:26214400 batch send interval:3 batch send size:262144 [2022-12-05 07:55:32.436558] [info] [000011] /src/core/logtail.cpp:183 IP is still empty, specified interface: try to get any available IP:10.3.0.14 [2022-12-05 07:55:32.436643] [info] [000011] /src/core/common/TimeUtil.cpp:232 get system boot time from /proc/uptime:1668743992 [2022-12-05 07:55:32.436710] [info] [000011] /src/core/config_manager/ConfigManagerBase.cpp:2138 invalid aliuid conf dir:/usr/local/ilogtail/./users error:No such file or directory [2022-12-05 07:55:32.436759] [info] [000011] /src/core/config_manager/ConfigManagerBase.cpp:2145 recreate aliuid conf dir success:/usr/local/ilogtail/./users [2022-12-05 07:55:32.447413] [info] [000011] /src/core/config_manager/ConfigManagerBase.cpp:2398 user yaml config file loaded:/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml [2022-12-05 07:55:32.447427] [info] [000011] /src/core/config_manager/ConfigManagerBase.cpp:2407 user yaml config removed or added, last:0 now:1 [2022-12-05 07:55:32.447448] [info] [000011] /src/core/config_manager/ConfigManagerBase.cpp:2013 load user defined id from env:default [2022-12-05 07:55:32.447688] [info] [000011] /src/core/config_manager/ConfigYamlToJson.cpp:245 Trans yaml to json:success config_name:/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml is_file_mode:false input_plugin_type:service_docker_stdout has_accelerate_processor:false accelerate_processor_plugin_type: log_split_processor: logtype:plugin user_local_json_config:{ "metrics" : { "config#/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml" : { "enable" : true, "log_type" : "plugin", "plugin" : { "flushers" : [ { "detail" : { "Brokers" : [ "kafka-cluster:9092" ], "Topic" : "access-log" }, "type" : "flusher_kafka" } ], "inputs" : [ { "detail" : { "IncludeK8sLabel" : { "app" : "nginx" }, "Stderr" : false, "Stdout" : true }, "type" : "service_docker_stdout" } ] } } } }

[2022-12-05 07:55:32.447930] [info] [000011] /src/core/config_manager/ConfigManagerBase.cpp:1070 load user_yaml_config.d config:true file:/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml now config count:1 [2022-12-05 07:55:32.448615] [info] [000011] /src/core/monitor/Monitor.cpp:537 machine cpu cores:64 [2022-12-05 07:55:32.449196] [info] [000011] /src/core/sls_control/SLSControl.cpp:65 user agent:ilogtail/1.3.0 (Linux; 4.15.0; x86_64) ip/10.3.0.14 env/K8S-Daemonset [2022-12-05 07:55:32.449205] [info] [000011] /src/core/sender/Sender.cpp:453 Set sender queue param depend value:10 [2022-12-05 07:55:32.449450] [info] [000019] /src/core/sender/Sender.cpp:1264 SendThread:start [2022-12-05 07:55:32.449511] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:307 load plugin base, config count:1 docker env config:false dl file:/usr/local/ilogtail/libPluginAdapter.so [2022-12-05 07:55:32.449620] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:328 check plugin adapter version success, version:300 [2022-12-05 07:55:32.593219] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:439 init plugin base:success [2022-12-05 07:55:33.369049] [warning] [000011] /src/core/plugin/LogtailPlugin.cpp:84 msg:load plugin error project: logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml content:{ "flushers" : [ { "detail" : { "Brokers" : [ "kafka-cluster:9092" ], "Topic" : "access-log" }, "type" : "flusher_kafka" } ], "inputs" : [ { "detail" : { "IncludeK8sLabel" : { "app" : "nginx" }, "Stderr" : false, "Stdout" : true }, "type" : "service_docker_stdout" } ] } result:1 [2022-12-05 07:55:33.369252] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:116 logtail plugin Resume:start [2022-12-05 07:55:33.369358] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:118 logtail plugin Resume:success [2022-12-05 07:55:33.369416] [info] [000011] /src/core/common/DynamicLibHelper.cpp:104 load glibc dynamic library:begin [2022-12-05 07:55:33.369454] [info] [000011] /src/core/common/DynamicLibHelper.cpp:118 load glibc dynamic library:success [2022-12-05 07:55:33.369579] [info] [000011] /src/core/checkpoint/CheckPointManager.cpp:154 load checkpoint, version:200 file check point:0 dir check point:0 [2022-12-05 07:55:33.369639] [info] [000011] /src/core/profiler/LogIntegrity.cpp:507 no integrity file to load:/usr/local/ilogtail/logtail_integrity_snapshot.json [2022-12-05 07:55:33.369669] [info] [000011] /src/core/profiler/LogLineCount.cpp:263 no line count file to load:/usr/local/ilogtail/logtail_line_count_snapshot.json [2022-12-05 07:55:33.370027] [info] [000011] /src/core/logtail.cpp:282 Logtail started, appInfo:{ "UUID" : "3186E44C-7472-11ED-A315-AE84100FC2D2", "build_date" : "20221202", "compiler" : "GCC 4.8.5", "git_hash" : "f900f5a7c94d9c531b9259143e40675877b59808", "hostname" : "k8s-master", "instance_id" : "3186DA9C-7472-11ED-802E-AE84100FC2D2_10.3.0.14_1670226932", "ip" : "10.3.0.14", "logtail_version" : "1.3.0 Community Edition", "os" : "Linux; 4.15.0-197-generic; #208-Ubuntu SMP Tue Nov 1 17:23:37 UTC 2022; x86_64", "update_time" : "2022-12-05 07:55:33" }

[2022-12-05 07:55:33.370135] [info] [000011] /src/core/controller/EventDispatcherBase.cpp:588 start add existed check point events, size:0 [2022-12-05 07:55:33.370143] [info] [000011] /src/core/controller/EventDispatcherBase.cpp:602 add existed checkpoint events, size:0 cache size:0 event size:0 delete size:0 [2022-12-05 07:55:33.370265] [info] [000052] /src/core/polling/PollingModify.cpp:224 PollingModify::Polling:start [2022-12-05 07:55:33.370491] [info] [000055] /src/core/processor/LogProcess.cpp:205 local timezone offset second:0

• logtail_plugin.LOG:

2022-12-05 07:55:32 [INF] [plugin_export.go:250] [setGCPercentForSlowStart] set startup GC percent from 30 to 20 2022-12-05 07:55:32 [INF] [plugin_export.go:234] [func1] init plugin base, version:0.1.0 2022-12-05 07:55:32 [INF] [global_config.go:55] [func1] load global config:{ "HostIP" : "10.3.0.14", "Hostname" : "k8s-master", "LogtailSysConfDir" : "/usr/local/ilogtail/./" }

2022-12-05 07:55:32 [INF] [plugin_manager.go:96] [Init] init plugin, local env tags:[node_name k8s-master node_ip 10.3.0.14] 2022-12-05 07:55:32 [INF] [checkpoint_manager.go:96] [Init] init checkpoint: 2022-12-05 07:55:32 [INF] [logstore_config.go:799] [loadBuiltinConfig] load built-in config statistics, config name: shennong_log_profile, logstore: logtail_plugin_profile 2022-12-05 07:55:32 [INF] [logstore_config.go:799] [loadBuiltinConfig] load built-in config alarm, config name: logtail_alarm, logstore: logtail_alarm 2022-12-05 07:55:32 [INF] [logstore_config.go:788] [LoadLogstoreConfig] load config:config#/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml logstore: 2022-12-05 07:55:32 [WRN] [flusher_kafka.go:65] [Init] [config#/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml,] AlarmType:FLUSHER_INIT_ALARM SASL information is not set, access Kafka server without authentication:logstore :config config#/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml: 2022-12-05 07:55:33 [ERR] [flusher_kafka.go:90] [Init] [config#/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml,] AlarmType:FLUSHER_INIT_ALARM init kafka flusher fail, error:kafka: client has run out of available brokers to talk to (Is your cluster reachable?) logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml 2022-12-05 07:55:33 [ERR] [plugin_export.go:92] [LoadConfig] AlarmType:CONFIG_LOAD_ALARM load config error, project: logstore: config:config#/usr/local/ilogtail/./user_yaml_config.d/cloud_stdout.yaml error:kafka: client has run out of available brokers to talk to (Is your cluster reachable?) 2022-12-05 07:55:33 [INF] [plugin_export.go:154] [Resume] Resume:start 2022-12-05 07:55:33 [INF] [logstore_config.go:146] [Start] [shennong_log_profile,logtail_plugin_profile] config start:begin 2022-12-05 07:55:33 [INF] [logstore_config.go:184] [Start] [shennong_log_profile,logtail_plugin_profile] config start:success 2022-12-05 07:55:33 [INF] [logstore_config.go:146] [Start] [logtail_alarm,logtail_alarm] config start:begin 2022-12-05 07:55:33 [INF] [logstore_config.go:184] [Start] [logtail_alarm,logtail_alarm] config start:success 2022-12-05 07:55:33 [INF] [checkpoint_manager.go:115] [Resume] checkpoint:Resume 2022-12-05 07:55:33 [INF] [plugin_export.go:162] [Resume] Resume:success 2022-12-05 07:55:33 [INF] [metric_wrapper.go:42] [Run] [shennong_log_profile,logtail_plugin_profile] start run metric :&{0xc0007cae40} 2022-12-05 07:55:33 [INF] [metric_wrapper.go:42] [Run] [logtail_alarm,logtail_alarm] start run metric :&{0xc0007cafc0}

Describe the bug A clear and concise description of what the bug is, ideally within 20 words. 容器文件日志收集，一个日志文件打开多个文件描述符，最后触发以下报错： [2022-09-21 04:36:24.752141] [warning] [000033] /src/core/event_handler/EventHandler.cpp:582 too many open files:skip this read operation log path:/logtail_host/xxxx/warning-2022-09-20.log [2022-09-21 04:36:24.752146] [error] [000033] /src/core/reader/LogFileReader.cpp:1007 log file reader fd limit, too many open files:/logtail_host/xxxx/warning-2022-09-21.log : limit:25000 查看文件描述符打开数量 [root@ip-10-53-0-57 ilogtail]# ls -l /proc/11/fd |wc -l 25057 查看/proc/11/fd 内容，可以看到多个文件描述符指向通一个文件 lr-x------ 1 root root 64 Sep 21 01:46 17355 -> /logtail_host/xxxx/warning-2022-09-21.log lr-x------ 1 root root 64 Sep 21 01:46 17382 -> /logtail_host/xxxx/warning-2022-09-21.log lr-x------ 1 root root 64 Sep 21 01:46 17401 -> /logtail_host/xxxx/warning-2022-09-21.log .... 每次check container path update flag:true ，文件描述符使用就增长 [2022-09-21 04:42:45.520378] [info] [000011] /src/core/config_manager/ConfigManagerBase.cpp:2642 check container path update flag:true size:1 [2022-09-21 04:42:45.520394] [info] [000011] /src/core/observer/network/NetworkObserver.cpp:54 hold on:observer [2022-09-21 04:42:45.520408] [info] [000011] /src/core/controller/EventDispatcherBase.cpp:1137 main thread:start update config [2022-09-21 04:42:45.525535] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:98 logtail plugin HoldOn:start [2022-09-21 04:42:45.529024] [info] [000011] /src/core/plugin/LogtailPlugin.cpp:102 logtail plugin HoldOn:success cost:4 iLogtail Running Environment Please provide the following information:

• ilogtail version: 1.1.1

• Yaml configuration:

• ilogtail.LOG: [2022-09-21 04:36:24.752141] [warning] [000033] /src/core/event_handler/EventHandler.cpp:582 too many open files:skip this read operation log path:/logtail_host/xxxx/warning-2022-09-20.log [2022-09-21 04:36:24.752146] [error] [000033] /src/core/reader/LogFileReader.cpp:1007 log file reader fd limit, too many open files:/logtail_host/xxxx/warning-2022-09-21.log : limit:25000

• logtail_plugin.LOG:

运行平台：arm64 运行版本：1.0.30

现象：system启动之后只看到一个ilogtail进程，正常都是2个,启动之后会存在时间跳变(可能向前跳变，也可能向后跳变)，不知道是否有影响？进系统后restart service又正常了，var/log/syslog里没看到错误

● bmilogtail.service - ilogtail service Loaded: loaded (/lib/systemd/system/bmilogtail.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2021-12-10 17:15:55 CST; 7 months 13 days ago Main PID: 1284 (ilogtail) Tasks: 1 (limit: 3217) CGroup: /system.slice/bmilogtail.service └─1284 /var/ilogtail/ilogtail

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.`

Describe the bug A clear and concise description of what the bug is, ideally within 20 words. collect host meta data with metric_meta_host does not work.

To Reproduce Please post your configurations and environments to reproduce the bug.

enable: true inputs:

• Type: metric_meta_host CPU: true Memory: false Net: false Disk: false Process: false ProcessIntervalRound: 60 ProcessNamesRegex:
  • ilogtail Labels: cluster: ilogtail-test-cluster flushers:
• Type: flusher_stdout OnlyStdout: true

start ilogtail. error in ilogtail.LOG

[2022-07-22 21:20:28.224154] [error] [015305] /src/core/plugin/LogtailPlugin.cpp:490 init plugin base error:1

The developer team will put a higher priority on bugs that can be reproduced. If you want a prompt reply, please keep your descriptions detailed.

• Your ilogtail version:

• 1.1.0

• Your platform:

• linux

• Your configuration:

• enable: true inputs:

  • Type: metric_meta_host CPU: true Memory: false Net: false Disk: false Process: false ProcessIntervalRound: 60 ProcessNamesRegex:
    • ilogtail Labels: cluster: ilogtail-test-cluster flushers:
  • Type: flusher_stdout OnlyStdout: true

• Your ilogtail logs:

• [2022-07-22 21:20:28.224154] [error] [015305] /src/core/plugin/LogtailPlugin.cpp:490 init plugin base error:1

Additional comments If possible, please also consider attaching the output of ilogtail diagnose tool. This produces detailed environment information and hopefully helps us diagnose faster.

If you have local commits (e.g. compile fixes before you reproduce the bug), please make sure you first make a PR to fix the build errors and then report the bug.

错误日志如下： [2022-04-25 15:51:56.031582] [error] [001268] /build/logtail/controller/EventDispatcher.cpp:1651 Bind domain socket:Failed 98:Address already in use File Descriptor:23

请问这个端口是否可以配置解决报错问题？

测试kafka的写入，使用不启用任何安全认证的kafka，按照样例，能正常写入。

而另一个kafka集群带安全认证，flusher里设置了SASLUsername，SASLPassword，但是写入失败。 logtail_plugin.LOG 里报错 error:kafka: client has run out of available brokers to talk to (Is your cluster reachable?) 此集群的要求设置 security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 这样的情况该如何在flusher中设置呢？

另外，ilogtail的相关文档和案例过于缺少。

感谢之前的回复：https://github.com/alibaba/ilogtail/issues/127

不过我是自己安装并配置的iLogtail在服务器上采集日志往kafka发送的，没有使用日志服务SLS，请问这样的情况怎么处理呢？ 当我使用了processor_split_log_regex插件后，kafka中收到的数据没有了__tag__:__path__字段。

实现新的 metrics 数据模型 (参考 <Metrics 数据模型改进设计>提案 https://github.com/alibaba/ilogtail/discussions/518 )，同时修改 Plugins Interface 来支持新的数据模型。 流水线改动

1. MetricInput ServiceInput Processor Aggregator Flusher 插件接口修改，显式区分 SlsPlugins 和 PipelinePlugins 如 Processor

// Processor also can be a filter
type Processor interface {
	// Init called for init some system resources, like socket, mutex...
	Init(Context) error

	// Description returns a one-sentence description on the Input
	Description() string
}

type PipelineProcessor interface {
	Processor
	Process(in *models.PipelineGroupEvents, context PipelineContext)
}

type SlsProcessor interface {
	Processor
	// Apply the filter to the given metric
	ProcessLogs(logArray []*protocol.Log) []*protocol.Log
}

2. Input Processor Flusher 可以同时实现 SlsPlugins 和 PipelinePlugins 的接口函数，来让插件兼容新旧数据模型。比如 metric_mock input 插件的修改

// 处理 sls log pb 数据
func (r *InputMock) CollectLogs(collector ilogtail.Collector) error {
	if r.MockSlsLogFormat {
                ...
	    collector.AddData(....)
	}
	return nil
}

// 处理 metrics 数据
func (r *InputMock) Collect(context ilogtail.PipelineContext) error {
        if r.MockMetricEventFormat {
	       singleValue := models.NewSingleValueMetric("single_metrics_mock", models.MetricTypeCounter, models.NewTagsWithMap(r.Tags), time.Now().UnixNano(), r.Index)

		values := models.NewMetricMultiValueValues()
		typedValues := models.NewMetricTypedValues()
		multiValues := models.NewMetric("multi_values_metrics_mock", models.MetricTypeUntyped, models.NewTagsWithMap(r.Tags), time.Now().UnixNano(), values, typedValues)
		context.Collector().Collect(group, singleValue, multiValues)
	}
	return nil
}

3. Aggregator 插件会保存数据的聚合状态，新旧模型不做兼容处理，需要分别实现
4. 现有的所有插件可以继续工作，但只能处理 sls.log pb 协议的数据，对 metrics 数据需要做兼容处理或者重新实现

ilogtail configmap配置如下： apiVersion: v1 items:

• apiVersion: v1 data: regex_log.yaml: | enable: true inputs: - Type: file_log LogPath: "/tmp/logtest/" FilePattern: ".log" ContainerInfo: K8sNamespaceRegex: default K8sPodRegex: ^(nginx.)$ flushers: - Type: flusher_stdout OnlyStdout: true kind: ConfigMap

被采集pod如下： root@010015001074:/opt# kubectl exec -it nginx-86bc475f6f-kr6zb sh cd /tmp/logtest ls -l total 8 -rw-r--r-- 1 root root 29 Aug 30 00:27 a.log -rw-r--r-- 1 root root 6 Aug 30 00:32 b.log

ilogtail pod标准输出日志如下： root@010015001074:/opt# kubectl logs ilogtail-ds-bplgx -n ilogtail
delay stop seconds: 0 ilogtail started. pid: 11 register fun v2 0xa34f3c 0xa34f86 0xa34fdc 0xa35576 2022/08/29 10:23:20 DEBUG Now using Go's stdlib log package (via loggers/mappers/stdlib). load log config /usr/local/ilogtail/plugin_logger.xml recover stdout recover stderr

如上，没有采集到对应日志的内容

Describe the bug When dlsym fails, close wrong pointer of shared library as pointer mismatch

iLogtail Running Environment Please provide the following information:

• ilogtail version:

  • dlopen with mPluginAdapterPtr : https://github.com/alibaba/ilogtail/blob/main/core/plugin/LogtailRuntimePlugin.cpp#L98
  • dlclose with mPluginBasePtr when add symbol fails : https://github.com/alibaba/ilogtail/blob/main/core/plugin/LogtailRuntimePlugin.cpp#L109

• Yaml configuration:

• ilogtail.LOG:

• logtail_plugin.LOG:

现象: 想用日志[2022-03-09 18:00:00] 来分割.但是没有实现.

[root@rh79 bin]# ./ilogtail register fun v2 0xa34f3a 0xa34f84 0xa34fda 0xa35574 2022/08/12 10:01:55 DEBUG Now using Go's stdlib log package (via loggers/mappers/stdlib). load log config /usr/local/ilogtail-1.1.0/bin/plugin_logger.xml recover stderr recover stdout

2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] xxx1\nyyyyy\nzzzzzz\n","time":"1660269753"} 2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:01] xxx2\nyyyyy\nzzzzzz\n","time":"1660269753"} 2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] line1\nyayyyy\nzzzzzz\n","time":"1660269753"} 2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:01] line2\nyyyyy\nzzzzzz\n","time":"1660269753"} 2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] xxx1\nyyyyy\nzzzzzz\n","time":"1660269753"} 2022-08-12 10:02:34 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:01] xxx2\nyyyyy\nzzzzzz","time":"1660269753"} 2022-08-12 10:03:13 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] xxx1\nyyyyy\nzzzzzz\n","time":"1660269792"} 2022-08-12 10:03:13 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:01] xxx2\nyyyyy\nzzzzzz","time":"1660269792"} 2022-08-12 10:04:37 {"tag:path":"./multiline.log","content":"[2022-03-03 18:00:00] xxx1\nyyyyy\nzzzzzz\n*************\nFri Aug 12 09:56:05 2022[2022-03-03 18:00:01] xxx2\nyyyyy\nzzzzzz","time":"1660269875"} 2022-08-12 10:06:01 {"tag:path":"./multiline.log","content":"[2022-03-09 18:00:00]\nFri Aug 12 09:56:05 2022\nARC2 started with pid=20, OS id=12957 \nARC1: Archival started\nARC2: Archival started\nARC1: Becoming the 'no FAL' ARCH\nARC1: Becoming the 'no SRL' ARCH\nARC2: Becoming the heartbeat ARCH\nFri Aug 12 09:56:05 2022\nARC3 started with pid=21, OS id=12959 \nARC3: Archival started\nARC0: STARTING ARCH PROCESSES COMPLETE\n[12951] Successfully onlined Undo Tablespace 2.\nUndo initialization finished serial:0 start:200583234 end:200583274 diff:40 (0 seconds)\nVerifying file header compatibility for 11g tablespace encryption..\nVerifying 11g file header compatibility for tablespace encryption completed\n[2022-03-09 18:00:02]\nQMNC started with pid=22, OS id=12961 \nCompleted: ALTER DATABASE OPEN\nStarting background process CJQ0\nFri Aug 12 09:56:05 2022\nCJQ0 started with pid=24, OS id=12974","time":"1660269958"}

配置文件:

[root@rh79 bin]# cat yaml/file_multiline.yaml enable: true inputs:

• Type: file_log LogPath: . FilePattern: multiline.log processors:
• Type: processor_split_log_regex SplitRegex: [\d+-\d+-\d+\s\d+:\d+:\d+]\s.* SplitKey: content PreserveOthers: true flushers:
• Type: flusher_stdout OnlyStdout: true

测试文本 (oracle 告警日志,手动加入了[2022-03-09 18:00:00] 以分割)

[root@rh79 bin]# cat b.txt > multiline.log [root@rh79 bin]# cat b.txt [2022-03-09 18:00:00] Fri Aug 12 09:56:05 2022 ARC2 started with pid=20, OS id=12957 ARC1: Archival started ARC2: Archival started ARC1: Becoming the 'no FAL' ARCH ARC1: Becoming the 'no SRL' ARCH ARC2: Becoming the heartbeat ARCH Fri Aug 12 09:56:05 2022 ARC3 started with pid=21, OS id=12959 ARC3: Archival started ARC0: STARTING ARCH PROCESSES COMPLETE Undo initialization finished serial:0 start:200583234 end:200583274 diff:40 (0 seconds) Verifying file header compatibility for 11g tablespace encryption.. Verifying 11g file header compatibility for tablespace encryption completed [2022-03-09 18:00:02] QMNC started with pid=22, OS id=12961 Completed: ALTER DATABASE OPEN Starting background process CJQ0 Fri Aug 12 09:56:05 2022 CJQ0 started with pid=24, OS id=12974

Concisely describe the proposed feature kafka input插件依赖github.com/bsm/sarama-cluster的库已经被放弃维护，历史原因是sarama不支持consumer group，但是新的sarama已经支持consumer group。当前只用的sarama-cluster也就是限制出了sarama无法升级，升级后无法编译通过，如果用户有高于kafka 2.7版本服务都无法支持。去掉zk依赖的高版本kafka未来也会企业升级落地的主流

最新的版本中因为vendor剔除，原始修改sarama-cluster中代码bug的https://github.com/alibaba/ilogtail/pull/472 也丢了，#471 提到的问题新版本仍然被带了回来，因此目前看也是需要升级修改的。 Describe the solution you'd like (if any) 在当前不更换kafka sarama这个客服端的情况下，直接采用sarama高版本的api实现kafka input，改动不基本不涉及到对外的用户配置改动。 实现时需要主要参考官方的https://github.com/Shopify/sarama/blob/main/examples/consumergroup/main.go 代码，尤其是里面的下面代码片段需要注意，否则可能导致rebanlance无法被感知到

go func() {
		defer wg.Done()
		for {
			// `Consume` should be called inside an infinite loop, when a
			// server-side rebalance happens, the consumer session will need to be
			// recreated to get the new claims
			if err := client.Consume(ctx, strings.Split(topics, ","), &consumer); err != nil {
				log.Panicf("Error from consumer: %v", err)
			}
			// check if context was cancelled, signaling that the consumer should stop
			if ctx.Err() != nil {
				return
			}
			consumer.ready = make(chan bool)
		}
	}()

Additional comments

• feat: add metadata group aggregator

• feat: http server input support raw format and metadata params (QueryParams & HeaderParams)

• feat: http flusher support raw protocol

• feat: stdout flusher support events of byteArray type

• fix: fix observePipeCollector.ToArray and keyValuesImpl.Contains