Enterprise Network Flow Collector (IPFIX, sFlow, Netflow)

Overview

vFlow

Build Status Go Report Card GoDev

High-performance, scalable and reliable IPFIX, sFlow and Netflow collector (written in pure Golang).

Features

  • IPFIX RFC7011 collector
  • sFLow v5 raw header / counters collector
  • Netflow v5 collector
  • Netflow v9 collector
  • Decoding sFlow raw header L2/L3/L4
  • Produce to Apache Kafka, NSQ, NATS
  • Replicate IPFIX and sFlow to 3rd party collector
  • Supports IPv4 and IPv6
  • Prometheus and RESTful APIs monitoring

Alt text

Documentation

Decoded IPFIX data

The IPFIX data decodes to JSON format and IDs are IANA IPFIX element ID

{"AgentID":"192.168.21.15","Header":{"Version":10,"Length":420,"ExportTime":1483484642,"SequenceNo":1434533677,"DomainID":32771},"DataSets":[[{"I":8,"V":"192.16.28.217"},{"I":12,"V":"180.10.210.240"},{"I":5,"V":2},{"I":4,"V":6},{"I":7,"V":443},{"I":11,"V":64381},{"I":32,"V":0},{"I":10,"V":811},{"I":58,"V":0},{"I":9,"V":24},{"I":13,"V":20},{"I":16,"V":4200000000},{"I":17,"V":27747},{"I":15,"V":"180.105.10.210"},{"I":6,"V":"0x10"},{"I":14,"V":1113},{"I":1,"V":22500},{"I":2,"V":15},{"I":52,"V":63},{"I":53,"V":63},{"I":152,"V":1483484581770},{"I":153,"V":1483484622384},{"I":136,"V":2},{"I":243,"V":0},{"I":245,"V":0}]]}

Decoded sFlow data

{"Version":5,"IPVersion":1,"AgentSubID":5,"SequenceNo":37591,"SysUpTime":3287084017,"SamplesNo":1,"Samples":[{"SequenceNo":1530345639,"SourceID":0,"SamplingRate":4096,"SamplePool":1938456576,"Drops":0,"Input":536,"Output":728,"RecordsNo":3,"Records":{"ExtRouter":{"NextHop":"115.131.251.90","SrcMask":24,"DstMask":14},"ExtSwitch":{"SrcVlan":0,"SrcPriority":0,"DstVlan":0,"DstPriority":0},"RawHeader":{"L2":{"SrcMAC":"58:00:bb:e7:57:6f","DstMAC":"f4:a7:39:44:a8:27","Vlan":0,"EtherType":2048},"L3":{"Version":4,"TOS":0,"TotalLen":1452,"ID":13515,"Flags":0,"FragOff":0,"TTL":62,"Protocol":6,"Checksum":8564,"Src":"10.1.8.5","Dst":"161.140.24.181"},"L4":{"SrcPort":443,"DstPort":56521,"DataOffset":5,"Reserved":0,"Flags":16}}}}],"IPAddress":"192.168.10.0"}

Decoded Netflow v5 data

{"AgentID":"114.23.3.231","Header":{"Version":5,"Count":3,"SysUpTimeMSecs":51469784,"UNIXSecs":1544476581,"UNIXNSecs":0,"SeqNum":873873830,"EngType":0,"EngID":0,"SmpInt":1000},"Flows":[{"SrcAddr":"125.238.46.48","DstAddr":"114.23.236.96","NextHop":"114.23.3.231","Input":791,"Output":817,"PktCount":4,"L3Octets":1708,"StartTime":51402145,"EndTime":51433264,"SrcPort":49233,"DstPort":443,"Padding1":0,"TCPFlags":16,"ProtType":6,"Tos":0,"SrcAsNum":4771,"DstAsNum":56030,"SrcMask":20,"DstMask":22,"Padding2":0},{"SrcAddr":"125.238.46.48","DstAddr":"114.23.236.96","NextHop":"114.23.3.231","Input":791,"Output":817,"PktCount":1,"L3Octets":441,"StartTime":51425137,"EndTime":51425137,"SrcPort":49233,"DstPort":443,"Padding1":0,"TCPFlags":24,"ProtType":6,"Tos":0,"SrcAsNum":4771,"DstAsNum":56030,"SrcMask":20,"DstMask":22,"Padding2":0},{"SrcAddr":"210.5.53.48","DstAddr":"103.22.200.210","NextHop":"122.56.118.157","Input":564,"Output":802,"PktCount":1,"L3Octets":1500,"StartTime":51420072,"EndTime":51420072,"SrcPort":80,"DstPort":56108,"Padding1":0,"TCPFlags":16,"ProtType":6,"Tos":0,"SrcAsNum":56030,"DstAsNum":13335,"SrcMask":24,"DstMask":23,"Padding2":0}]}

Decoded Netflow v9 data

{"AgentID":"10.81.70.56","Header":{"Version":9,"Count":1,"SysUpTime":357280,"UNIXSecs":1493918653,"SeqNum":14,"SrcID":87},"DataSets":[[{"I":1,"V":"0x00000050"},{"I":2,"V":"0x00000002"},{"I":4,"V":2},{"I":5,"V":192},{"I":6,"V":"0x00"},{"I":7,"V":0},{"I":8,"V":"10.81.70.56"},{"I":9,"V":0},{"I":10,"V":0},{"I":11,"V":0},{"I":12,"V":"224.0.0.22"},{"I":13,"V":0},{"I":14,"V":0},{"I":15,"V":"0.0.0.0"},{"I":16,"V":0},{"I":17,"V":0},{"I":21,"V":300044},{"I":22,"V":299144}]]}

Supported platform

  • Linux
  • Windows

Build

Given that the Go Language compiler (version 1.14.x preferred) is installed, you can build it with:

go get github.com/EdgeCast/vflow/vflow
cd $GOPATH/src/github.com/EdgeCast/vflow

make build
or
cd vflow; go build 

Installation

You can download and install pre-built debian package as below (RPM and Linux binary are available).

dpkg -i vflow-0.9.0-x86_64.deb

Once you installed you need to configure the below files, for more information check configuration guide:

/etc/vflow/vflow.conf
/etc/vflow/mq.conf

You can start the service by the below:

service vflow start

Kubernetes

kubectl apply -f https://github.com/EdgeCast/vflow/blob/master/kubernetes/deploy.yaml

Docker

docker run -d -p 2181:2181 -p 9092:9092 spotify/kafka
docker run -d -p 4739:4739 -p 4729:4729 -p 6343:6343 -p 8081:8081 -e VFLOW_KAFKA_BROKERS="172.17.0.1:9092" mehrdadrad/vflow

License

Licensed under the Apache License, Version 2.0 (the "License")

Contribute

Welcomes any kind of contribution, please follow the next steps:

  • Fork the project on github.com.
  • Create a new branch.
  • Commit changes to the new branch.
  • Send a pull request.
Comments
  • Slack link is broken in README

    Slack link is broken in README

    I have a question about logging in the docker container and was going to use the slack link, but it redirects and says the shared join link is no longer valid. Can you please fix so I can ask my question there?

    Thanks.

    opened by punisherVX 10
  • latest version from git

    latest version from git "operation not supported" when running

    Hi Just trying to install and run configs: cat /etc/vflow/vflow.conf ipfix-workers: 600 sflow-workers: 300 sflow-port: 9999 stats-http-port: 10080 log-file: /var/log/vflow.log

    cat /etc/vflow/mq.conf brokers: - 127.0.0.1:9092 retry-max: 1 retry-backoff: 30

    netstat -na | grep 9092 netstat -na | grep 9092 tcp6 0 0 :::9092 :::* LISTEN

    logs for vflow with -verbose show no errors - last in log: [vflow] 2017/11/06 19:51:20 netflow v9 is running (UDP: listening on [::]:4729 workers#: 200) [vflow] 2017/11/06 19:51:20 ipfix is running (UDP: listening on [::]:4739 workers#: 600)

    opened by paolss 10
  • Swap sarama for segmentio kafka library

    Swap sarama for segmentio kafka library

    The kafka library from segmentio provides the same kafka connectivity capability in a smaller package size. According to the authors it should also use less memory. Also included is functionality to support the use of a bootstrap server ( a dns record that points to available brokers ).

    opened by awillis 9
  • Can not decode sflow data

    Can not decode sflow data

    • I build with the git master branch

    run command

    ./vflow -config vflow.conf -mqueue-conf mq.conf  \
                     -sflow-max-udp-size 100000 -sflow-port 6343
    

    vflow.conf

    cat vflow.conf
    sflow-workers: 1
    log-file: /var/log/vflow.log
    verbose: true
    mq-name: kafka
    ipfix-enabled: false
    netflow9-enabled: false
    sflow-topic: vflow
    

    mq.conf

    brokers:
      - xxxxx.cn:9092
    

    output log

    [vflow] 2018/04/26 00:50:49 rcvd sflow data from: xxxx:6343, size: 1396 bytes
    [vflow] 2018/04/26 00:50:49 rcvd sflow data from: xxxx:6343, size: 1268 bytes
    [vflow] 2018/04/26 00:50:49 rcvd sflow data from: xxxx:6343, size: 1220 bytes
    [vflow] 2018/04/26 00:50:49 rcvd sflow data from: xxxx:6343, size: 1312 bytes
    

    monitor

    {
        "UDPQueue": 0,
        "MessageQueue": 0,
        "UDPCount": 750,
        "DecodedCount": 0,
        "MQErrorCount": 0,
        "Workers": 1
    }
    

    tcpdump of sflow sending

    01:09:45.347017 IP xxxxx.sflow > xxxxx.sflow: sFlowv5, IPv4 agent xxxxx.com, agent-id 8, length 1216
    01:09:45.482522 IP xxxxx.com.sflow > 1xxxxw: sFlowv5, IPv4 agent xxxx, agent-id 8, length 1372
    

    Problem

    • No one sflow msg was decoded, and I cann't tell the reason.
    opened by jackpgao 9
  • Netflow: fatal error: runtime: out of memory

    Netflow: fatal error: runtime: out of memory

    Hello All,

    i get "fatal error: runtime: out of memory" after some minutes vflow is running and this is my vflow.conf:

    netflow9-workers: 50 log-file: /var/log/vflow.log ipfix-tpl-cache-file: /usr/local/vflow/vflow.templates netflow9-topic: anomaly ipfix-enabled: false sflow-enabled: false dynamic-workers: false

    opened by franks231 9
  • netflow v9 some templates not being parsed - Multiple errors: can not read the data

    netflow v9 some templates not being parsed - Multiple errors: can not read the data

    I'm having trouble with some v9 templates not being parsed from a Juniper SRX.. Some are, and some aren't.. As an example below - template id# 261 seems to fail to be defined, even though its definition gets transmitted 60 seconds by the sending router. First some background:

    • sending device - SRX210HE JUNOS Software Release [12.1X46-D67]
    • receiving devices (similar results from both)
      • smartos - go version go1.8 solaris/amd64 - joyent_20170928T144204Z
      • debian stretch - go version go1.7.4 linux/amd64 - Linux version 4.9.0-4-amd64 ([email protected]) (gcc version 6.3.0 20170516 (Debian 6.3.0-18) ) #1 SMP Debian 4.9.51-1 (2017-09-28)
    • built from git (today), VERSION=0.4.1 in Makefile (although interestingly that version doesn't populate into the version [vflow] 2017/12/03 14:31:45 Welcome to vFlow v.unknown Apache License 2.0)

    Router config:

    set services flow-monitoring version9 template template_1 ipv4-template
    set services flow-monitoring version9 template template_2 ipv6-template
    set forwarding-options sampling input rate 1
    set forwarding-options sampling family inet output flow-inactive-timeout 30
    set forwarding-options sampling family inet output flow-active-timeout 60
    set forwarding-options sampling family inet output flow-server 10.232.6.89 port 4729
    set forwarding-options sampling family inet output flow-server 10.232.6.89 version9 template template_1
    set forwarding-options sampling family inet output inline-jflow source-address 10.232.4.5
    set forwarding-options sampling family inet6 output flow-inactive-timeout 30
    set forwarding-options sampling family inet6 output flow-active-timeout 60
    set forwarding-options sampling family inet6 output flow-server 10.232.6.89 port 4729
    set forwarding-options sampling family inet6 output flow-server 10.232.6.89 version9 template template_2
    set forwarding-options sampling family inet6 output inline-jflow source-address 10.232.4.5
    
    

    From the vflow.log file.. The "can not read data" appears every minute.

    [vflow] 2017/11/28 11:10:16 Multiple errors:
    - 10.232.4.5 unknown netflow template id# 261
    - can not read the data
    - can not read the data
    - can not read the data
    [vflow] 2017/11/28 11:10:16 rcvd netflow v9 data from: 10.232.4.5:63651, size: 144 bytes
    [vflow] 2017/11/28 11:10:16 Multiple errors:
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    - can not read the data
    [vflow] 2017/11/28 11:10:18 rcvd netflow v9 data from: 10.232.4.5:58938, size: 144 bytes
    [vflow] 2017/11/28 11:10:18 10.232.4.5 unknown netflow template id# 261
    

    (After a few days it also crashes with a backtrace that I haven't really had a look at yet)

    To me it looks like the definition of template 261 is being sent every minute, along with a few flows using the same template id.. Here's a packet.

    Frame 16: 618 bytes on wire (4944 bits), 618 bytes captured (4944 bits) on interface 0
        Interface id: 0 (net0)
        Encapsulation type: Ethernet (1)
        Arrival Time: Dec  3, 2017 15:02:37.876916000 AEDT
        [Time shift for this packet: 0.000000000 seconds]
        Epoch Time: 1512273757.876916000 seconds
        [Time delta from previous captured frame: 2.002214000 seconds]
        [Time delta from previous displayed frame: 2.002214000 seconds]
        [Time since reference or first frame: 19.046828000 seconds]
        Frame Number: 16
        Frame Length: 618 bytes (4944 bits)
        Capture Length: 618 bytes (4944 bits)
        [Frame is marked: False]
        [Frame is ignored: False]
        [Protocols in frame: eth:ethertype:ip:udp:cflow]
    Ethernet II, Src: JuniperN_cb:2f:01 (80:71:1f:cb:2f:01), Dst: 82:c5:5d:98:41:4a (82:c5:5d:98:41:4a)
        Destination: 82:c5:5d:98:41:4a (82:c5:5d:98:41:4a)
            Address: 82:c5:5d:98:41:4a (82:c5:5d:98:41:4a)
            .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
            .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        Source: JuniperN_cb:2f:01 (80:71:1f:cb:2f:01)
            Address: JuniperN_cb:2f:01 (80:71:1f:cb:2f:01)
            .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
            .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
        Type: IPv4 (0x0800)
    Internet Protocol Version 4, Src: 10.232.4.5, Dst: 10.232.6.89
        0100 .... = Version: 4
        .... 0101 = Header Length: 20 bytes (5)
        Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
            0000 00.. = Differentiated Services Codepoint: Default (0)
            .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
        Total Length: 604
        Identification: 0x7864 (30820)
        Flags: 0x00
            0... .... = Reserved bit: Not set
            .0.. .... = Don't fragment: Not set
            ..0. .... = More fragments: Not set
        Fragment offset: 0
        Time to live: 63
        Protocol: UDP (17)
        Header checksum: 0xe0ff [validation disabled]
        [Header checksum status: Unverified]
        Source: 10.232.4.5
        Destination: 10.232.6.89
    User Datagram Protocol, Src Port: 54060, Dst Port: 4729
        Source Port: 54060
        Destination Port: 4729
        Length: 584
        Checksum: 0xfb15 [unverified]
        [Checksum Status: Unverified]
        [Stream index: 0]
    Cisco NetFlow/IPFIX
        Version: 9
        Count: 10
        SysUptime: 2613.569000000 seconds
        Timestamp: Dec  3, 2017 15:02:37.000000000 AEDT
            CurrentSecs: 1512273757
        FlowSequence: 372
        SourceId: 142
        FlowSet 1 [id=261]
            FlowSet Id: (Data) (261)
            FlowSet Length: 184
            Data (180 bytes), no template found
                [Expert Info (Warning/Malformed): Data (180 bytes), no template found]
                    [Data (180 bytes), no template found]
                    [Severity level: Warning]
                    [Group: Malformed]
        FlowSet 2 [id=1] (Options Template): 256
            FlowSet Id: Options Template(V9) (1)
            FlowSet Length: 24
            Options Template (Id = 256) (Scope Count = 1; Data Count = 2)
                Template Id: 256
                Option Scope Length: 4
                Option Length: 8
                Field (1/1) [Scope]: System
                    Scope Type: System (1)
                    Length: 0
                Field (1/2): SAMPLING_ALGORITHM
                    Type: SAMPLING_ALGORITHM (35)
                    Length: 1
                Field (2/2): SAMPLING_INTERVAL
                    Type: SAMPLING_INTERVAL (34)
                    Length: 4
            Padding: 0000
        FlowSet 3 [id=256] (1 flows)
            FlowSet Id: (Data) (256)
            FlowSet Length: 12
            [Template Frame: 16]
            Flow 1
                Sampling algorithm: Random sampling (2)
                Sampling interval: 1
            Padding: 000000
        FlowSet 4 [id=0] (Data Template): 261
            FlowSet Id: Data Template (V9) (0)
            FlowSet Length: 92
            Template (Id = 261, Count = 21)
                Template Id: 261
                Field Count: 21
                Field (1/21): IP_SRC_ADDR
                    Type: IP_SRC_ADDR (8)
                    Length: 4
                Field (2/21): IP_DST_ADDR
                    Type: IP_DST_ADDR (12)
                    Length: 4
                Field (3/21): IP_TOS
                    Type: IP_TOS (5)
                    Length: 1
                Field (4/21): PROTOCOL
                    Type: PROTOCOL (4)
                    Length: 1
                Field (5/21): L4_SRC_PORT
                    Type: L4_SRC_PORT (7)
                    Length: 2
                Field (6/21): L4_DST_PORT
                    Type: L4_DST_PORT (11)
                    Length: 2
                Field (7/21): ICMP_TYPE
                    Type: ICMP_TYPE (32)
                    Length: 2
                Field (8/21): INPUT_SNMP
                    Type: INPUT_SNMP (10)
                    Length: 4
                Field (9/21): SRC_MASK
                    Type: SRC_MASK (9)
                    Length: 1
                Field (10/21): DST_MASK
                    Type: DST_MASK (13)
                    Length: 1
                Field (11/21): SRC_AS
                    Type: SRC_AS (16)
                    Length: 4
                Field (12/21): DST_AS
                    Type: DST_AS (17)
                    Length: 4
                Field (13/21): BGP_NEXT_HOP
                    Type: BGP_NEXT_HOP (18)
                    Length: 4
                Field (14/21): TCP_FLAGS
                    Type: TCP_FLAGS (6)
                    Length: 1
                Field (15/21): OUTPUT_SNMP
                    Type: OUTPUT_SNMP (14)
                    Length: 4
                Field (16/21): IP_NEXT_HOP
                    Type: IP_NEXT_HOP (15)
                    Length: 4
                Field (17/21): BYTES
                    Type: BYTES (1)
                    Length: 4
                Field (18/21): PKTS
                    Type: PKTS (2)
                    Length: 4
                Field (19/21): FIRST_SWITCHED
                    Type: FIRST_SWITCHED (22)
                    Length: 4
                Field (20/21): LAST_SWITCHED
                    Type: LAST_SWITCHED (21)
                    Length: 4
                Field (21/21): IP_PROTOCOL_VERSION
                    Type: IP_PROTOCOL_VERSION (60)
                    Length: 1
        FlowSet 5 [id=261] (4 flows)
            FlowSet Id: (Data) (261)
            FlowSet Length: 244
            [Template Frame: 16]
            Flow 1
                SrcAddr: xxx.xxx.xxx.xxx
                DstAddr: xxx.xxx.xxx.xxx
                IP ToS: 0x00
                Protocol: TCP (6)
                SrcPort: 443 (443)
                DstPort: 17776 (17776)
                ICMP Type: 0x0000
                InputInt: 539
                SrcMask: 32
                DstMask: 32
                SrcAS: 0
                DstAS: 0
                BGPNextHop: 0.0.0.0
                TCP Flags: 0x1b, ACK, PSH, SYN, FIN
                    00.. .... = Reserved: 0x0
                    ..0. .... = URG: Not used
                    ...1 .... = ACK: Used
                    .... 1... = PSH: Used
                    .... .0.. = RST: Not used
                    .... ..1. = SYN: Used
                    .... ...1 = FIN: Used
                OutputInt: 0
                NextHop: 0.0.0.0
                Octets: 6166
                Packets: 9
                [Duration: 1.320000000 seconds (switched)]
                    StartTime: 2609.845000000 seconds
                    EndTime: 2611.165000000 seconds
                IPVersion: 4
            Flow 2
                SrcAddr: xxx.xxx.xxx.xxx
                DstAddr: xxx.xxx.xxx.xxx
                IP ToS: 0x00
                Protocol: TCP (6)
                SrcPort: 17776 (17776)
                DstPort: 443 (443)
                ICMP Type: 0x0000
                InputInt: 536
                SrcMask: 32
                DstMask: 32
                SrcAS: 0
                DstAS: 0
                BGPNextHop: 0.0.0.0
                TCP Flags: 0x1b, ACK, PSH, SYN, FIN
                    00.. .... = Reserved: 0x0
                    ..0. .... = URG: Not used
                    ...1 .... = ACK: Used
                    .... 1... = PSH: Used
                    .... .0.. = RST: Not used
                    .... ..1. = SYN: Used
                    .... ...1 = FIN: Used
                OutputInt: 539
                NextHop: 0.0.0.0
                Octets: 1441
                Packets: 13
                [Duration: 1.319000000 seconds (switched)]
                    StartTime: 2609.816000000 seconds
                    EndTime: 2611.135000000 seconds
                IPVersion: 4
            Flow 3
                SrcAddr: xxx.xxx.xxx.xxx
                DstAddr: xxx.xxx.xxx.xxx
                IP ToS: 0x00
                Protocol: UDP (17)
                SrcPort: 53 (53)
                DstPort: 12494 (12494)
                ICMP Type: 0x0000
                InputInt: 539
                SrcMask: 32
                DstMask: 29
                SrcAS: 0
                DstAS: 0
                BGPNextHop: 0.0.0.0
                TCP Flags: 0x00
                    00.. .... = Reserved: 0x0
                    ..0. .... = URG: Not used
                    ...0 .... = ACK: Not used
                    .... 0... = PSH: Not used
                    .... .0.. = RST: Not used
                    .... ..0. = SYN: Not used
                    .... ...0 = FIN: Not used
                OutputInt: 536
                NextHop: 10.232.4.3
                Octets: 76
                Packets: 1
                [Duration: 0.000000000 seconds (switched)]
                    StartTime: 2552.968000000 seconds
                    EndTime: 2552.968000000 seconds
                IPVersion: 4
            Flow 4
                SrcAddr: xxx.xxx.xxx.xxx
                DstAddr: xxx.xxx.xxx.xxx
                IP ToS: 0x00
                Protocol: UDP (17)
                SrcPort: 12494 (12494)
                DstPort: 53 (53)
                ICMP Type: 0x0000
                InputInt: 536
                SrcMask: 29
                DstMask: 32
                SrcAS: 0
                DstAS: 0
                BGPNextHop: 0.0.0.0
                TCP Flags: 0x00
                    00.. .... = Reserved: 0x0
                    ..0. .... = URG: Not used
                    ...0 .... = ACK: Not used
                    .... 0... = PSH: Not used
                    .... .0.. = RST: Not used
                    .... ..0. = SYN: Not used
                    .... ...0 = FIN: Not used
                OutputInt: 539
                NextHop: 0.0.0.0
                Octets: 60
                Packets: 1
                [Duration: 0.000000000 seconds (switched)]
                    StartTime: 2552.811000000 seconds
                    EndTime: 2552.811000000 seconds
                IPVersion: 4
    

    Some other templates are working fine.. I haven't worked out what the relationship is between the successful and failing ones yet.

    [vflow] 2017/12/03 14:52:29 {"AgentID":"10.232.4.5","Header":{"Version":9,"Count":7,"SysUpTime":1708483,"UNIXSecs":1512272852,"SeqNum":427,"SrcID":142},"DataSets":[[{"I":35,"V":2},{"I":34,"V":1},{"I":1,"V":"0x"}],[{"I":35,"V":0},{"I":34,"V":0},{"I":1,"V":"0x"}],[{"I":35,"V":1},....... Any thoughts on how to debug this futher?

    opened by jgc234 8
  • I get error

    I get error

    C:\Users\59401>go get github.com/VerizonDigital/vflow/vflow

    github.com/VerizonDigital/vflow/mirror

    d:\users\go\src\github.com\VerizonDigital\vflow\mirror\mirror.go:65: undefined: syscall.IPPROTO_RAW d:\users\go\src\github.com\VerizonDigital\vflow\mirror\mirror.go:88: cannot assign syscall.Handle to conn.fd (type int) in multiple assignment d:\users\go\src\github.com\VerizonDigital\vflow\mirror\mirror.go:95: cannot use c.fd (type int) as type syscall.Handle in argument to syscall.Sendto d:\users\go\src\github.com\VerizonDigital\vflow\mirror\mirror.go:100: cannot use c.fd (type int) as type syscall.Handle in argument to syscall.Close

    github.com/VerizonDigital/vflow/producer

    d:\users\go\src\github.com\VerizonDigital\vflow\producer\kafka.go:78: undefined: sarama.CompressionLZ4

    opened by aaaasssdd 8
  • Monitoring with influxdb

    Monitoring with influxdb

    How can i use Grafana and InfluxDb for monitoring vflow? I could not find any documentation for enabling monitoring. I am running vflow and influxdb on a separate dockers.

    opened by parmenides 7
  • Errors in trying to use this project as a library

    Errors in trying to use this project as a library

    Last release was 0.9.0 back in 2020.

    At that point in time, The go.mod in this project referred to this project as,github.com/VerizonDigital/vflow. See

    Go tooling tries to pull the latest tag which is 0.9.0 and then fails because it's trying to download github.com/EdgeCast/vflow but the project in that tag is still called, github.com/VerizonDigital/vflow.

    emerald :: ~/hobby/netflow.test 130 » go get -u github.com/EdgeCast/vflow/vflow
    go: github.com/EdgeCast/[email protected]: parsing go.mod:
    	module declares its path as: github.com/VerizonDigital/vflow
    	        but was required as: github.com/EdgeCast/vflow
    

    Please create a new release. The project name in go.mod has been updated already so a new release should fix the problem.

    opened by ishanjain28 6
  • Netflow: index out of range [Huawei]

    Netflow: index out of range [Huawei]

    we are receiving netflow v9 flows from a huawei device. vflow immediately errors out:

    panic: runtime error: index out of range
    
    goroutine 66 [running]:
    github.com/VerizonDigital/vflow/ipfix.Interpret(0xc4202ceb20, 0xf, 0xc4202ceac0, 0xc420126918)
            /root/go/src/github.com/VerizonDigital/vflow/ipfix/interpret.go:67 +0x63c
    github.com/VerizonDigital/vflow/netflow/v9.(*Decoder).decodeData(0xc4202ceeb0, 0x160523, 0xc4203b6000, 0x16, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
            /root/go/src/github.com/VerizonDigital/vflow/netflow/v9/decoder.go:345 +0x1f3
    github.com/VerizonDigital/vflow/netflow/v9.(*Decoder).decodeSet(0xc4202ceeb0, 0xc4203702c0, 0x20, 0x2a, 0xc4202d6140, 0x0, 0x0)
            /root/go/src/github.com/VerizonDigital/vflow/netflow/v9/decoder.go:463 +0x422
    github.com/VerizonDigital/vflow/netflow/v9.(*Decoder).Decode(0xc4202ceeb0, 0xc4203702c0, 0x20, 0x2a, 0x1, 0xb7c620, 0xc4202de000)
            /root/go/src/github.com/VerizonDigital/vflow/netflow/v9/decoder.go:399 +0x156
    main.(*NetflowV9).netflowV9Worker(0xc420121b60, 0xc4203400c0)
            /root/go/src/github.com/VerizonDigital/vflow/vflow/netflow_v9.go:204 +0x394
    main.(*NetflowV9).run.func1(0xc420121b60)
            /root/go/src/github.com/VerizonDigital/vflow/vflow/netflow_v9.go:107 +0x89
    created by main.(*NetflowV9).run
            /root/go/src/github.com/VerizonDigital/vflow/vflow/netflow_v9.go:108 +0x180
    

    can send a pcap file in a private mail if you prefer.

    we are running the binary downloaded from the release page on redhat 7.2

    opened by naude-r 6
  • Error in parse strings with variable length

    Error in parse strings with variable length

    According to RFC7011 Section 6.1.6, "The string is sent as an array of zero or more octets using Information Element fixed or variable length." we can have a variable length "In the Template Set, the Information Element Field Length is as 65535. This reserved length value notifies the that the value of the Information Element will be carried the Information Element content itself." but this is not supported by vFlow and use 65535 for the size of all of the strings.

    bug 
    opened by mohamadrezarostami 5
  • Vflow support of  Kafka Partition key

    Vflow support of Kafka Partition key

    Does Vflow support the use of Kafka partition key to direct flow messages to a specific partition based on device ID for example? This feature will be helpful to do correlation, bit rate calculations ,etc ..

    opened by mayouche77 0
  • SourceID should be uint32

    SourceID should be uint32

    According to https://sflow.org/SFLOW-DATAGRAM5.txt the SourceID should be treated as uint32. The current implementation only allows decoding of one byte instead of the whole data in source_id.

    /* sFlowDataSource encoded as follows:
         The most significant byte of the source_id is used to indicate the type
         of sFlowDataSource:
            0 = ifIndex
            1 = smonVlanDataSource
            2 = entPhysicalEntry
         The lower three bytes contain the relevant index value. */
    
    typedef unsigned int sflow_data_source;
    
    struct flow_sample {
       unsigned int sequence_number;  /* Incremented with each flow sample
                                         generated by this source_id.
                                         Note: If the agent resets the
                                               sample_pool then it must
                                               also reset the sequence_number.*/
       sflow_data_source source_id;   /* sFlowDataSource */
       unsigned int sampling_rate;    /* sFlowPacketSamplingRate */
       unsigned int sample_pool;      /* Total number of packets that could have
                                         been sampled (i.e. packets skipped by
                                         sampling process + total number of
                                         samples) */
       unsigned int drops;            /* Number of times that the sFlow agent
                                         detected that a packet marked to be 
                                         sampled was dropped due to
                                         lack of resources. The drops counter
                                         reports the total number of drops
                                         detected since the agent was last reset.
                                         A high drop rate indicates that the 
                                         management agent is unable to process 
                                         samples as fast as they are being 
                                         generated by hardware. Increasing 
                                         sampling_rate will reduce the drop 
                                         rate. Note: An agent that cannot 
                                         detect drops will always report
                                         zero. */
    
       interface input;               /* Interface packet was received on. */
       interface output;              /* Interface packet was sent on. */
    
       flow_record flow_records<>;    /* Information about a sampled packet */
    }
    
    opened by D3luxee 0
  •  CVE-2019-17543 - LZ4 vulnerability

    CVE-2019-17543 - LZ4 vulnerability

    Are you aware of the following vulnerability https://www.whitesourcesoftware.com/vulnerability-database/CVE-2019-17543 tied to https://github.com/cloudflare/golz4? Thank you

    opened by mayouche77 0
  • Sflow paring issue

    Sflow paring issue

    Hello,

    When I am parsing sflow using vflow so in that 2 cases is happening.

    1. When I have given the below flow it's parsed successfully and given respected output, you can see that I am getting data into Samples

    12:22:28.035013 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto UDP (17), length 216) 172.16.14.5.50315 > 10.20.40.34.6343: sFlowv5, IPv4 agent 192.168.2.3, agent-id 0, seqnum 10897, uptime 1741164485, samples 1, length 188 flow sample (1), length 152, seqnum 3168, type 0, idx 527, rate 1000, pool 3169000, drops 0, input 527 output 0 records 1 enterprise 0 Raw packet (1) length 112 protocol Ethernet (1), length 99, stripped bytes 4, header_size 95

    {"Version":5,"IPVersion":1,"AgentSubID":0,"SequenceNo":10897,"SysUpTime":1741164485,"SamplesNo":1,"Samples":[{"SequenceNo":3168,"SourceID":0,"SamplingRate":1000,"SamplePool":3169000,"Drops":0,"Input":527,"Output":0,"RecordsNo":1,"Records":{"RawHeader":{"L2":{"SrcMAC":"00:50:56:bb:1f:4b","DstMAC":"33:33:00:01:00:03","Vlan":0,"EtherType":34525},"L3":{"Version":6,"TrafficClass":0,"FlowLabel":0,"PayloadLen":41,"NextHeader":17,"HopLimit":1,"Src":"fe80::9de:899c:c1e4:c19c","Dst":"ff02::1:3"},"L4":{"SrcPort":53081,"DstPort":5355}}}}],"Counters":[],"AgentID":"192.168.2.3","ColTime":1635317548}

    1. But from the same server when I am getting below type of flow it's not returning values into Samples, but getting data into Counters

    12:15:38.171862 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto UDP (17), length 232) 172.16.14.5.50315 > 10.20.40.34.6343: sFlowv5, IPv4 agent 192.168.2.3, agent-id 0, seqnum 10851, uptime 1740755104, samples 1, length 204 counter sample (2), length 168, seqnum 4281, type 0, idx 526, records 2 enterprise 0, Generic counter (1) length 88 ifindex 526, iftype 6, ifspeed 1000000000, ifdirection 1 (full-duplex) ifstatus 3, adminstatus: up, operstatus: up In octets 27308635, unicast pkts 303822, multicast pkts 0, broadcast pkts 0, discards 0 In errors 0, unknown protos 0 Out octets 24091432, unicast pkts 184520, multicast pkts 0, broadcast pkts 0, discards 0 Out errors 0, promisc mode 0 enterprise 0, Ethernet counter (2) length 52 align errors 0, fcs errors 0, single collision 0, multiple collision 0, test error 0 deferred 0, late collision 0, excessive collision 0, mac trans error 0 carrier error 0, frames too long 0, mac receive errors 0, symbol errors 0

    {"Version":5,"IPVersion":1,"AgentSubID":0,"SequenceNo":10851,"SysUpTime":1740755104,"SamplesNo":1,"Samples":[],"Counters":[{"SequenceNo":4281,"SourceIDType":0,"SourceIDIdx":526,"RecordsNo":2,"Records":{"EthInt":{"AlignmentErrors":0,"FCSErrors":0,"SingleCollisionFrames":0,"MultipleCollisionFrames":0,"SQETestErrors":0,"DeferredTransmissions":0,"LateCollisions":0,"ExcessiveCollisions":0,"InternalMACTransmitErrors":0,"CarrierSenseErrors":0,"FrameTooLongs":0,"InternalMACReceiveErrors":0,"SymbolErrors":0},"GenInt":{"Index":526,"Type":6,"Speed":1000000000,"Direction":1,"Status":3,"InOctets":27308635,"InUnicastPackets":303822,"InMulticastPackets":0,"InBroadcastPackets":0,"InDiscards":0,"InErrors":0,"InUnknownProtocols":0,"OutOctets":24091432,"OutUnicastPackets":184520,"OutMulticastPackets":0,"OutBroadcastPackets":0,"OutDiscards":0,"OutErrors":0,"PromiscuousMode":0}}}],"AgentID":"192.168.2.3","ColTime":1635317138}

    So here I want to understand between two returning counters and the reason for that.

    If any further details require please let me know.

    opened by KrunalT 3
  • SONIC Sflow support

    SONIC Sflow support

    Does vflow support Sflow collection from SONIC. I'm able to collect sflow from some devices in our environment but I'm not seeing information such as source IP, Source port, DST IP and DST port https://github.com/Azure/SONiC/blob/master/doc/sflow/sflow_hld.md

    opened by mayouche77 0
  • Getting

    Getting "IPFIX element key (890) not exist" from VMware

    Followed the steps mentioned in https://github.com/VerizonDigital/vflow/issues/71 but still getting the same error. changed the file /etc/vflow/ipfix.elements and restarted vflow usinf service vflow restart but still getting the error.

    opened by hitesh-vmware 1
Releases(v0.9.1)
This application is a NetFlow/IPFIX/sFlow collector in Go.

GoFlow This application is a NetFlow/IPFIX/sFlow collector in Go. It gathers network information (IP, interfaces, routers) from different flow protoco

Alireza Rostami 3 Mar 7, 2022
This application is a NetFlow/IPFIX/sFlow collector in Go.

GoFlow This application is a NetFlow/IPFIX/sFlow collector in Go. It gathers network information (IP, interfaces, routers) from different flow protoco

null 2 Mar 11, 2022
A limited Flow Access API which runs outside of the Flow Network using the DPS

Access API Flow DPS implements the Flow Access API Specification, except for the following endpoints: SendTransaction GetLatestProtocolStateSnapshot G

optakt 3 Jul 28, 2022
A library for performing OAuth Device flow and Web application flow in Go client apps.

oauth A library for Go client applications that need to perform OAuth authorization against a server, typically GitHub.com. Traditionally,

GitHub CLI 344 Sep 9, 2022
network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes.

Network Node Manager network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of ku

kakao 98 Sep 22, 2022
K8s-network-config-operator - Kubernetes network config operator to push network config to switches

Kubernetes Network operator Will add more to the readme later :D Operations The

Daniel Hertzberg 6 May 16, 2022
A CLI tool that generates OpenTelemetry Collector binaries based on a manifest.

OpenTelemetry Collector builder This program generates a custom OpenTelemetry Collector binary based on a given configuration. TL;DR $ go get github.c

OpenTelemetry - CNCF 52 Sep 14, 2022
Open-metrics endpoint collector for ONTAP

NetApp Harvest 2.0 The swiss-army knife for monitoring datacenters. The default package collects performance, capacity and hardware metrics from ONTAP

NetApp 92 Sep 19, 2022
File Collector is an application that uses HTTP protocol to collect homework and files.

File Collector File Collector is an application that uses HTTP protocol to collect homework and files. Usage When filecollector is run, it will automa

毛亚琛 1 Jun 16, 2022
Go-based search engine URL collector , support Google, Bing, can be based on Google syntax batch collection URL

Go-based search engine URL collector , support Google, Bing, can be based on Google syntax batch collection URL

Re 60 Sep 29, 2022
This POC is built with the goal to collect events/logs from the host systems such as Kubernetes, Docker, VMs, etc. A buffering layer is added to buffer events from the collector

What is does This POC is build with the goal to collect events/logs from the host systems such as Kubernetes, docker, VMs etc. A buffering layer is ad

Gufran  Mirza 3 Dec 16, 2021
Alibaba iLogtail : The Lightweight Collector of SLS in Alibaba Cloud

Alibaba iLogtail - The Lightweight Collector of SLS in Alibaba Cloud | 中文版本 iLogtail was born for observable scenarios and has many production-level f

Alibaba 931 Sep 26, 2022
A collector api for golang

Install go 1.17 curl https://go.dev/dl/go1.17.3.linux-amd64.tar.gz tar -C /usr/local -xzf go1.17.3.linux-amd64.tar.gz Build for alpine container CGO_

Christophe Varoqui 0 Dec 28, 2021
System information collector

Gohai Gohai is a tool which collects an inventory of system information. It aims to implement some parts of features from facter and ohai. It's forked

null 0 Mar 19, 2020
Observe host metrics collector Written in Go

Observe Host Metrics Collector Experimental collector and forwarder of host metr

Max Skybin 0 Jan 4, 2022
The stats collector for CSC demo files

demoScrape2 The stats collector for CSC demo files. Join the CSC discord

null 1 Jan 8, 2022
Coletor-mpac - Collector of the Public Ministry of Acre (MPAC) using CDP technology

Ministério Público do Acre(MPAC) Esse coletor é baseado na tecnologia Chrome Dev

DadosJusBR 1 Jan 28, 2022
Metrics collector and ebpf-based profiler for C, C++, Golang, and Rust

Apache SkyWalking Rover SkyWalking Rover: Metrics collector and ebpf-based profiler for C, C++, Golang, and Rust. Documentation Official documentation

The Apache Software Foundation 66 Sep 22, 2022
Go library containing a collection of financial functions for time value of money (annuities), cash flow, interest rate conversions, bonds and depreciation calculations.

go-finance Go library containing a collection of financial functions for time value of money (annuities), cash flow, interest rate conversions, bonds

Alejandro Pedraza 143 Sep 26, 2022
🚧 Flexible mechanism to make execution flow interruptible.

?? breaker Flexible mechanism to make execution flow interruptible. ?? Idea The breaker carries a cancellation signal to interrupt an action execution

Kamil Samigullin 7 Sep 26, 2022