Threat intelligence data gathering is a big data challenge. Professionally captured PCAP files are rarely less than 20GBs, making them challenging to process in Wireshark. Even opening a 5GB PCAP on a PC with reasonable CPU spec can take a while, let alone the time for PCAP to process each filter search. As an unstructured data format, searching PCAP is usually limited to simple filtering rules, or requires the data to be uploaded to elastic search tools.
PCAP compressions tools reduce file volumes by 60-80% using standard zip compression, but searching compressed files is usually restricted to timestamp or packet number. Conversely, PCAP indexing tools make it easier to search vast volumes of PCAP, but do not reduce file volumes.
So the Botprobe team developed a different way to turn PCAP into structured traffic data at significantly reduced volumes without compression or indexing - finally making PCAP useful in network traffic forensics:
improved searchability through structured data – replacing filter tools and regex searches with general purpose business tools like Excel, or uploadable to specialist threat detection tools or SIEM solutions;
significantly reduced data footprint – by allowing you to select just the fields or protocols you want to retain, rather than storing the entirety of EVERY packet.
Today, we output to a range of common data formats including .csv, .json, .solr, or directly to elastic search tools. Data is sortable by over 400 packet fields from layers 2 through 7, from protocols including TCP, UDP, DNS, HTTP, SSL/TLS, IRC, SMTP, POP, ARP, ICMP, IGMP, CDP, MODBUS, SS7, DNP 3.0, CIP, Zigbee.