Graphoscope: a solution to access multiple independent data sources from a common UI and show data relations as a graph

Overview

Graphoscope

A solution to access multiple independent data sources from a common UI and show data relations as a graph:

Contains a list of by default available data sources plugins. 3rd party plugins are also possible.

Features

  • Can query both Web GUI and API
  • User registration
  • Every single node & edge type can have its own style
  • No need to remember all the connected data source names - select it from a dropdown
  • Search for more node neighbors
  • Expand all selected nodes or just one
  • Combine the same node attributes from different sources
  • Select multiple nodes with a mouse right button
  • Ability to add notes to nodes and edges, visible to everybody
  • Center graph when it was scrolled/moved far outside of the visible area
  • Delete selected nodes with Del
  • Common fields (like ip, domain, timestamp) that work across all the data sources
  • Green inclusion filters, which work similar to Kibana's filters
  • Red exclusion filters are applied to all nodes (data source independent)
  • Get node & edge details by clicking on them
  • Pie-chart statistics when the amount of requested data exceeds the limit
  • Save & restore dashboards, private or shared with a team
  • Usage examples to skip searching the correct SQL syntax
  • Use SQL features like BETWEEN, IN, LIMIT
  • Use ... OR ... or field IN (...) queries even if data source doesn't support them
  • Export all graph data as a file
  • Display the amount of visible nodes grouped by type
  • Group node neighbors of specific types into the clusters
  • Cache results for a faster re-query
  • Save canvas as an image
  • Notification about new features
  • Format comma or space separated values into a valid search query
  • Upload indicators list and wait for the report
  • Personal graph settings
  • Global graph settings for administrators
  • User management by administrators
  • Built-in documentation
  • Real-time communication between the browser and server through the Websockets

Architecture

Documentation

Installation instructions can be found in INSTALL.md file.

Additional information is available in a Web GUI built-in documentation (also available in docs/):

  • UI elements in section UI
  • Search features in Search
  • Administration and connecting your own data sources in Administration

Plugins for the external data sources

By default available plugins are in plugins/src. 3rd party compiled *.so plugins should go to the plugins directory.

In general there are 3 types of objects:

  • plugin - technically knows how to use a data source (e.g. database or API)
  • source - a named set of parameters describing how to access and interpret a particular data source (using a plugin)
  • collector - actually uses a data source according to the parameters in the source

Plugins development

Check a built-in documentation, section Administration.

TODO & ideas

  • If session's auth or encryption key is changed in a config file - can't access the service until browser's old cookies are deleted
  • Display available data source's fields
  • debug option to see resulting query to the data source and its response
  • In graph.js remove custom zoom limiting when 'https://github.com/visjs/vis-network/pull/629' or similar is merged & new version released
  • Skip a non-unique entry only if all attributes are the same too

  • Generate DEB and RPM packages
  • Log rotation system based on Zerolog
  • More advanced datetime range selector. With options like Last 24h, Last used ranges, etc.
  • Edges groups styling. TODO from search.js. Implement https://github.com/visjs/vis-network/issues/1229
  • Generate PDF documentation from the existing *.md files
  • Implement other SQL features, like NOT BETWEEN
  • Plugins:
    • Redis
    • MS SQL
    • Oracle SQL
    • Apache Cassandra
    • MISP
    • VirusTotal
    • Shodan
    • Poland malware Sandbox API

Usage from Web GUI

Request all people with an age over 30:

FROM sample WHERE age > 30

The results will be similar to:

results

Now it's possible to extend the graph by searching for more of John's neighbors - right click on John and choose Search Sample to search for more data in a Sample data source. We find that Jennifer and Kate also are his friends:

results-extended

Querying the global special source retrieves data from all the data sources connected to this namespace concurrently:

FROM global WHERE age > 30

API usage demo

API can be queried by the external tools, for example with curl:

# SELECT * FROM sample WHERE age > 30
curl -XGET 'https://localhost:443/api?uuid=09e545f2-3986-493c-983a-e39d310f695a&sql=FROM+sample+WHERE+age>30'
# SELECT * FROM global WHERE datetime BETWEEN '2019-01-20T07:27:54+02:00' AND '2019-01-20T07:27:54+02:00'
curl -XGET 'https://localhost:443/api?uuid=09e545f2-3986-493c-983a-e39d310f695a&sql=FROM+global+WHERE+datetime+BETWEEN+%272019-01-20T07:27:54%2B02:00%27+AND+%272019-01-20T07:27:54%2B02:00%27'
# SELECT * FROM intelmq WHERE feed.provider='ShadowServer' AND source.ip='10.10.10.1'
curl -XGET 'https://localhost:443/api?uuid=09e545f2-3986-493c-983a-e39d310f695a&sql=FROM+intelmq+WHERE+feed.provider=%27ShadowServer%27+AND+source.ip=%2710.10.10.1%27'

Response example for the first query:

{
    "relations": [
        {
            "from": {
                "id": "Monica",
                "params": {
                    "age": 35
                },
                "search": "name",
                "group": "name"
            },
            "edge": {
                "label": "lives in"
            },
            "to": {
                "id": "Canada",
                "search": "country",
                "group": "country"
            },
            "source": "sample"
        },{
            "from": {
                "id": "Chin",
                "search": "name",
                "group": "name"
            },
            "to": {
                "id": "Ben",
                "search": "name",
                "group": "name"
            },
            "source": "sample"
        }
    ]
}

... where main fields can be:

  • relations - list of relations, which consist of:
    • from - describes From node of a single relation
    • to - describes To node of the same relation
    • edge - describes a single connection between From and To nodes
    • source - data source name
  • stats - statistics for the processed data when the amount of entries exceeds the limit
  • error - possible error message

Fields to use in queries

sources/*.yaml data sources definitions allow to create common query fields, like ip, domain, datetime, etc. Other fields come from a related data source.

Icons

Possible icons for the node types. Set in files/groups.json. For the other styling options check the documentation's section AdministrationCustom graph elements style.

  • ip -> light blue circle
  • domain -> dark blue circle
  • identifier - vulnerability name -> red rhombus
  • institution -> ping square
  • person -> green circle
  • taxonomy - identifier group's name -> yellow square
  • rtir - ticketing system event's ID -> orange /\ triangle
  • application -> red / triangle
  • email -> green envelope icon

Useful info

License

This project is released under the GNU Affero General Public License v3 or later.

See LICENSE to see the full text.

You might also like...
A Connected Graph Generator tool that construct graphs of some given size

graph graph is a Connected Graph Generator tool that construct graphs of some given size. Notice that it generates all possible connected, undirected

My clean Go solution that's faster than 100%, takes up less memory than 100%.

Remove-element My very clean Go solution that's faster than 100% of all solutions on Leetcode. Leetcode Challenge: "Given an integer array nums and an

Go Solution for LeetCode algorithms problems, 100% coverage.

LeetCode 的 Go 解答 进度 统计规则:1.免费题,2.算法题,3.能提交 Go 解答 Easy Medium Hard Total Accepted 265 456 187 908 Total 267 472 197 936 题解 题号 题目 通过率 难度 收藏 1250 * Check

Two approach for solving common items problem using Golang

Compare Two Arrays For Common Items Given two seperate arrays of integers, create a function that take two arrays and check for common itemss. Example

Common algorithms written in Go.

Common Algorithms in Go This repository contains a collection of a variety of common algorithms implemented using Go. Algorithms Implemented Search Li

A Golang lock-free thread-safe HashMap optimized for fastest read access.

hashmap Overview A Golang lock-free thread-safe HashMap optimized for fastest read access. Usage Set a value for a key in the map: m := &HashMap{} m.S

Access LeetCode problems via id, Golang implementation

LCid-Go Introduction This naive toy is based on bunnyxt/lcid, and implemented in Golang for practice. They are same in program logic and static files.

☔️ A complete Go cache library that brings you multiple ways of managing your caches
☔️ A complete Go cache library that brings you multiple ways of managing your caches

Gocache Guess what is Gocache? a Go cache library. This is an extendable cache library that brings you a lot of features for caching data. Overview He

Stalin sort in multiple languages!

stalin-sort Stalin sort in multiple languages, contributions are welcome! Motivation This repo is motivated by this tweet, this tweet contains a refer

Comments
  • Setting cacheTTL does not seem to have any effect

    Setting cacheTTL does not seem to have any effect

    HI all, When writing plugins, it's useful to launch the same search again and again but the cache hinders the progression. I am not sure it behaves consistently, I tried to lower the cacheTTL in graphoscope.yaml or set it to 0 to no avail.

    Is there a way to disable the cache altogether or to shorten its TTL to close to zero?

    opened by gallypette 4
  • Searching on 'global' did not bring the same results that from 'demo'

    Searching on 'global' did not bring the same results that from 'demo'

    Trying to reproduce the documentation's example:

    demo Searching on global did not bring any result:

    global

    Maybe I missed something, but I expected to get the same results.

    opened by gallypette 1
  • Graph limits

    Graph limits

    Hi,

    I am implementing the LIMIT feature for hashlookup and it looks like the graph cannot display much elements by default. HASHLOOKUP has too many results, add filters manually or use the charts (based on limited data) to reduce the amount of returned data. Or close the charts to see the possible data from the other sources

    Is there a way to set a higher value ?

    opened by gallypette 1
  • Including hashlookup expansion in graphoscope

    Including hashlookup expansion in graphoscope

    What would be the best way to integrate hashlookup in graphoscope. Not sure if there is a kind of skeleton to extend graphoscope or a documentation somewhere on how to do it properly?

    opened by adulau 4
Owner
CERT.LV
CERT.LV
Graph algorithms and data structures

Your basic graph Golang library of basic graph algorithms Topological ordering, image by David Eppstein, CC0 1.0. This library offers efficient and we

Algorithms to Go 9 Jan 25, 2021
A demo to show clearly how Consistent Hashing works.

Consistent Hashing Demo A simple demo of consistent hashing. Features These features have been implemented: Core consistent-hashing-algorithm Consiste

Jasonkay 13 Nov 21, 2022
Go framework to simplify CRUD of structured data using Graph operations

gocrud Go framework to simplify creating, reading, updating, and deleting arbitrary depth structured data — to make building REST services fast and ea

Manish R Jain 311 Nov 28, 2022
Common data structures for solving problems in Golang

datastructs Common data structs for solving problems in Golang. List of data structures can be found in datastructs/pkg Rules for data structures Don'

Akira Masuda 1 Nov 7, 2021
Grokking-algorithms-go - Solutions to common Data Structures problems

This is a repository dedicated to study, learn and solve Data Structure algorith

Gabriel Magalhães 0 Apr 4, 2022
Levenshtein distance and similarity metrics with customizable edit costs and Winkler-like bonus for common prefix.

A Go package for calculating the Levenshtein distance between two strings This package implements distance and similarity metrics for strings, based o

AGExt 74 Dec 15, 2022
Graph algorithms written in Go

Graph Algorithms in Go This repository contains implementations of various graph algorithms written in Go. I’ve written them to learn about these algo

Thomas Cyron 59 Dec 26, 2022
Some algorithms in go: maxflow(min-cuts or graph-cuts), edit-distance.

Algorithms In this repository, some algorithms are implemented in go language. GoDoc link: ed maxflow About Max-flow problem: A flow network is repres

Yi Deng 15 Sep 8, 2022
dagger is a fast, concurrency safe, mutable, in-memory directed graph library with zero dependencies

dagger is a blazing fast, concurrency safe, mutable, in-memory directed graph implementation with zero dependencies

Coleman Word 266 Dec 19, 2022
graph package by golang

graph-go sample package main import ( "fmt" "github.com/Iovesophy/graph-go" ) func main() { samplePlace := []graph.Node{ {ID: 1, Element: "plac

Iovesophy 4 Oct 24, 2021