This is a close to decentralized RSS3 Network implementation of RSS3 protocol v0.4.0 with full indexing function in Go

Related tags

Cryptography web3 rss3
Overview

RSS3

RSS3

RSS3 is a next-generation feed standard that aims to support efficient and decentralized information distribution.

RSS3

Introduction

🚧 The project is under development and the first version is expected to be released in March or April 2022.

We've seen the Web1 era when everyone was happy to use RSS as a standard format for syndication and the Web2 era when RSS declined and the closure of information between platforms are created.

We all agree that information should be more freely distributed in the Web3 era, and to facilitate this we need a common format that we can all use to syndicate information, reducing the cost of use and development. So we created a new protocol RSS3, RSS stands for a tribute to the history and 3 stands for Web3.

This is a close to decentralized RSS3 Network implementation of RSS3 protocol v0.4.0 with full indexing function in Go.

Technical architecture diagram:

Technical architecture diagram

Issues
  • fix: unittest CI on github complete

    fix: unittest CI on github complete

    We should use branching mode instead of fork mode to use the secrets for CI.


    Still has these 2 promblems:

    1. ethers.VerifyMessage returns False when running hub/pkg/verify/verify_test.go "random-1"
    2. gitcoin tests will fail sometimes due to network promblems (usually bot authentication)
    opened by sljeff 3
  • feat(hub): refine hub http api

    feat(hub): refine hub http api

    Test URL

    http://localhost:8080/api/account:[email protected]
    http://localhost:8080/api/account:[email protected]/list/link/following/0
    http://localhost:8080/api/account:[email protected]/list/backlink?limit=10
    
    • [x] Implement business logic (account, linklist, backlinklist)
    • [x] Wrap SQL logic
    • [x] Database logger
    opened by kallydev 3
  • fix: job still restart onfailure

    fix: job still restart onfailure

    according to https://stackoverflow.com/questions/51657105/how-to-ensure-kubernetes-cronjob-does-not-restart-on-failure

    We must set a backoffLimit: 0 in combination with restartPolicy: Never in combination with concurrencyPolicy: Forbid to make sure the cronjob will not restart on failure

    opened by sljeff 1
  • feat: optimize autoupdater cronjob

    feat: optimize autoupdater cronjob

    The diffs:

    • concurrencyPolicy: Allow -> concurrencyPolicy: Forbid
      • The cron job does not allow concurrent runs; if it is time for a new job run and the previous job run hasn't finished yet, the cron job skips the new job run
    • no limit -> activeDeadlineSeconds: 3600
      • one job will run for 3600 secs for maximum
    • schedule: "*/10 * * * *" -> schedule: "0 */6 * * *"
      • runs every 6 hours
    • restartPolicy: OnFailure -> restartPolicy: Never
      • don't restart on failure
    opened by sljeff 1
  • v0.4.0 features, bugs and improvements

    v0.4.0 features, bugs and improvements

    We'll hopefully implement or fix these in the next launch.

    features

    • [x] add profile response for notes/assets. @kallydev
    • [x] page with last_identifier (with one more sql to query the date_created of this identifier then query with that timestamp and offset 1). @kallydev
    • [ ] crossbell indexer (should be an auto-crawler?). [P1]
    • [ ] ens & lens indexer. [P3]
      • [ ] ens was implemented. but there seems a bug of many 429 requests. need find a better way. @Songkeys
    • [ ] search api: rns, ens, name, content, etc. [P2]
    • [ ] notification api (design it in protocol?). [P3]
    • [ ] webhooks (design it in protocol?). [P3]
    • [x] autoupdater dashboard. @sljeff

    improvements

    • [x] add a 5min cache for indexer to avoid too many 429 requests. @Songkeys
    • [ ] notes/assets' profile should always exist even if its owner's notes/assets hasn't been requested. - kurora
    • [x] more explicit params error hint. e.g. invalid param: "last_time".
    • [ ] @kallydev suggests we switching to echo framework for a better error handling
    • [x] crawler_metadata for indexer's auto_crawler @HenryQW [P3]

    bugs

    • [ ] transferred assets should not be still stored in database. (this may need a calculation from notes, or comparison b/w old and new data?) - kurora

    documentation

    • [ ] swagger docs. [P3] - postman
    • [ ] api cookbook. @KyLoc20 suggested us creating a "getting started" guide for api usages, with more focus on "how to build x with rss3 api", e.g. react.js - getting started. currently either the protocol docs or the planned swagger docs looks more like a "handbook", e.g. react.js - api references, which makes clients hard to find a correct way to build something.
    opened by Songkeys 1
  • feat(hub): refine v0.4.0-rc.2 api

    feat(hub): refine v0.4.0-rc.2 api

    Test cases:

    http://localhost:8080/v0.4.0/account:[email protected] http://localhost:8080/v0.4.0/account:[email protected]/profiles http://localhost:8080/v0.4.0/account:[email protected]/links http://localhost:8080/v0.4.0/account:[email protected]/backlinks http://localhost:8080/v0.4.0/account:[email protected]/assets http://localhost:8080/v0.4.0/account:[email protected]/notes

    http://localhost:8080/v0.4.0/account:[email protected]/profiles?profile_sources=Crossbell http://localhost:8080/v0.4.0/account:[email protected]/links?link_sources=Crossbell&limit=10 http://localhost:8080/v0.4.0/account:[email protected]/backlinks?link_sources=Crossbell&limit=10

    http://test-pregod.rss3.dev/v0.4.0/account:[email protected] http://test-pregod.rss3.dev/v0.4.0/account:[email protected]/profiles http://test-pregod.rss3.dev/v0.4.0/account:[email protected]/links http://test-pregod.rss3.dev/v0.4.0/account:[email protected]/backlinks http://test-pregod.rss3.dev/v0.4.0/account:[email protected]/assets http://test-pregod.rss3.dev/v0.4.0/account:[email protected]/notes

    opened by kallydev 1
  • fix: split auto crawler and API; try fix mem leak

    fix: split auto crawler and API; try fix mem leak

    1. Indexer still has mem leak (but much less than before). And it will auto restart when the mem limit is reached.
      • image
    2. The crawler that start with HTTP server should exist only one at a time. So add a new Deployment with replicas: 1 for the crawler.
      • So only the crawler will has mem leak
    3. add pprof for debugging mem leak
    opened by sljeff 1
  • ci: add unittest to ci; add TODOs for fail testing

    ci: add unittest to ci; add TODOs for fail testing

    • Add local config and docker-compose.yml for local testing
    • Add CI unittest
    • Currently there are some unittests can not pass on my local env; TODOs added
    • Why remove all t.Parallel()?
      • some map init will fail (data race detected)
      • data in db will be inconsistent (when updating and checking concurrently)
    opened by sljeff 1
  • refactor: cli migrate and database design

    refactor: cli migrate and database design

    CLI migrate

    # go run ./cli/main.go --help
    go run ./cli/main.go migrate --mongo-dsn "mongodb://rss3:[email protected]/rss3-prod" --postgres-dsn "postgresql://rss3:[email protected]:5432/pregod"
    

    Database

    image

    opened by kallydev 1
  • fix(indexer): panic when headers is nil

    fix(indexer): panic when headers is nil

    Fixed a panic in the indexer util package, as the headers parameter maybe nil.

    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    	panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1299bc2]
    
    opened by kallydev 1
  • [Protocol Adaption] Make distinction between network name and network symbol

    [Protocol Adaption] Make distinction between network name and network symbol

    According to the protocol, network symbol is used in instance uri, wherease the network name is used in metadata.

    Currently code only uses network symbol. In the future, network name is supposed to be added.

    opened by AtlasQuan 0
Releases(v0.1.72)
Go language implementation of a blockchain based on the BDLS BFT protocol. The implementation was adapted from Ethereum and Sperax implementation

BDLS protocol based PoS Blockchain Most functionalities of this client is similar to the Ethereum golang implementation. If you do not find your quest

Yongge Wang 0 Jan 1, 2022
Community-run technology powering the cryptocurrency, and decentralized applications on TrustFi Network

Go TrustFi-Ethereum Official Golang implementation of the TrustFi-Ethereum protocol. Automated builds are available for stable releases and the unstab

TrustFi Network 786 May 26, 2021
A full node Bitcoin (BSV) implementation written in Go

bsvd bsvd is a full node Bitcoin (BSV) implementation written in Go (golang). This project is a port of the bchd codebase to Bitcoin (BSV). It provide

null 41 Feb 7, 2022
Eunomia is a distributed application framework that support Gossip protocol, QuorumNWR algorithm, PBFT algorithm, PoW algorithm, and ZAB protocol and so on.

Introduction Eunomia is a distributed application framework that facilitates developers to quickly develop distributed applications and supports distr

Cong 2 Sep 28, 2021
Go implementation of BLAKE2 (b) cryptographic hash function (optimized for 64-bit platforms).

Go implementation of BLAKE2b collision-resistant cryptographic hash function created by Jean-Philippe Aumasson, Samuel Neves, Zooko Wilcox-O'Hearn, an

Dmitry Chestnykh 90 Jul 11, 2022
Vso-hash - Golang implementation of the BuildXL paged hash function

vso-hash Golang implementation of the BuildXL paged hash function See https://gi

Peter Ebden 6 Jan 3, 2022
A decentralized, cryptocurrency platform that can change this world!

Go Detonus Official Golang implementation of the Detonus protocol. Building the source For prerequisites and detailed build instructions please read t

Detonus (DT) 3 Oct 19, 2021
Cosmochat - A decentralized chat app on Cosmos.

cosmochat cosmochat is a blockchain built using Cosmos SDK and Tendermint and created with Starport. Get started starport chain serve serve command i

Sharad Chand 0 Jan 2, 2022
Interblockchain communication protocol (IBC) implementation in Golang.

ibc-go Interblockchain communication protocol (IBC) implementation in Golang built as a SDK module. Components Core The core/ directory contains the S

COSMOS 231 Aug 9, 2022
Implementation of the Filecoin protocol, written in Go

Project Lotus - 莲 Lotus is an implementation of the Filecoin Distributed Storage Network. For more details about Filecoin, check out the Filecoin Spec

Filecoin 2.4k Aug 10, 2022
Official Golang implementation of the Ethereum protocol

Go Ethereum Official Golang implementation of the Ethereum protocol. Automated builds are available for stable releases and the unstable master branch

StarWORKS Global 0 Nov 24, 2021
Security research and open source implementation of the Apple 'Wireless Accessory Configuration' (WAC) protocol

Apple 'Wireless Accessory Configuration' (WAC) research Introduction This repository contains some research on how the WAC protocol works. I was mostl

Bertold Van den Bergh 8 Jul 28, 2022
Official Go implementation of the Ethereum protocol

Go Ethereum Official Golang implementation of the Ethereum protocol. Automated builds are available for stable releases and the unstable master branch

null 38.7k Aug 10, 2022
RepoETH - Official Golang implementation of the Ethereum protocol

HANNAGAN ALEXANDRE Powershell Go Ethereum Official Golang implementation of the

Alexandre Hannagan 0 Jan 3, 2022
Go-ethereum - Official Golang implementation of the Ethereum protocol

Go Ethereum Official Golang implementation of the Ethereum protocol. Automated b

i06 0 Jan 4, 2022
Dxc - Go implementation of DxChain3.0 protocol

DxChain 3.0 The Ecosystem Powered by DxChain 3.0 Smart Contract Platform While c

DxChain Network 6 Aug 6, 2022
Official Golang implementation of the Ethereum protocol

Go Ethereum Official Golang implementation of the Ethereum protocol. Automated builds are available for stable releases and the unstable master branch

Covalent 18 Jul 23, 2022
Koisan-chain - Official Golang implementation of the Koisan protocol

Go Ethereum Official Golang implementation of the Koisan protocol. Building the

null 0 Feb 6, 2022
Ethereum go-ethereum - Official Golang implementation of the Ethereum protocol

Go Ethereum Official Golang implementation of the Ethereum protocol. Automated b

null 6 Feb 17, 2022